JP2006191357A - Reproduction device and reproduction program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a reproduction device and a reproduction program for switching and reproducing a two-dimensional image and a three-dimensional image on the basis of the two-dimensional image and depth information when a two-dimensional image and depth information of each pixel are recorded in a recording medium specified by a predetermined format. <P>SOLUTION: This reproduction device 100 is provided with: an information separator 102 and a video decoder 103 for separating a first bit stream from a multiplexing stream recorded in a recording medium 9 based on DVD video specifications, and for reproducing a two-dimensional image; an information separator 102, video decoder 103 and depth information extraction unit 104 for separating a second bit stream from the multiplexing stream, and for reproducing information related with the two-dimensional image; and a visual field converter 106 and a stereoscopic image display unit 105 for generating a three-dimensional image based on the reproduced two-dimensional image and information related with the depth of the two dimensional image. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a playback apparatus and a playback program for playing back information related to the depth of a two-dimensional image obtained for each predetermined unit of a two-dimensional image recorded on a recording medium formatted to a predetermined standard.

  Conventionally, for example, in Patent Document 1, stereo images (a pair of two-dimensional images; a left-eye image and a right-eye image) related to a target object imaged by a pair of left and right imaging cameras are associated with each other by a correlation function or the like. An apparatus for obtaining a disparity vector between a pair of pixel points and obtaining a distance from a stereo image to a target object based on the disparity vector is disclosed.

  That is, according to Patent Document 1, for example, the direction of an epipolar line that is a projection line on the right eye image of the attention point of the left eye image and a direction orthogonal to the epipolar line are obtained.

  Then, the two-dimensional search range on the left-eye image is determined by determining the two-dimensional search range on the left-eye image composed of the epipolar line direction search range and the orthogonal direction search range based on the obtained epipolar line direction and the orthogonal directions respectively. The difference between the coordinates of a certain point in the range and the coordinates of the corresponding point on the right-eye image corresponding to this point is obtained as the disparity vector of that point. This disparity vector is obtained at every point in the two-dimensional search range on the left eye image, and the distance from the stereo image to the target object using the obtained disparity vector, in other words, the target object displayed on the stereo image Depth information is calculated.

  Based on the calculated distance from the stereo image to the target object (depth information), the target object can be displayed as a stereoscopic image, that is, a three-dimensional image.

As a technique for recording two types of video data corresponding to two types of two-dimensional video necessary for creating the above-described three-dimensional video, Patent Document 2 discloses two types of video signals as main video streams. One video data is stored, and the other video data created from the one video data by, for example, MPEG (Motion Picture Experts Group) predictive coding is converted into a private stream or user area in the video signal. A method of recording two types of video data necessary for stereoscopic image display by recording as auxiliary data in any usable area such as the above is disclosed.
JP 2000-28356 A Japanese Patent Laid-Open No. 9-327041

  However, in the conventional techniques such as the techniques disclosed in Patent Documents 1 and 2, the two-dimensional video that forms the two-dimensional video (two-dimensional video data) necessary for generating the three-dimensional video from the two-dimensional video data. For the recording medium in which the depth information of the image is standardized in a predetermined format, while maintaining compatibility with the standardized format, switching reproduction (2D video or 3D video efficiently) No specific recording method has been developed so that recording can be performed. Therefore, a reproduction method for generating a three-dimensional image using depth information recorded by the specific recording method has not been developed. There was a demand for development.

  The present invention has been made in view of the above circumstances, and when a two-dimensional image and depth information for each pixel are recorded on a recording medium standardized in a predetermined format, the two-dimensional image and depth information are recorded. It is an object of the present invention to provide a playback device and a playback program capable of switching and playing back two-dimensional images and three-dimensional images based on the above.

  In order to solve the above problems, the invention described in claim 1 is recorded on a recording medium formatted according to a predetermined standard, and a two-dimensional image is compressed by a compression encoding format conforming to the format of the recording medium. A multiplexed stream in which a first bit stream formed by converting and a second bit stream formed by formatting information on the depth of the two-dimensional image obtained for each predetermined unit of the two-dimensional image are multiplexed. A reproducing apparatus for reproducing, wherein the first bit stream is separated from the multiplexed stream to reproduce the two-dimensional image; and the second bit stream is separated from the multiplexed stream and the 2 Means for reproducing information relating to the depth of a two-dimensional image, the reproduced two-dimensional image, and information relating to the depth of the two-dimensional image. And summarized in that a section for generating a three-dimensional image based on.

  In order to solve the above problems, the invention according to claim 2 is recorded on a recording medium formatted according to a predetermined standard, and a two-dimensional image is compressed by a compression encoding format conforming to the format of the recording medium. A multiplexed stream formed by multiplexing a video stream formed by combining the information about the depth of the two-dimensional image stored in an arbitrary use area set in the compression encoding format in the video stream A reproduction apparatus, wherein the bit stream is separated from the multiplexed stream to reproduce the two-dimensional image, and information about the depth of the two-dimensional image stored in the arbitrary use area in the separated bit stream , The reproduced two-dimensional image and the back of the two-dimensional image And summarized in that a section for generating a 3-dimensional image based on the information about the air.

  In order to solve the above problem, the two-dimensional image is a plurality of frame images constituting a two-dimensional video, and the depth information is included in each frame image constituting the two-dimensional video. The pixel value for each pixel as the predetermined unit, and the pixel value representing the depth information for each pixel in each frame image constituting the two-dimensional video is expressed by a compression encoding format compliant with the format of the recording medium. A first compression encoding process for differential encoding in a frame image; a second compression encoding process for run length encoding; at least a future and a past on the time axis of each frame image and each frame image; Encode using a third compression encoding process for predictive encoding from one frame image and orthogonal transform in each frame image Means for reproducing information relating to the depth, which is compressed and encoded by at least one of the four compression encoding processes and recorded on the recording medium. A pixel value representing depth information for each pixel of each frame image that has been compression-encoded by at least one of the compression-encoding processes is used as at least one compression-encoding process. And a means for reproducing information relating to the depth of the two-dimensional image by decoding processing corresponding to the above.

  In order to solve the above-mentioned problem, the invention according to claim 4 is recorded on a recording medium formatted according to a predetermined standard, and a two-dimensional image is compressed by a compression encoding format conforming to the format of the recording medium. A multiplexed stream in which a first bit stream formed by converting and a second bit stream formed by formatting information on the depth of the two-dimensional image obtained for each predetermined unit of the two-dimensional image are multiplexed. A reproduction program executable by a computer for reproduction, wherein the computer separates the first bit stream from the multiplexed stream and reproduces the two-dimensional image; and A process of separating a second bitstream and reproducing information about the depth of the two-dimensional image; It is executed without the 2-dimensional image and the two-dimensional image of a process for generating a three-dimensional image based on depth information respectively summarized as.

  In order to solve the above-mentioned problem, the invention according to claim 5 is recorded on a recording medium formatted according to a predetermined standard, and a two-dimensional image is compressed by a compression encoding format conforming to the format of the recording medium. A multiplexed stream formed by multiplexing a video stream formed by combining the information about the depth of the two-dimensional image stored in an arbitrary use area set in the compression encoding format in the video stream A computer-executable reproduction program for separating the bit stream from the multiplexed stream and reproducing the two-dimensional image in the computer, and for the arbitrary use area in the separated bit stream. Play back information about the depth of the stored 2D image And processing, and summarized in that to execute the reproduced 2D image and the two-dimensional image of a process for generating a three-dimensional image based on depth information respectively.

  In order to solve the above problem, the two-dimensional image is a plurality of frame images constituting a two-dimensional video, and the depth information is included in each frame image constituting the two-dimensional video. The pixel value for each pixel as the predetermined unit, and the pixel value representing the depth information for each pixel in each frame image constituting the two-dimensional video is expressed by a compression encoding format compliant with the format of the recording medium. A first compression encoding process for differential encoding in a frame image; a second compression encoding process for run length encoding; at least a future and a past on the time axis of each frame image and each frame image; Encode using a third compression encoding process for predictive encoding from one frame image and orthogonal transform in each frame image The processing for reproducing the information related to the depth is compressed and encoded by at least one of the four compression encoding processes and recorded on the recording medium. A pixel value representing depth information for each pixel of each frame image that has been compression-encoded by at least one of the compression-encoding processes is used as at least one compression-encoding process. And a process of reproducing information related to the depth of the two-dimensional image by decoding processing corresponding to the above.

  According to the first and fourth aspects of the present invention, by reproducing the multiplexed stream recorded on the recording medium without using the depth information that is integrally multiplexed and recorded, two-dimensional based on the first bit stream is obtained. Images can be played back. Then, by playing back the first video stream using the depth information that is integrally multiplexed and recorded, the first video stream can be switched and played back to the two-dimensional image and the three-dimensional image generated from the depth information.

  According to the second and fifth aspects of the present invention, a bit stream is generated by compressing and encoding a two-dimensional image using a compression encoding format conforming to the format of the recording medium, and information regarding the depth of the two-dimensional image is generated. A bit stream with depth information is generated by storing in an arbitrary use area set in the compression encoding format in the stream, and is recorded on a recording medium by multiplexing.

  Therefore, in a playback apparatus (not shown), a 2D image based on the multiplexed stream is played back by playing back the multiplexed stream recorded on the recording medium without using the depth information recorded in a multiplexed manner. Can do. Then, by reproducing the multiplexed stream using the integrally multiplexed depth information, it is possible to switch to and reproduce the two-dimensional image and the three-dimensional image generated from the information related to the depth.

  In particular, according to the invention described in claims 3 and 6, the pixel value representing the depth information for each pixel in each frame image constituting the two-dimensional video is at least one of the first to fourth compression encoding processes. Depth information for each pixel in each frame image constituting a two-dimensional video by decoding processing corresponding to at least one of the compression encoding processing when the compression encoding processing is performed by one compression encoding processing. Can be played.

  Hereinafter, a plurality of embodiments, which are the best mode for carrying out the present invention, will be described with reference to the drawings.

  In each of the embodiments according to the present invention, 2D video and depth information necessary for 3D video playback obtained from the 2D video are converted into a DVD (Digital Versatile Disk) video standard as a standardized format. A recording apparatus and a recording program for recording on a recording medium (media) in a state compatible with a compliant format, and a reproducing apparatus and a reproduction program for reproducing the two-dimensional video and depth information recorded in this way are described. To do. The recording format corresponding to the recording apparatus and the program according to the present invention is not limited to the DVD video standard compliant format, and a mechanism capable of reproducing 2D video (including both hardware configuration and software configuration). It can be applied to the format of an application program having the same format as that of the DVD video standard and the format of other recording media.

(First embodiment)
FIG. 1 is a block diagram showing a schematic configuration of a recording apparatus 200 according to the first embodiment of the present invention, and FIG. 2 is an object (target object) that is a video recording target of the recording apparatus 200 of the present embodiment. FIG. 4 is a diagram illustrating an arrangement relationship of a recording apparatus 200 with respect to OB.

  As shown in FIGS. 1 and 2, the recording apparatus 200 includes imaging lenses L1 and L2, respectively, and a left-eye moving image that is two-dimensional video {video data (video bitstream)} on the left and right of the target object OB. And a pair of left and right imaging cameras 1A and 1B for capturing a right eye moving image and a right eye moving image, respectively. The imaging lenses L1A and L1B of the imaging cameras 1A and 1B are juxtaposed at the same height on the left and right along a predetermined direction with a fixed or variable interval corresponding to the interval between human eyes. The lenses L1A and L1B have a fixed or variable focal length set in common. That is, the imaging cameras 1A and 1B are two-dimensionally arranged so that the optical axes of the imaging lenses L1A and L1B are included on the same plane and the respective imaging areas are positioned on the same plane.

  The recording apparatus 200 also compresses at least one of the two-dimensional images captured by the imaging cameras 1A and 1B (in this embodiment, the two-dimensional image captured by the imaging camera 1B). 2 and a physical quantity (parallax vector: parallax direction and parallax amount) representing a difference in appearance (parallax) between the frame images (left eye image and right eye image) captured by the imaging cameras 1A and 1B, respectively. A disparity vector extractor 3.

  Furthermore, the recording apparatus 200 calculates the size of the parallax vector extracted by the parallax vector extractor 3, and uses the calculated size of the parallax vector, and the left eye image and the right eye image necessary for generating the three-dimensional video. A depth information calculator 4 for calculating information on depth (depth information), and a depth information formatter 5 for formatting the depth information calculated by the depth information calculator 4. The depth information formatted by the depth information formatter 5 is sent to the video compressor 2 and compressed according to the setting, or directly sent to the information multiplexer 8 described later so as to be multiplexed. It has become.

  The recording device 200 converts a microphone (hereinafter referred to as a microphone) 6 that acquires sound (sound or the like) around the target object OB and an audio signal corresponding to the ambient sound acquired by the microphone 9 to the DVD video standard. And an audio compressor 7 which compresses the MPEG format (MPEG audio) which is a compliant compression format.

  Furthermore, the recording apparatus 200 multiplexes the video data compressed by the video compressor 2, the audio data compressed by the audio compressor 7, and the depth information, thereby data in a format compliant with the DVD video standard. An information multiplexer 8 that generates the data, a recording medium 9 that conforms to the DVD video standard, and a recorder 10 that records data in a format conforming to the DVD video standard multiplexed by the information multiplexer 8 on the recording medium 9 And.

  The recording apparatus 200 includes an imaging camera 1A and 1B, a video compressor 2, a disparity vector extractor 3, a depth information calculator 4, a depth information formatter 5, a microphone 6, an audio compressor 7, an information multiplexer 8, And a controller 11 that is connected to the recorder 10 and controls the entire apparatus. That is, the control unit 11 performs the imaging operation of the imaging cameras 1A and 1B, the compression process of the video compressor 2, the parallax vector extraction process of the parallax vector extractor 3, the depth information calculation process of the depth information calculator 4, and the depth information format. The controller 5 controls the formatting process of the recorder 5, the ambient sound acquisition process of the microphone 6, the compression process of the audio compressor 7, the multiplexing process of the information multiplexer 8, and the recording process of the recorder 10.

  The disparity vector extractor 3 receives the left eye image Pl and the right eye image Pr of the target object OB imaged at the same timing by the pair of left and right imaging cameras 1A and 1B, and receives each pixel (pixel) of the received left eye image Pl. ) And the corresponding pixels of the right eye image Pr are performed using a predetermined evaluation function (for example, the evaluation function disclosed in Japanese Patent Laid-Open No. 9-33249).

  Here, as described above, the pair of left and right imaging cameras 1A and 1B have the same plane with the optical axes Ol and Or of the imaging lenses L1A and L1B spaced apart from each other by a predetermined distance (represented as B in FIG. 3A). (For example, in the present embodiment, the left eye image Pl and the right eye image Pr of the target object OB are arranged so as to be included in the same XZ plane as shown in FIG. As shown to Fig.3 (a), it is obtained so that it may be contained in the same XY plane in the near side (imaging camera side) of the target object OB.

  As long as the optical axes of the imaging cameras 1A and 1B are correctly arranged on the same XZ plane, the search for points (pixels) between the right eye images Pr corresponding to the respective points (pixels) of the left eye image Pl May be performed only on a scanning line that is an epipolar line that is a projection line of each pixel of the left eye image Pl onto the right eye image Pr. However, in practice, it is rather rare that the corresponding pixels for one pixel are arranged without error on the scanning line on the right eye image Pr.

  Therefore, in the present embodiment, the disparity vector extractor 3 calculates each pixel on the right eye image Pr corresponding to each pixel on the left eye image Pl, that is, a corresponding pixel between the left eye image Pl and the right eye image Pr. , Epipolar line direction that is a projection line on the right eye image Pr of each pixel on the left eye image Pl {in this embodiment, the optical axes Ol and Or of the imaging lenses L1A and L1B are included in the same XZ plane Therefore, in addition to the epipolar line direction, a corresponding pixel search and a parallax vector extraction process between the searched corresponding pixels are performed along the vertical direction orthogonal to the epipolar line direction}.

  Here, the corresponding pixel search processing of the disparity vector extractor 3 and the disparity vector calculation processing between the searched corresponding pixels will be described with reference to FIGS. 2, 3A, and 3B.

  For example, as shown in FIG. 3A, a point that has been previously determined in the target object OB or a feature point (coordinates (x, y, z)) that is very easy to determine is a point in the left eye image Pl. Assume that R (Xl, Yl) exists at the point S (Xr, Yr) in the right eye image Pr. The epipolar line on the right eye image Pr of the point R on the left eye image Pl is represented as EPr, and the epipolar line on the left eye image Pl of the point S on the right eye image Pr is represented as EP1.

  At this time, since the optical axes of the imaging cameras 1A and 1B are arranged on the same XZ plane, the Y coordinate Yl of the point R and the Y coordinate Yr of the point S are equal, and the epipolar line EPr and The direction of the epipolar line EPl can be obtained as a direction D1 of a straight line connecting the points R and S (hereinafter referred to as the direction D1 of the epipolar line EP).

  In the present embodiment, installation errors in the height direction (Y-axis direction) of the imaging cameras 1A and 1B can be ignored, and the optical axes Ol and Or of the imaging cameras 1A and 1B are substantially accurate on the same XZ plane. , The direction D1 of the epipolar line EP is substantially parallel to the X axis, as shown in FIG. Therefore, for example, the search for the corresponding point S (Xr, Yr) on the right eye image Pr corresponding to the point R (Xl, Yl) of the left eye image Pl is set on the left eye image Pl and the right eye image Pr. This is executed in the search range W constituted by the width ΔE along the direction D1 of the epipolar line EP and the width ΔT along the direction orthogonal to the direction of the epipolar line EP.

  Specifically, a small block (for example, a pixel block of four horizontal pixels and two vertical pixels) is set in the vicinity of the left eye image Pl including the point R (Xl, Yl) on the search range W. The pixel value of each pixel {represented by a total of 16 bits, for example, 8 bits for luminance signal (Y signal) and 4 bits for color difference signals (Cb signal, Cr signal)} and the right on the search range W The right or the right at which the evaluation parameter is the minimum value, with the difference or sum from the pixel value (16 bits above) of each pixel of each small block corresponding to each point of the eye image Pr as the evaluation parameter. A point on the eye image Pr is obtained as a corresponding point of the point R (Xl, Yl) of the left eye image Pl. In the present embodiment, this corresponding point is the point S (Xr, Yr).

  As a result of considering installation errors in the height direction (Y-axis direction) of the imaging cameras 1A and 1B, for example, when the direction D1 of the epipolar line EP is inclined downward, for example, by an angle θ from the X-axis (FIG. 3C ) In the direction D1 ′ of the epipolar line EP), by expanding the vertical search range by the range calculated by the product of the horizontal search range and tan θ, even in the height direction of the imaging cameras 1A and 1B. Even when an installation error has occurred, the point S (Xr, Yr) on the right eye image Pr corresponding to the point R (Xl, Yl) of the left eye image Pl can be accurately extracted.

  After extracting the point S (Xr, Yr) on the right eye image Pr corresponding to the point R (Xl, Yl) of the left eye image Pl in this way, the disparity vector extractor 3 uses the point of the left eye image Pl. The difference between R (Xl, Yl) and the corresponding point S (Xr, Yr) is taken, and the obtained result is expressed as a disparity vector V (Xl-Xr, Yl-Yr) at the point P (Xl, Yl). This disparity vector is described above for all points in all macroblocks (MB: pixel block of 16 pixels in the horizontal direction × 16 pixels in the vertical direction) in the search range W in the left eye image Pl. It is obtained by the same method as the extraction method at the point R (Xl, Yl). The disparity vector extractor 3 performs the above-described processing over the entire left eye image Pl, and as a result, extracts and extracts all disparity vectors V related to all corresponding points on the left eye image Pl and the right eye image Pr. All the parallax vectors V thus transmitted are sent to the depth information calculator 4.

  The depth information calculator 4 receives each transmitted disparity vector V and calculates the magnitude of each received disparity vector (distance in the depth direction, hereinafter referred to as depth information). Then, the depth information calculator 4 performs a search by setting a small block of 4 horizontal pixels and 2 vertical pixels (4 × 2), for example, and position information {horizontal position (X ), Vertical position (Y), and depth information (Z)}.

  The depth information formatter 5 develops the depth information value of each small block sent as, for example, 8-bit data (hereinafter referred to as depth data) for each pixel constituting each small block. For example, the depth information formatter 5 can set the same depth data value set in the corresponding small block or the average value of the depth data value in units of pixels for each pixel constituting each small block. is there. In addition, the depth information formatter 5 performs a smoothing process (a process in which the depth data values are smoothly connected between the small blocks), for example, via a low-pass filter on the data values set in the pixels of the adjacent small blocks. You can also.

  Based on the depth data of each pixel of the left eye image Pl obtained in this way, the depth information formatter 5 arranges the depth data in, for example, raster order (the order of scanning from the upper left pixel to the lower right pixel). Format. This formatted depth data is transmitted to the video compressor 2.

  An outline of the DVD video format is shown in FIG.

  As shown in FIG. 4, in the DVD video format, a recording layer of a recording medium compliant with the DVD video standard is set as a Volume space, and this Volume space is Volume and File structure 19, DVD-video zone 20, and DVD-others. It is divided into zone 21. The DVD-video zone 20 includes one VMG (Video Manager) 22 and a plurality (n pieces) of VTS (Video Title Set) 23a1 to 23an as all files necessary for DVD video playback. Stored. The VMG 22 includes information such as identification information of subsequent VTSs 23a1 to 23an such as VMGI (Video Manager Information), start addresses and end addresses of various information itself, and from which video stream playback is started. It is.

  The DVD-others zone 21 is an area set as a free use area in the DVD video format that complies with the DVD video standard.

  Each VTS 23a1 to 23an has address information, identification information, etc. of audio data (audio bit stream, hereinafter simply referred to as audio stream) and video data (video bit stream, hereinafter also simply referred to as video stream) to be reproduced. MPEG (Motion) in which a video stream and an audio stream called VOBS (Video Object Set) 25 stored after the control data field 24 and a VOBS (Video Object Set) 25 stored after the control data field 24 are multiplexed. This VOBS 25 is composed of a plurality of MPEG streams called a plurality of VOBs 26 (Video Objects).

  Each VOB 26 is composed of a plurality of subdivided cells (CELL) 27 units. This CELL 27 represents a reproduction unit and is given a unique ID number. Each CELL 27 is composed of a plurality of VOBUs (Video Object Units) 28. Each VOBU 28 has a structure corresponding to a GOP (Group of Pictures) of the MPEG stream, and has a reproduction time length of about 0.4 to 1.0 seconds. This GOP is a structure for predictive coding in MPEG2, which will be described later, and a group {P-pictures (P-pictures: multiple pictures) from I-picture (I picture: intra-frame predictive coded picture) to the next I picture. Forward-predicted encoded image) and B-Pictures (B-pictures: a plurality of bidirectional predictive encoded images)}. For example, a storage recording medium is generally configured as a group of 15 pictures (15 images).

  In each VOBU 28, NV_PACK29, A_PACK30, V_PACK31, and D_PACK32, which are separate MPEG multiplexed stream data, are stored by time division multiplexing, and stream search information and the like are packed in the first NV_PACK29 (for example, And packed in units of transport packets). The A_PACK30 is packed with compression-coded audio data, and the V_PACK31 is packed with compression-coded video data.

  As described above, the D_PACK 32 is time-divided, and each frame layer 33 is configured by a plurality of D_PACKs 32. Each frame layer 33 has a 2-bit start code 34 at the head, and is set to start from 000001FF in hexadecimal, for example. This is also the case when a run length encoding process, a discrete cosine transform (DCT) process, and / or a variable length encoding process (VLC) is performed as a data encoding process to be described later. In order to distinguish from the encoded data, the start code 34 is set as special data starting from 000001FF in hexadecimal.

  In the present embodiment, the right eye image Pr captured by the imaging camera 1B is transmitted to the video compressor 2, and the captured two-dimensional image (right eye image Pr) is shown in FIG. The data is compressed and encoded in the video compressor 2 by the MPEG compression format, which is a compression format compliant with the DVD video format, and transmitted to the information multiplexer 8 as a bit stream (MPEG video stream). Also, the audio signal corresponding to the ambient sound collected by the microphone 6 is compression-encoded in the audio compressor 7 by the MPEG audio format compliant with the DVD video format shown in FIG. 4, and is information multiplexed as audio data (MPEG audio stream). Sent to the generator 8.

  In the information multiplexer 8, the transmitted video data (MPEG bit stream) and audio data (MPEG audio stream) are packed and multiplexed according to the DVD video format shown in FIG. As a result, MPEG multiplexed transport data corresponding to the DVD video format is generated. The generated MPEG multiplexed transport data is sent to the recorder 10 and is recorded on the recording medium 9 compliant with the DVD video standard by the processing of the recorder 10.

  Here, MPEG which is an example of the standard of compression encoding processing in the video compressor 2 will be briefly described.

  MPEG is the name of an organization that examines video coding standards established in 1988 by ISO / IEC JTC1 / SC2 (International Organization for Standardization / International Electrotechnical Standards Meeting Technical Committee 1 / Technical Committee 2, now SC29). Abbreviation for (Moving Pictures Expert Group). MPEG1 (MPEG Phase 1) is a compression standard for storage media of about 1.5 Mbps, and JPEG for the purpose of still image compression coding and moving images for low transfer rates of ISDN video conferences and videophones. It inherits the basic technology of H.261 (CCITT SGXV, standardized by the current ITU-T SG15) for the purpose of compression, and introduces a new compression encoding technology for storage media. These were established in August 1993 as ISO / IEC 11172.

  MPEG2 (MPEG phase 2) was established as ISO / IEC 13818, H.262 in November 1994 as a standard of general-purpose compression coding technology that can be used for various applications such as communication and broadcasting.

  MPEG is designed by combining several technologies, and in particular, a predictive coding technique for removing redundancy in the time domain {intraframe predictive coding (interframe differential coding) technique, motion compensation technique}, and frequency domain. In combination with a discrete cosine transform technique for redundancy removal.

  That is, the inter-frame difference encoding technique in the predictive encoding technique takes advantage of the fact that two consecutive frame images in video data are similar to each other, and takes the difference between the two frame images. This is a technique for deleting a time redundant portion by encoding.

  The prediction direction in the motion compensation technique of the predictive coding technique has three modes of past, future, past and future. The three modes can be switched and used in units of 16 × 16 pixel macroblocks (MB). The prediction direction is determined by the type of frame image (picture) given as an input image, for example.

  As described above, there are three types of images (pictures) having different data amounts in a frame subjected to MPEG predictive encoding.

  That is, encoding is performed by switching between a mode for performing interframe predictive encoding in the forward direction from a past frame image and a mode for independently encoding pixels in a macroblock without performing prediction. There is a P picture.

  In addition, a mode for performing forward interframe prediction encoding from a past frame image, a mode for performing interframe predictive encoding in the reverse direction from a future frame image, and bi-directional from two past and future frame images The B picture is encoded by switching between the mode in which the inter-frame predictive encoding is performed and the mode in which the pixels in the macro block are independently encoded without performing the prediction.

  In the I picture, the pixels of all the macroblocks in the frame image are intraframe-coded independently for each macroblock.

  That is, the motion compensation technique detects a motion region (motion vector) between frame images with half pixel accuracy by pattern matching for each macroblock, and shifts the detected motion vector by the detected motion vector before the frame. This is a technique for performing inter prediction encoding. The motion vector has a horizontal direction and a vertical direction, and the encoded data of the motion vector can be transmitted as additional information of the macroblock together with MC (Motion Compensation) mode information indicating the prediction start position.

  Discrete cosine transform technology (DCT technology) is a technology for performing orthogonal transform by DCT (discrete transform) to integral space using a cosine function as an integral kernel, and using this DCT, a pixel value of a frame image is converted into a frequency domain. It is possible to cut high frequency components in the frame image by converting into data.

  FIG. 5 is a block diagram showing an example of an encoding apparatus for executing the above-described MPEG compression encoding process.

  As shown in FIG. 5, the encoding device 50 includes a subtraction processing computing unit 51 and a motion compensation predictor 52 to which video data (moving images in frame units) to be compressed and encoded are input, and the above-described operation. DCT unit 53 for discrete transform processing, quantizer 54 for performing quantization processing, VLC unit 55 for performing variable length coding processing, buffer 56, and code amount control for performing feedback control based on the code amount Instrument 57.

  The encoding device 50 also includes an inverse quantizer 58 for performing inverse quantization processing, an inverse DCT device 59 for performing inverse DCT, an arithmetic unit 60 for addition processing, and an image memory 61 for storing images. It has.

  According to the encoding device 50 shown in FIG. 5, the Nth frame image in the bit stream (MPEG video stream) VS input to the computing unit 51 is moved by the motion compensated predictor 52 described later. The difference from the compensated reference (reference) decoded image (N−1th frame image) is calculated (inter-frame difference encoding process), and the obtained difference image is transmitted to the DCT unit 53. In the DCT unit 53, the transmitted difference image is converted into frequency domain image data. That is, in the DCT unit 53, the transmitted difference image is divided into 8 × 8 DCT blocks divided into four macro blocks, and a two-dimensional DCT process is executed for each DCT block. Here, since video data generally has many components in the low frequency band and few components in the high frequency band, the image data after DCT processing (each DCT coefficient block) is concentrated in the low frequency band.

  Image data (each DCT coefficient block) obtained by the DCT processing is sent to the quantizer 54 and quantized.

  That is, in the quantizer 54, each DCT coefficient block weights the value of each cell of two-dimensional frequency values expressed as an 8 × 8 matrix based on visual characteristics (human visual recognition characteristics), and For example, a corresponding cell based on an 8 × 8 quantization matrix composed of quantized values obtained by multiplying by a quantization scale value for multiplying the whole specified in frame units or macroblock units by a scalar Divide between each other. As a result, the lower right data (high frequency band coefficient) in the 8 × 8 DCT coefficient data becomes 0, and the data in the high frequency band can be removed.

  In this way, the quantized data (8 × 8 quantized data blocks) from which the high frequency band data has been removed is sent to the VLC unit 55 and the inverse quantizer 58, respectively.

  In the VLC unit 55, variable length coding processing is performed for each block.

  Here, the upper left cell of each 8 × 8 quantized data block corresponds to a constant term after DCT, and represents the average value of the entire waveform, in other words, the direct current component (DC component), and the remaining cells. Corresponds to an AC component (higher order component).

  At this time, in the VLC unit 55, the DC component of the upper left cell in each quantized data block is predictively encoded by differential PCM {Differential Pulse Code Modulation)} which is one of predictive encoding. In addition, the remaining cells (AC component) constituting the AC component are directed in a direction perpendicular to the diagonal line connecting the upper left side and the lower right side from the low frequency region (upper left) to the high frequency region (lower right). Are sequentially read out by a zigzag scan and aligned in a line. The data in the rear part of the data read out by the zigzag scan and arranged in a line corresponds to a high frequency component, and 0s are continuously generated.

  Therefore, in the VLC unit 55, for example, a continuous run length of 0 and a valid data value other than 0 are regarded as one event, and a code with a short code length is assigned from a data value with a high probability of appearance, so that the Huffman code It becomes. The motion vector obtained by the motion compensation predictor 52 is also sent to the VLC unit 55 and Huffman coded.

  The Huffman-encoded encoded data (the motion vector encoded data is added), that is, the MPEG video stream VS of the differential image MPEG-encoded by the encoding device 50 is temporarily stored. Are stored in the buffer 56 and output at a predetermined transfer rate. At this time, the data representing the generated code amount for each macroblock in the output MPEG video stream VS is transmitted to the code amount controller 57, and the generated code amount data and the target data transmitted by the code amount controller 57 are transmitted. An error code amount between the code amount to be obtained is obtained. This error code amount is fed back to the quantizer 54, and the quantizer 54 adjusts the quantization scale value based on the fed back error code amount. As a result, an optimal quantization scale value can be set according to the encoding amount of the MPEG video stream VS after MPEG compression encoding.

  On the other hand, each 8 × 8 quantized data block sent from the quantizer 54 to the inverse quantizer 58 is inversely quantized and obtained by performing a process reverse to the quantization process of the quantizer 54. The DCT coefficient block is subjected to inverse DCT processing in an inverse DCT unit 59, and as a result, a difference image is obtained. The obtained difference image is sent to the computing unit 60, where it is added to the motion-compensated decoded reference image (previous frame image) sent from the motion compensation predictor 52, and the added frame. An image, that is, a playback frame image corresponding to the input Nth frame image is stored in the image memory 61 as a reference decoded image for the next input N + 1th frame image.

  In the motion compensated predictor 52, every macroblock between the current input frame image (for example, the Nth frame image) and the previous frame image (for example, the (N-1) th frame image) stored in the image memory 61. The previous frame image shifted by the motion vector obtained as a result is transmitted to the computing units 51 and 60 as a decoded image for reference, respectively.

  As described above, the video data input to the encoding device 50 is encoded with MPEG technology, that is, time domain predictive encoding processing (frame difference encoding, motion compensation), frequency domain compression encoding processing (discrete cosine). Conversion) and variable length coding processing (Huffman coding).

  On the other hand, FIG. 6 is a block diagram showing a decoding apparatus 70 that decodes and reproduces video data compression-encoded by the encoding apparatus 50 shown in FIG.

  As shown in FIG. 6, the decoding device 70 includes a buffer 71 to which the MPEG video stream VS compressed and encoded by the encoding device 50 is input, and a VLD (Variable length Decoding) device 72 that performs variable length decoding processing. An inverse quantizer 73 for performing inverse quantization processing, an inverse DCT device 74 for performing inverse DCT, an arithmetic unit 75 for addition processing, a motion compensation predictor 76, and an image memory 77. Yes.

  According to the decoding device 70 shown in FIG. 6, the input MPEG video stream (for example, corresponding to the Nth frame image) VS is stored (buffered) in the buffer 71, and is stored in the VLD unit 72. Entered.

  In the VLD unit 72, the MPEG video stream VS is subjected to variable length decoding processing to obtain DC component data and AC component data, respectively. The DC component data is arranged in the upper left cell of the 8 × 8 quantized data block, and the AC component data is stored in the low frequency band described above with respect to the remaining cells of the 8 × 8 quantized block. They are arranged in order by zigzag scanning from the (upper left) to the high frequency range (lower right). The VLD unit 72 also decodes the encoded motion vector, and this motion vector is transmitted to the motion compensation predictor 76.

  Each 8 × 8 quantized data block obtained in this way is inversely quantized by performing a process reverse to the quantization process of the quantizer 54 by the inverse quantizer 73. The DCT coefficient block obtained by inverse quantization is subjected to inverse DCT processing in an inverse DCT unit 74, and as a result, a differential image corresponding to the input MPEG video stream VS is obtained.

  The obtained difference image is sent to the computing unit 75, where it is added to the motion-compensated reference decoded image (previously reproduced frame image) sent from the motion compensation predictor 76. The reproduced frame image, that is, the reproduction frame image (decoded data) corresponding to the input Nth frame image is reproduced using a reproduction device (not shown). The reproduction frame image is stored in the image memory 77 as a reference decoded image for the N + 1th frame image to be input next.

  In the motion compensation predictor 76, a macro between the current input frame image (for example, the Nth frame image) and the previously reproduced frame image (for example, the (N-1) th frame image) stored in the image memory 77 is displayed. The pattern matching process for each block is executed, and the frame image reproduced last time shifted by the motion vector obtained as a result is transmitted to the calculator 75 as a decoded image for reference.

  As described above, the MPEG video stream VS encoded by the encoding device 50 and input to the decoding device 70 is decoded by the decoding device 70 as a playback frame image.

  The depth data sent from the depth information formatter 5 is compression-encoded by, for example, providing the configuration and / or function of the compression-encoding device 50 to the video compressor 2 in the recording apparatus 200 of the present embodiment. Can do. Then, by providing the reproducing apparatus (not shown) with the configuration and / or function of the decoding apparatus 70, it is possible to reproduce the corresponding frame image from the MPEG video stream VS compression-encoded by the recording apparatus 200.

  Here, a schematic configuration of a hierarchical structure (syntax) of the MPEG video stream VS generated by the encoding device 50 is shown in FIG.

  As shown in FIG. 7, the MPEG video stream VS is composed of six hierarchies (layers), and the sequence layer, which is the highest layer, has the syntax shown in FIG. 8A as the table T1. is doing. Note that the syntax of an MPEG video stream VS, which will be described later, including this table T1 is a syntax on the side of the decoding device 70 used for extracting a data element from the MPEG bit stream VS. The syntax on the encoding device 50 side is a syntax in which conditional statements such as an if statement and a while statement are omitted from the syntax on the decoding device 70 side.

  As shown in FIG. 8A, Next_Start_code in video_sequence () in the sequence layer is a function for searching for a start code described in the MPEG bitstream, and the do {} while syntax is defined by a while statement. This is a syntax for extracting the data element described based on the function in {} of the do statement from the MPEG bitstream while the specified condition is true. Therefore, in this sequence layer, while the condition of the sequence header code (Sequence_header_code) described by the while statement is true, the data element described based on the sequence header (Sequence_header) function is extracted from the MPEG bit stream. While the condition of the group start code (Group_start_code) of the GOP layer (group_of_pictures layer) described in the while statement is true, the data elements described based on the function of the GOP layer are further extracted from the extracted data elements Describes the decoding method.

  The sequence header is a layer for designating a screen format and the like, and has a syntax shown as a table T2 in FIG.

  The GOP layer has a syntax structure shown as a table T3 in FIG. 9, and stores arbitrary data unrelated to video data and audio data by the user as shown as “user_data” in FIG. A user data area UD1 (see FIG. 7) is set.

  The Picture layer that is a lower layer of the GOP layer and defines the data element to be extracted has a syntax structure shown as a table T4 in FIG. A user data area UD2 (see FIG. 7) is set in which arbitrary data unrelated to audio data can be stored.

  A slice layer that is a lower layer of the picture layer and defines a data element to be extracted represents a band of a macroblock, and has a syntax structure shown as a table T5 in FIG.

  A Macroblock layer that is a lower layer of the Slice layer and defines a data element to be extracted has a syntax structure shown as a table T6 in FIG. As macroblocks (MB) in normal MPEG intraframe prediction coding, four 8 × 8 DCT blocks for Y signal and two 8 × 8 DCT blocks for color difference signals (Cb and Cr) are used. Therefore, the repetition range of the for statement in the if statement (macroblock_pattern) in the syntax is set as “for (i = 0; i <6; i ++)”.

  Further, the block layer that is a lower layer of the macroblock layer and defines the data element (8 × 8 small block) to be extracted has a syntax structure shown as a table T7 in FIG.

  That is, the video data (right eye image Pr) input to the video compressor 2 is compression-encoded by the MPEG compression encoding process shown in FIG. 5 of the video compressor 2 and has the structure shown in FIGS. An MPEG video stream VS is generated.

  Next, a process for recording the depth data in the present embodiment so as to be compatible with the MPEG compression encoding format based on the DVD video format will be described.

  In the present embodiment, the video compressor 2 converts the 8-bit depth data of each pixel sent from the depth information formatter 5 based on any one of the following data formats (1) to (4). Thus, for example, the MPEG video stream VS is stored in the user data area UD1 and / or the user data area UD2 in the MPEG compression-encoded video data (MPEG video stream) VS based on the right eye image Pr, and the MPEG video stream VS including depth data is information multiplexed. Is configured to transmit to the generator 6.

(1) Format stored in order from the depth data of the upper left pixel to the depth data of the lower right pixel (raster order) (2) When the depth data value of each pixel shows the same value, the same value A format in which the data representing the number of pixels (run length) indicating 7 bits and the same data value are compressed and stored so that the same data values are arranged. (3) Depth data of each pixel is converted into an 8-bit grayscale image (Y signal). In addition to the intra-frame encoding of the formats (4) and (3) for storing by intra-frame predictive encoding in the same manner as the I picture image in the above-mentioned MPEG compression encoding, A forward inter-frame predictive encoding and a bi-directional inter-frame insert predictive encoding for P picture images and B picture images are stored. Matte For example, in the format (2), as shown in FIG. 4, each frame layer 33 has a 1-bit depth data valid flag (DE) 35 following the start code 34, and this depth. When “1” is set in the data valid flag 35, the following depth data is valid.

  On the other hand, when “0” is set in the depth data valid flag 35, the following depth data is invalid, and the depth of the corresponding frame layer 33 is not changed at all, so that each pixel has no change. The depth data is uniformly set to an OFFSET value described later.

  Following the depth data valid flag 35, the frame layer 33 stores a 7-bit reserved field 36 and an 8-bit offset (OFFSET) value to be added to the depth data of each pixel. Field OFFSET37.

  In this OFFSET 37, a value from −127 to +127 (8 bits) is stored as an offset value. The offset data is stored in a pixel layer to be described later, and the depth data value to which the offset is added is 255 (8 When the value exceeds the maximum value that can be expressed in bits (overflow), the data width of the depth data can be suppressed to 8 bits by the limiter function each time.

  For example, if “0” is set in the depth data valid flag 35 and “0” is also set in the OFFSET 37, it is assumed that some depth data value is stored in each data field (pixel layer) described later. In other words, the depth data values of all the pixels are determined to be “0”.

  The frame layer 33 has a field (pixel layer 38) for storing depth data of each pixel (pixel) following the field OFFSET37.

  The pixel layer 38 includes a NumOfSkipPixel field 38a for representing the number of consecutive pixels having the same depth data as a run length in 8 bits, and a field (ZP) for representing the same depth data as a value from −128 to +127 in 8 bits. 38b.

  For example, when four pixels having depth data of “100” are consecutive in the row direction, for example, a pixel layer is assigned to each of the four pixels, and the depth data is not stored in the ZP field 38b. Depth data “100” is stored by storing “4” as the number of continuous pixels (run length) in the NumOfSkipPixel field 38 a of the layer 38 and storing “100” as the depth data value in the corresponding ZP field 38 b. Can be compressed and stored in the same manner as when four pixels are stored continuously. As an application of the format (2), the depth data of adjacent pixels are often very similar. Therefore, the difference between adjacent pixels is taken, and the difference data as the number of adjacent pixels and the difference result are respectively obtained from the pixel layer 38. The NumOfSkipPixel field 38a and the ZP field 38b can be stored, and the depth data can be further compressed to improve the encoding efficiency.

  In the data format (3), since the depth data of each pixel is regarded as an 8-bit grayscale image (Y signal), there is no color difference signal. Therefore, four 8 × 8 DCT blocks for Y signal and two 8 × 8 DCT blocks for color difference signals (Cb and Cr) are used as macroblocks (MB) in normal intra-frame prediction encoding of MPEG. While a total of 6 DCT blocks are used (see the for statement repetition range “for (i = 0; i <6; i ++)” in table T6 in FIG. 12), in this embodiment, the depth of each pixel Since the 8-bit grayscale image, which is data, is subjected to intraframe prediction encoding using four 8 × 8 DCT blocks for Y signal, the for sentence repetition range is “for (i = 0; i <4; i ++) ”.

  Specifically, as shown as table T4 in FIG. 10, in the picture layer of the MPEG bitstream, the last if statement before the transition to the data element extraction process defined by the slice layer (Slice function) A mechanism is defined in which user data (user_data) can be recorded in 8-bit units after sending a user data start code (user_data_start_code).

  More specifically, in the MPEG bitstream, the user data start code (user_data_start_code) is generally defined as “0x000001B2” before moving to the data element extraction process defined by the slice layer. That is, the video compressor 2 according to the present embodiment transmits a user code as an MPEG video stream VS to the information multiplexer 8, and subsequently uses a function used for authentication in the user data area UD1 or UD2. For example, a code of 0x0f0f0f0f2428fdaa, which is a uniquely identifiable code (identification code) indicating the presence of a value, is transmitted. This identification code is recorded for the purpose of identifying user data (user_data) in other devices or applications, and the value of the identification code has no special meaning. Then, following the identification code, the video compressor 2 stores the depth data of each pixel configured as the frame layer structure shown in FIG. 4 in the user data area UD2 of the picture layer in the MPEG video stream VS in raster order, for example. It is supposed to be.

  In this way, the MPEG video stream VS in which the depth data of each pixel is stored in the user data area UD1 or UD2 is transmitted to the information multiplexer 8 and is sent from the audio compressor 7 according to the DVD video format shown in FIG. The transmitted audio data (MPEG audio stream AS) is multiplexed in accordance with the DVD video format shown in FIG. 4, and this multiplexed data stream (MPEG multiplexed transport stream TS) is recorded by the recorder 10 in the DVD video format. Is recorded on the recording medium 9 compliant with the above.

  As described above, according to the recording apparatus 200 according to the present embodiment, the depth information of each pixel for generating 3D video obtained from 2D video is obtained by MPEG compression encoding processing compliant with the DVD video format. It can be stored in the user data area UD1 or UD2, which is a user arbitrary use area in the compression-coded MPEG video stream VS. Then, the MPEG video stream in which the depth information is integrated in this way can be multiplexed with the MPEG audio stream and recorded as an MPEG multiplexed transport stream TS on the recording medium 9 compliant with the DVD video format.

  For this reason, in a playback apparatus (not shown), the MPEG multiplexed transport stream TS recorded on the recording medium 9 is played back without using the depth information recorded in an integrated manner, so that the MPEG video streams VS and MPEG are recorded. Two-dimensional video and audio signals based on the audio stream AS can be reproduced. Then, by reproducing the MPEG multiplexed transport stream TS recorded on the recording medium 9 using the depth information integrally recorded by multiplexing, 3D video and MPEG audio based on the depth information and the MPEG video stream VS are reproduced. Each audio signal based on the stream AS can be reproduced.

  Therefore, according to the recording apparatus 200 according to the present embodiment, on the playback apparatus side, switching playback of 2D video and 3D video is enabled by switching between using / not using depth information recorded on the recording medium 9. A recording system can be constructed, and the reproduction efficiency of 2D video and 3D video recorded on the recording medium 9 by the recording device 200 can be improved.

(Second Embodiment)
FIG. 14 is a block diagram showing a schematic configuration of a recording apparatus 200A according to the second embodiment of the present invention. In the recording apparatus 200A in the present embodiment, the processing of the video compressor and the information multiplexer is different from that of the recording apparatus 200 shown in FIG.

  In the recording apparatus 200A according to the present embodiment, 8-bit depth data of each pixel sent from the depth information formatter 5 is compressed directly or by the video compressor 2A by the run length encoding (2). Data for each encoded frame layer, data that has been subjected to intraframe prediction encoding according to (3), or data that has been compression encoded by interframe insertion prediction encoding according to (4), MPEG compression code by the video compressor 2A The encoded MPEG video stream VS is transmitted to the information multiplexer 8A as a bit stream. The MPEG video stream VS and the MPEG audio stream AS MPEG-encoded by the audio compressor 7 are also transmitted to the information multiplexer 8A.

  The information multiplexer 8A uses the transmitted data corresponding to the depth data of each pixel of the left eye image Pl as the data content (for example, 2 kB) of D_PACK32 in the DVD video format (see FIG. 4). By multiplexing with V_PACK31 corresponding to VS, A_PACK30 corresponding to MPEG audio stream AS, etc., the depth data is converted into a DVD video format.

  In this way, the MPEG multiplexed transport stream TS in which the depth data of each pixel is multiplexed is recorded on the recording medium 9 compliant with the DVD video format by the recorder 10.

  As described above, according to the recording apparatus 200A according to the present embodiment, the depth information of each pixel for generating a 3D image obtained from the 2D image is obtained by multiplexing the DVD video format. An MPEG multiplexed transport stream TS in which depth information is integrated by storing in each pixel layer in each frame layer 33 corresponding to the DVD video format is generated, and the generated MPEG multiplexed transport stream TS is converted into the DVD video format. Can be recorded on the recording medium 9 compliant with the above.

  For this reason, as in the first embodiment, in a playback device (not shown), the MPEG multiplexed transport stream TS recorded on the recording medium 9 is played back without using the depth information recorded in a multiplexed manner. Thus, it is possible to reproduce the two-dimensional video and audio signal based on the MPEG video stream VS and the MPEG audio stream AS, respectively. Then, by reproducing the MPEG multiplexed transport stream TS using the depth information recorded in an integrated manner, the 3D video based on the depth information and the MPEG video stream VS and the audio signal based on the MPEG audio stream AS are obtained. Each can be played.

  Therefore, also in the recording apparatus 200A according to the present embodiment, it is possible to obtain the effect of improving the reproduction efficiency of the 2D video and the 3D video recorded on the recording medium 9, which is the same effect as the first embodiment. .

  As a modification of the first and second embodiments, the information multiplexer uses a “transport data flag (transport_private_data_flag) in the system layer (see table T8 in FIG. 15) of the generated example MPEG multiplexed transport stream TS. ) ”Is set to“ 1 ”to clearly indicate that private data (private_data) exists, and then, under the restriction that the data length does not protrude from the transport packet, the MPEG multiplexed transport stream TS Depth data of each pixel can be stored as private_data in a private data field having a data length set in transport_private_data_length and recorded on the recording medium 9.

  Further, the recording apparatus according to the modification of the first and second embodiments sets a private stream (private_stream) in the stream identification information (stream_id), declares a dedicated packet, and is secured as this dedicated packet. It is also possible to pack the depth data into a field (private stream field) and record it on the recording medium 9 via the recorder 10 for each frame image.

(Third embodiment)
FIG. 16 is a block diagram showing a schematic configuration of a recording apparatus 200B according to the third embodiment of the present invention. In the recording apparatus 200B in the present embodiment, the processing of the video compressor and the information multiplexer is different from that of the recording apparatus 200A shown in FIG.

  In the recording apparatus 200B according to the present embodiment, 8-bit depth data of each pixel sent from the depth information formatter 5 is compressed by the run length encoding (2) directly or by the video compressor 2A. Data for each encoded frame layer, data that has been subjected to intraframe prediction encoding according to (3), or data that has been compression encoded by interframe insertion prediction encoding according to (4), MPEG compression code by the video compressor 2A The data stream is transmitted to the information multiplexer 8B as a data stream different from the converted MPEG video stream VS. The MPEG video stream VS and the MPEG audio stream AS MPEG-encoded by the audio compressor 7 are also transmitted to the information multiplexer 8B.

  As shown in FIG. 17, the information multiplexer 8 </ b> B according to the present embodiment uses data corresponding to the transmitted depth data of each pixel of the left eye image Pl as shown in FIG. 17 instead of D_PACK32 in the DVD video format shown in FIG. 4. The data content of the DVD-others zone 21, which is an area set as a free use area in the DVD video format compliant with the DVD video standard, is linked to the DVD-video zone 20 and multiplexed.

  The DVD-others zone 21 includes one DVMG (D-Video Manager) 82 and a plurality (n) of DVTS (D-Video Title Set) 83a1 to 83an. The DVMG 82 describes information such as identification information of subsequent DVTSs 83a1 to 83an such as video manager information, start addresses and end addresses of various information itself, and from which video stream playback is started.

  Each of the DVTSs 83a1 to 83an includes a field DVTSI 84 in which control data (control data) such as address information and identification information of audio data and video data (video stream) to be reproduced is stored, and a DVOBS ( A D-Video Object Set (Video Object Set) 85 is composed of a set (content) of MPEG (Motion Picture Experts Group) streams in which a video stream and an audio stream are multiplexed. The DVOBS 85 includes a plurality of DVOBs 86 ( D-Video Object) is composed of a small unit MPEG stream.

  Each DVOB 86 is composed of a plurality of subdivided cells (DCELL) 87 units. This DCELL 87 represents a reproduction unit and is given a unique ID number. Each DCELL 87 further includes a plurality of DVOBUs (D-Video Object Units) 88.

  In the present embodiment, each DVOB 88 is configured by grouping several frame layers 33 each having the depth data of each pixel stored as data of each pixel layer 38 in the second embodiment.

  That is, the information multiplexer 8B according to the present embodiment generates the frame layer 33 from the depth data of each pixel for each frame image transmitted from the depth information formatter 5, and corresponds to the generated plurality of frame images. A plurality of frame layers 33 are grouped to generate DVOBU 88 in the DVD video format, thereby converting the depth data into the DVD video format.

  In this way, the MPEG multiplexed transport stream TS in which the depth data of each pixel is multiplexed is recorded on the recording medium 9 compliant with the DVD video format by the recorder 10.

  Also in the present embodiment, as in the first and second embodiments, in a playback apparatus (not shown), multiplexed data recorded on the recording medium 9 is recorded in an area corresponding to the DVD-others zone 21 on the recording medium 9. By reproducing without using the depth information, a two-dimensional video and an audio signal based on the MPEG video stream VS and the MPEG audio stream AS can be reproduced, respectively. Then, by performing the reproduction process of the MPEG multiplexed transport stream TS using the depth information recorded in the area corresponding to the DVD-others zone 21 in the recording medium 9, the 3D video based on the depth information and the MPEG video stream VS is obtained. And an audio signal based on the MPEG audio stream AS can be reproduced.

  Therefore, the recording apparatus 200B according to this embodiment also has the same effect as the first and second embodiments, that is, the effect of improving the reproduction efficiency of the two-dimensional video and the three-dimensional video recorded on the recording medium 9. be able to.

  In particular, in the present embodiment, the depth data stored in the DVD-others zone 21 has the same data structure as the video data stored in the DVD-video zone 20, and frames corresponding to the DVOB 86, DCELL 87, and DVOB 88 of the DVD-others zone 21 are used. By setting the number of images (long playback time) equal to the number of frames (long playback time) corresponding to the VOB 26, CELL 27, and VOB 28 of the DVD-video zone 20, accessibility to data recorded on the recording medium 9 such as search Can be increased.

  In this way, the depth information of each pixel of each frame image corresponding to the 2D video is stored in one of the DVD-video zone 20 and the DVD-others zone 21 in a format compliant with the DVD video standard. Thus, 2D video and 3D video can be recorded on the recording medium 9 in conformity with the DVD video standard.

(Fourth embodiment)
FIG. 18 is a block diagram showing a schematic configuration of a recording apparatus 200C according to the fourth embodiment of the present invention.

  The recording apparatus 200C according to the present embodiment is based on a computer graphics (CG) technique, for example, by converting new 2D image data or completely new 2D image data from image data corresponding to an existing 2D image into a frame. The computer 91 includes a two-dimensional image generation function (module) 91a generated in units and an audio signal generation function (module) 91b that generates an audio signal by signal processing. The computer 91 includes a parallax vector extractor 3, The video compressor 2 and the audio compressor 7 are connected respectively.

  The computer 91 includes a memory (not shown), and the two-dimensional image generation function 91a and the audio signal generation function 91b are activated by a program (software) stored in the memory.

  The two-dimensional image data (CG image data) generated based on the CG technique by the two-dimensional image generation function 91a has position information and depth information for each small size unit (for example, polygon).

  That is, according to the present embodiment, the two-dimensional image generation function 91a of the computer 91 sequentially generates two-dimensional image data in units of frames by the CG technique, and transmits the data to the video compressor 2 as video data (video stream). At the same time, position information {horizontal position (X), vertical position (Y), depth information (Z)} for each polygon unit of each two-dimensional image data is transmitted to the depth information formatter 5.

  The audio signal generation function 91b processes the audio signal as the original material or generates a completely new audio signal and transmits it to the audio compressor 7.

  The processing of the video compressor 2, the depth information formatter 5, the audio compressor 7, the information multiplexer 8, and the recorder 10 is substantially the same as the corresponding constituent elements of the recording apparatus 200 of the first embodiment. Therefore, the description thereof is omitted.

  That is, according to the present embodiment, position information for each polygon unit of each 2D image data generated by the 2D image generation function 91a of the computer 91 {horizontal position (X), vertical position (Y), depth information ( Z)} are sequentially transmitted to the depth information formatter 5A.

  At this time, in the depth information formatter 5A, the depth information value of each polygon is set in accordance with the comparison result of the position information of each polygon and the position information of each small block {(4 × 2) pixel range}. The depth information value of each small block is converted into, for example, 8-bit depth data for each pixel constituting each converted small block, as in the first embodiment.

  Other processes are the same as those in the first embodiment.

  That is, according to the present embodiment, it is possible to generate a two-dimensional image and its depth information without using an imaging camera. The generated depth information is integrated with an MPEG video stream and multiplexed with an MPEG audio stream. The multiplexed transport stream TS can be recorded on the recording medium 9 compliant with the DVD video format.

  For this reason, it is possible to obtain the effect of improving the reproduction efficiency of the 2D video and the 3D video recorded on the recording medium 9, which is the same effect as the first embodiment.

(Fifth embodiment)
FIG. 19 is a block diagram showing a schematic configuration of a recording apparatus 200D according to the fifth embodiment of the present invention.

  The recording apparatus 200D in the present embodiment includes a computer 91 and a depth information formatter 5A having the same functional configuration and connection relationship as those in the fourth embodiment. Note that the processing of the video compressor 2A, the audio compressor 7, the information multiplexer 8A, and the recorder 10 is substantially the same as the corresponding components of the recording apparatus 200A of the second embodiment, and therefore the description thereof will be given. Is omitted.

  That is, also in the present embodiment, as in the fourth embodiment, the position information for each polygon unit of each two-dimensional image data generated by the two-dimensional image generation function 91a of the computer 91 is sequentially obtained from the depth information formatter 5A. Sent to.

  At this time, in the depth information formatter 5A, the depth information value of each polygon is set in accordance with the comparison result of the position information of each polygon and the position information of each small block {(4 × 2) pixel range}. The depth information of each small block is converted into a value, and is developed as, for example, 8-bit depth data for each pixel constituting each converted small block.

  Other processes are the same as those in the second embodiment.

  That is, according to the present embodiment, it is possible to generate a two-dimensional image and its depth information without using an imaging camera. Then, the generated depth information is stored in each pixel layer in each frame layer 33 corresponding to the DVD video format by multiplexing processing according to the DVD video format, and the MPEG multiplexed transport in which the depth information is integrated. A stream TS can be generated, and the generated MPEG multiplexed transport stream TS can be recorded on a recording medium 9 compliant with the DVD video format.

  For this reason, it is possible to obtain the same effect as the second embodiment, that is, the improvement of the reproduction efficiency of the two-dimensional video and the three-dimensional video recorded on the recording medium 9.

(Sixth embodiment)
FIG. 20 is a block diagram showing a schematic configuration of a recording apparatus 200E according to the sixth embodiment of the present invention.

  The recording apparatus 200E in the present embodiment includes a computer 91 and a depth information formatter 5A having the same functional configuration and connection relationship as those in the fourth embodiment. Note that the processing of the video compressor 2A, the audio compressor 7, the information multiplexer 8B, and the recorder 10 is substantially the same as the corresponding components of the recording apparatus 200B of the third embodiment, and therefore the description thereof is omitted. Is omitted.

  That is, also in the present embodiment, as in the fourth embodiment, the position information for each polygon unit of each two-dimensional image data generated by the two-dimensional image generation function 91a of the computer 91 is sequentially obtained from the depth information formatter 5A. Sent to.

  At this time, in the depth information formatter 5A, the depth information value of each polygon is set in accordance with the comparison result of the position information of each polygon and the position information of each small block {(4 × 2) pixel range}. The depth information of each small block is converted into a value, and is developed as, for example, 8-bit depth data for each pixel constituting each converted small block.

  Other processes are the same as those in the third embodiment.

  That is, according to the present embodiment, it is possible to generate a two-dimensional image and its depth information without using an imaging camera. Next, a frame layer 33 is generated from the generated depth information, and a plurality of frame layers 33 corresponding to the generated plurality of frame images are grouped to generate a DVOBU 88 in the DVD video format, thereby converting the depth information into the DVD video format. Then, the MPEG multiplexed transport stream TS in which the depth information of each pixel is multiplexed can be recorded on the recording medium 9 conforming to the DVD video format.

  Therefore, also in the recording apparatus 200E according to the present embodiment, it is possible to obtain the same effect as the third embodiment, that is, the improvement of the reproduction efficiency of the 2D video and the 3D video recorded on the recording medium 9. .

(Seventh embodiment)
FIG. 21 is a block diagram showing a schematic configuration of the playback apparatus 100 according to the seventh embodiment of the present invention. This playback apparatus 100 includes an MPEG multiplexed transport stream recorded on any one of the recording apparatuses 200, 200A to 200E according to the first to sixth embodiments of the present invention and the modifications thereof. This is an apparatus capable of switching and reproducing 2D video and 3D video based on TS.

  That is, the playback apparatus 100 accesses the recording medium 9 and can read and play back data recorded on the recording medium 9, and information separation for packet separation connected to the playback apparatus 101. For example, a video decoder 103 having the configuration of the decoding apparatus shown in FIG. 6 and depth information for extracting depth data connected to the video decoder 103. And a take-out device 104.

  Furthermore, the playback apparatus 100 includes a stereoscopic image display 105 that can display a stereoscopic video (three-dimensional moving image) by a predetermined stereoscopic display method, and a stereoscopic image (parallax image) corresponding to the stereoscopic display method of the stereoscopic image display 105 ) To generate a visual field converter 106.

  The playback apparatus 100 includes an audio decoder 107 for decoding audio data connected to the information separator 102, and a speaker 108 for playing back the audio signal decoded by the audio decoder 107. Yes.

  The playback apparatus 100 is connected to a playback apparatus 101, an information separator 102, a video decoder 103, a depth information extractor 104, a stereoscopic image display 105, a field of view converter 106, and an audio decoder 107, respectively. A controller 109 for controlling the whole is provided.

  Next, the overall operation of the playback apparatus 100 according to this embodiment will be described.

  According to the reproducing apparatus 100, the MPEG multiplexed transport stream TS recorded on the recording medium 9 by the recording method described in the first embodiment is read from the recording medium 9 by the reproducing device 101 and read by the reproducing device 101. The MPEG multiplexed transport stream TS is sent to the information separator 102, where the MPEG video stream VS packet and the MPEG audio stream AS packet are separated from the MPEG multiplexed stream TS. Is done.

  The packet of the MPEG video stream VS separated by the information separator 102 is received by the video decoder 103, and the received video packet of the MPEG video stream VS is received by the video decoder 103 by the method shown in FIG. Is decoded to generate decoded data (reproduction frame image).

  Further, in the video decoder 103, user data stored in the user data area UD1 or UD2 of the MPEG video stream VS is read out, and from this read out user data, the depth information extractor 104, for example, FIG. Depth data recorded in the frame layer format of FIG. 17 is extracted. When the user data is stored in the D_PACK 32, the private data field, or the private stream field in the MPEG multiplexed transport stream TS, the user data is read by the information separator 102 and the depth information extractor 104 is read out. Depth data is extracted by.

  The playback frame image generated by the video decoder 103 and the depth data extracted by the depth information extractor 104 are respectively received by the visual field converter 106.

  In the field-of-view converter 106, a parallax image corresponding to the stereoscopic display method of the stereoscopic image display 105 is generated and generated based on the received playback frame image and the depth data extracted by the depth information extractor 104. The displayed parallax image is stereoscopically displayed by the stereoscopic image display 105.

  On the other hand, the MPEG audio stream AS packet separated by the information separator 102 is decoded by the audio decoder 107 to reproduce decoded data (audio signal), and the generated audio signal is reproduced by the speaker 108. Is done.

  Next, the parallax image generation processing of the visual field converter 106 in this embodiment will be described.

  In order to generate a parallax image, a visual field conversion method is used as a coordinate system conversion method in CG. This visual field conversion method can obtain an image with a different viewpoint by a conversion formula to the viewpoint coordinate system. If a two-dimensional image and its depth information can be obtained, the depth information can be freely used. An image (stereoscopic image) viewed from various viewpoints can be generated.

For example, the coordinates of the viewpoint corresponding to the principal point on the optical axis of the lens L1A or L1B of the imaging camera 1B shown in FIG. 1 (x _i , y _i , z _i ) are used as the feature points of the target object OB. Let the coordinates of the corresponding gaze point be (x _a , y _a , z _a ). Further, when the distance between the viewpoint and the gazing point is (x _f , y _f , z _f ), the distance between the viewpoint and the gazing point (x _f , y _f , z _f ) is expressed by the following equation: x _f = x _i − x _a
y _f = y _i −y _{a
z f} = z i -z _a
Represented as:

In this case, in view transformation method according to the present embodiment sets the first fixation point O _a by moving the position of the origin by translation to the origin of the orthogonal coordinate system (x, y, z). This conversion to _{T 1.} This transformation T ₁ is a transformation that represents a parallel movement of only (−x _a , −y _a , −z _a ). Next, the direction of the coordinate value is changed by rotation. As shown in FIG. 22, an orthogonal coordinate system (x, y, z) vector from the origin (gazing point) O _a to the point O _f direction, only first α angular vector of the z-axis from the origin O _a y-axis Is then rotated around the x axis by the β angle. In practice, the rotational direction is reversed so move the coordinates of the point O _f.

として表される。 here,
Converting _T 2:-.alpha. rotational transform to the y-axis _T 3: represented as -β rotating the x-axis. Here, alpha is because an angle between the projection point obtained by projecting the point O _f the xy plane O _{f 'between} a line and Z-axis connecting the origin O _a, sin .alpha and cosα, respectively following formula

Represented as:

として表される。 Moreover, beta is the origin _{O a} and point _O length between ^{_{f (x f 2 + y f}} 2) 1/2 and the point _{O f} and projection point _{O f} 'between length _{y f,} the following formula

Represented as:

として表される。 As a final conversion, the positive direction of the z-axis with respect to xy plane such that the viewpoint side from the origin O _a (x, y, z) coordinate system (see FIG. 22), forward the origin of the z-axis with respect to the xy plane O opposite direction side perspective and from _a, i.e., transformation T ₄ to allow the z-axis direction across the xy plane (the direction of the eye from the viewpoint) is positive through the origin O _a from the perspective I do. This is simply z → -z. Multiplying these four transformation matrices T _{1 to} T ₄ gives the viewpoint coordinate transformation matrix.

Represented as:

  For example, if the stereoscopic display method of the stereoscopic image display device 105 is a twin-lens stereoscopic display method using a parallax barrier described later, the visual field converter 106 is based on the depth data extracted by the depth information extractor 104. By substituting α that can be set into an expression representing the conversion matrix T and setting β and γ to 0, it is possible to generate a right eye image and a left eye image having parallax.

  If the stereoscopic display method of the stereoscopic image display 105 is a method using IP (Integral Photography), which will be described later, the visual field converter 106 includes a plurality of lens arrays. Based on a plurality of element images obtained by imaging a target object with an imaging camera corresponding to each lens position constituting, using the conversion matrix T of the viewpoint coordinates together with the size of each element image Thus, stereoscopic image data can be generated. It is fair to transmit the stereoscopic image data generated in this way to the stereoscopic image display 105 and perform stereoscopic image reproduction.

  Here, a typical parallax barrier method and an IP method among the stereoscopic image display methods will be described. The parallax barrier method can be realized by liquid crystal, for example. That is, the parallax barrier method is configured by laminating two liquid crystal panels, and as shown in FIG. 23, one liquid crystal panel (liquid crystal light-shielding panel) in which narrow slit-shaped openings 110a are formed. The barrier 110 is opposed to the back side of the predetermined viewpoint (left eye, right eye) at an appropriate interval, and the left eye image and right eye image (L and L) are displayed on the screen on the liquid crystal light shielding barrier side. R) is a system comprising the other liquid crystal panel 112 in which the backlight 111 is disposed along the longitudinal direction on the opposite side of the openings alternately arranged in the same cycle as the opening.

  According to this parallax barrier method, when the left eye image and the right eye image are viewed through the opening of the liquid crystal light blocking barrier 110 from a predetermined viewpoint of the user, the right eye image is displayed for the right eye and the left eye is displayed for the left eye. It is configured so that the eye images can be perceived in a separated state. With this configuration, the user can change the imaging position of the composite image from the right eye / left eye image position (screen position) based on the different left eye image and right eye image recognized by the right eye and the left eye, respectively. It can be perceived as a stereoscopic image.

  However, although the focus of the eye is always on the screen of the liquid crystal panel 112, the imaging position is perceived at a position different from the screen, which may be accompanied by physiological unnaturalness. There was also the possibility of fatigue and video sickness. Therefore, in recent years, there are five physiological factors of stereoscopic vision, namely, convergence adjustment contradiction (conflict between the convergence point and the focus position), binocular parallax (when viewing an object, the left and right eyes of a human are different. That captures two different images viewed from different directions), focus adjustment (the property of changing the lens thickness by controlling the lens thickness as the distance from the viewing object changes), and convergence (perspective Satisfying the nature of the eyeball rotating inward or rotating outward due to changes) and motion parallax (property of seeing the difference of images by the user moving or changing the viewing angle) Such a stereoscopic image display method has also been proposed.

  As a promising method among the proposed ones, a method announced by Lippmann in 1908 is the IP method. The IP method acquires depth information of a display target object by using a two-dimensionally arranged lens array (also referred to as a fly-eye lens, a fly-eye lens, or a compound eye lens). In the 1990s, a technology for generating moving images by IP was developed by replacing the conventional photographic plate recording with electronic technology. Furthermore, the literature (NHK Broadcasting Technology Laboratory “Basics of 3D Video”) was developed. A researcher's hand captures a display object (subject) using a gradient index lens array {also referred to as a GRIN (Gradient Index) lens array} and a high-definition camera, and acquires an element image group corresponding to the lens array. Each elemental image is transmitted and displayed in real time on a liquid crystal display, and is successfully imaged as a 3D image (3D video) in space by a fly-eye lens, enabling the realization of 3D television broadcasting by the IP method. Sex was shown.

  FIG. 24 is a diagram for explaining the principle of imaging an IP-type stereoscopic image. As shown in FIG. 24A, a large number of minute lens elements 121 are arranged at the time of photographing to constitute a GRIN lens array as an IP lens array 122. The image of the imaging target object OB1 imaged for each element lens 121 is imaged as an element image on the high-vision camera 124 via the condenser lens 123, and is collectively photographed by this camera 124. At the time of reproduction, as shown in FIG. 24B, the image from the camera is reproduced by a display, for example, a liquid crystal display (LCD) 125, and the light of all the element lenses 121 is reproduced. Collected at one point, a reproduced image viewed from one direction as a whole is created.

  It is possible to create motion parallax in the horizontal and vertical directions by arranging minute element lenses two-dimensionally. If they are arranged in the horizontal direction, it is possible to have parallax only in the horizontal direction.

  In this embodiment, in order to generate and display a parallax image corresponding to the IP method, the field of view converter 106 extracts a plurality of element images viewed via a plurality of element lenses by the depth information extractor 104. By creating each of the above-described viewpoint conversions based on the data and arranging the created element images, the liquid crystal display in the stereoscopic image display 105 is displayed as if it was captured with the configuration of FIG. By displaying the element image array, stereoscopic reproduction can be realized. As shown in FIG. 25, the stereoscopic image display 105 that actually corresponds to the IP system includes a liquid crystal display 125 and an IP lens array (for example, a GRIN lens array) 122 composed of a plurality of element lenses 121. The liquid crystal display 125 and the lens array 121 are arranged close to each other and face each other. That is, the stereoscopic image display 105 is designed as a thin display having a configuration that is substantially the same as the configuration for displaying a two-dimensional video.

  On the other hand, in accordance with the control of the controller 109, the visual field converter 106 of the playback device 105 may not perform the visual field conversion process using the depth data. In this case, the stereoscopic image display device 105 controls the controller 109. Accordingly, the playback frame image (two-dimensional image) sent from the video decoder 103 is sequentially displayed for each frame.

  As described above, according to the playback apparatus 100 according to the present embodiment, two-dimensional video and three-dimensional video can be switched and played by switching use / non-use of depth information recorded on the recording medium 9, The reproduction efficiency of 2D video and 3D video recorded on the recording medium 9 can be improved.

  Further, in the present embodiment, since the data for stereoscopic viewing is created from the depth information, it is possible to prevent only one-eye image from being deteriorated as in the past and improve the satisfaction of the user who performs stereoscopic viewing. Can do.

  In the first to sixth embodiments and modifications thereof, the recorder 10 records the MPEG multiplexed transport stream TS in which the depth information is multiplexed on the recording medium 9, but the present invention has this configuration. It is not limited to. For example, as shown in FIG. 26, a recorder 10A in a recording apparatus 200F according to this modification uses an MPEG transport stream TS transmitted from the information multiplexer 8 as a communication (broadcast) packetizer 130. The stream may be packetized according to a communication or broadcast format via the communication / broadcast format, and the stream packetized in the communication / broadcast format may be transmitted via the communication network (or broadcast network) 131 corresponding to the format.

  In the seventh embodiment, the playback device 100 reads the MPEG multiplexed transport stream TS recorded on the recording medium 9 by the playback device 101 from the recording medium 9 and performs playback processing. The configuration is not limited. For example, as shown in FIG. 27, the playback device 100A according to the present modification receives a stream packetized in a communication / broadcasting format transmitted via a communication network (or broadcast network) 131. Release the packet. Then, the playback device 100A sends the released data to the information separator 102, and the information separator 102 transmits the MPEG video stream VS packet and the MPEG audio stream AS packet from the MPEG multiplexed transport stream TS, respectively. It is also possible to execute the reproduction processing described above (see FIGS. 21 to 25).

  According to this modified example, the MPEG multiplexed transport stream TS is not read from the recording medium 9, but the stream transmitted via the communication network (broadcasting network) 131 is read and the read stream is converted into the read stream. Corresponding two-dimensional or three-dimensional video can be reproduced.

(Eighth embodiment)
FIG. 28A is a diagram showing at least one computer with a built-in memory (including a general-purpose computer and a microcomputer) 150 in which a recording program according to the eighth embodiment of the present invention is installed.

  As shown in FIG. 28A, the computer 150 is connected to the pair of imaging cameras 1A and 1B, the microphone 6, and the recording medium 9 shown in FIG. The processing procedure of the computer shown in FIG. 28B is the hardware block component (video compressor 2, disparity vector extractor 3, depth information calculator 4, depth information formatter 5 and audio compression shown in FIG. Since it corresponds to the processing functions of the device 7, the information multiplexer 8, and the recorder 10), the flow of processing will be briefly described.

  That is, as shown in FIG. 28 (b), the computer 150, according to the recording program recorded in the memory 150a, captures a target object OB for a predetermined time captured at the same timing by the pair of left and right imaging cameras 1A and 1B. The left eye image Pl and the right eye image Pr are input and stored in the memory 150a (step S110).

  Next, the computer 150 executes a process corresponding to the disparity vector extractor 3 shown in FIG. 1, and based on the epipolar line direction and the direction orthogonal to the epipolar line direction, as described in the first embodiment. Disparity vectors V for all corresponding points in all macroblocks within the set search range are extracted (step S120).

  Subsequently, the computer 150 executes the process corresponding to the depth information calculator 4 illustrated in FIG. 1, thereby determining the size of each disparity vector extracted by the process of step S <b> 120 as described in the first embodiment. The position information {horizontal position (X), vertical position (Y), depth information (Z)} of each parallax vector calculated in small blocks is generated (step S130).

  Then, the computer 150 executes processing corresponding to the depth information formatter 5 shown in FIG. 1 to format the depth information value of each small block as depth data of each pixel (step S140).

  Next, the computer 150 executes processing corresponding to the video compressor 2 shown in FIG. 1 to generate, for example, the MPEG video stream VS by MPEG compression encoding the right eye image Pr, and obtain the processing by the processing of step S140. The 8-bit depth data of each pixel obtained is based on any one of the data formats (1) to (4) described in the first embodiment, and the user data areas UD1 / UD2 in the MPEG video stream VS. Stored in any of D_PACK32, private data field, or private stream field in the MPEG multiplexed transport stream TS (step S150).

  In parallel with the processing of steps S110 to S150, the computer 150 executes processing corresponding to the audio compressor 7 shown in FIG. 1 to MPEG-encode the audio signal collected by the microphone 6 to MPEG-encoded the audio stream. AS is generated (step S160).

  Subsequently, the computer 150 multiplexes the MPEG video stream VS and the MPEG audio stream AS according to the DVD video format shown in FIG. 4 by executing processing corresponding to the information multiplexer 8 shown in FIG. 1 (step S170). ).

  Then, the computer 150 records the data stream (MPEG multiplexed transport stream TS) generated by multiplexing on the recording medium 9 in a predetermined unit (for example, 2 kB unit corresponding to the DVD video standard), or If the computer 150 is connected to a communication network or broadcast network having a predetermined communication format, the MPEG multiplexed transport stream TS generated by multiplexing is converted into a packet corresponding to the predetermined communication format for communication. Transmission is performed via a network or a broadcast network (step S180).

  Subsequently, the computer 150 determines whether image data (frame image data) is input from the imaging cameras 1A and / or 1B (step S190). If the result of this determination is YES, step 150 is performed. Returning to the processing of S110, the processing after step S110 is continued.

  On the other hand, if the result of determination in step S190 is NO, the computer 150 ends the process.

  That is, according to the recording program according to the present embodiment, the computer 150 stores the depth data of each pixel in the user data area UD1 or UD2 of the MPEG video stream VS, and the MPEG video stream VS is stored in the MPEG audio stream AS. And can be recorded on the recording medium 9. For this reason, similarly to the first embodiment, switching between 2D video and 3D video can be performed by switching use / non-use of depth information recorded on the recording medium 9, and 2D video and 3D video can be switched. The reproduction efficiency can be improved.

(Ninth embodiment)
FIG. 29A is a diagram showing at least one memory built-in computer (including a general-purpose computer and a microcomputer) 160 on which a reproduction program according to the ninth embodiment of the present invention is installed.

  As shown in FIG. 29A, the computer 160 is connected to the recording medium 9, the stereoscopic image display 105, and the speaker 108 shown in FIG. The processing procedure of the computer shown in FIG. 29 (b) is the same as the hardware block components (reproducer 101, information separator 102, video decoder 103, depth information extractor 104, field of view converter 106 shown in FIG. , And the audio decoder 107) correspond to the respective processing functions, and therefore the processing flow will be briefly described.

  That is, as shown in FIG. 29B, the computer 160 reproduces the MPEG multiplexed transport stream TS recorded on the recording medium 9 according to the reproduction program recorded on the memory 160a, or the computer 160 When connected to a communication network or broadcast network having a predetermined communication format, an MPEG multiplexed transport stream TS (formatted in a predetermined communication format) transmitted from the communication network or broadcast network is received. (Step S210).

  Next, the computer 160 executes a process corresponding to the information separator 102 shown in FIG. 21, and as described in the seventh embodiment, the packet of the MPEG video stream VS from the MPEG multiplexed stream TS and the MPEG audio are transmitted. Each packet of the stream AS is separated (step S220). If the stream TS is packetized in a predetermined communication format, the process of step S220 is executed after releasing the packet of this communication format.

  Subsequently, the computer 160 executes processing corresponding to the video decoder 103 shown in FIG. 21, thereby decoding and decoding the packet of the separated MPEG video stream VS as described in the seventh embodiment. Data (reproduction frame image) is generated, and user data stored in the user data area UD1 or UD2 of the MPEG video stream VS is read (step S230).

  When user data is stored in the D_PACK32, the private data field, or the private stream field in the MPEG multiplexed transport stream TS, the computer 160 determines that the MPEG multiplexed transport stream TS in step S220. User data is read out from the corresponding storage area at.

  Then, the computer 160 executes the process corresponding to the depth information extractor 104 shown in FIG. 21, and as described in the seventh embodiment, from the user data read out in the process of step S230, for example, FIG. Depth data for each macroblock recorded in the frame layer format of FIG. 17 is extracted (step S240). Next, the computer 160 generates a parallax image corresponding to the stereoscopic display method of the stereoscopic image display 105 based on the playback frame image and the depth data (step S250), and the generated parallax image is generated by the stereoscopic image display 105. Three-dimensional display is performed (step S260).

  On the other hand, the computer 160 decodes the packet of the MPEG audio stream AS separated by the process of step S220 (step S270), and reproduces the decoded data (audio signal) by the speaker 108 (step S280).

  The computer 160 determines whether image data (frame image data) to be reproduced is still recorded on the recording medium 9 or whether image data is input to the computer 160 from a communication network (broadcast network). If it is determined (step S290) and the result of this determination is YES, processing returns to step S210 and processing from step S210 onward is continued.

  On the other hand, if the result of determination in step S290 is NO, the computer 160 ends the process.

  In other words, according to the playback program according to the present embodiment, switching between 2D video and 3D video can be performed by using / not using depth information recorded on the recording medium 9 as in the sixth embodiment. Thus, the reproduction efficiency of 2D video and 3D video can be improved.

  In the sixth and eighth embodiments, the IP method and the parallax barrier method have been described as the stereoscopic image display method of the stereoscopic image display 105, but the present invention is not limited to this method, and the lenticular lens Any method can be applied as long as it is a method capable of stereoscopic perception such as a method, a super multi-view method, a binocular method using deflection glasses, and an anaglyph method. Further, in the present invention, it is not always necessary to record an MPEG multiplexed transport stream including depth data on a recording medium, as shown in FIG. 26, via various transmission media such as a communication network and a broadcast network. It is possible to transmit an MPEG multiplexed transport stream. Therefore, in this case, the recording device is used as a transmission device. As shown in FIG. 27, the playback device according to the present invention can also be used as a receiving device that receives an MPEG multiplexed transport stream via various transmission media such as a communication network and a broadcasting network. is there.

  Moreover, since the recording media according to the first to eighth embodiments have a medium-specific effect of recording depth information for 3D video display, a plurality of 3D video (3D video) can be reproduced. It is possible to preferably realize a system that enables playback of 3D video in accordance with the method.

  Furthermore, the definition of “medium” in the recording medium according to the present invention represents not only a narrowly-defined medium that can record data, but also a concept including a physical medium such as an electromagnetic wave or light that is a data transmission medium. It is. In addition, the information recorded on the recording medium includes data itself such as an electronic file when not recorded.

  In the above description, the depth information of the video data is recorded for each frame image. However, in the present invention, the depth information may be recorded about every 0.5 second or about every 1 second. In that case, it can be realized by using user data recorded in a user data area prepared in the MPEG GOP layer.

  In each of the above embodiments, the video data and the audio data are respectively compressed and encoded and multiplexed. However, the video data may be multiplexed mainly without using the audio data. Of course, not only audio data and video data, but also data such as other sub-pictures and control information may be multiplexed.

  Further, in each of the above embodiments, the depth data is obtained for each pixel of the left eye image, for example, but the present invention is not limited to this configuration, and the depth data may be obtained for each unit other than the pixel.

  Note that the program installed in the computer in the seventh and eighth embodiments may be installed in the computer from various recording media such as CD-ROM and DVD-ROM accessible by the computer. Alternatively, the program transmitted via the communication network may be installed in the computer.

1 is a block diagram showing a schematic configuration of a recording apparatus according to a first embodiment of the invention. It is a figure which shows the arrangement | positioning relationship of the recording device with respect to the object which is a video recording target of the recording device of 1st Embodiment in this invention. (A) is a perspective view which shows each of the right eye image, left eye image, target object, and epipolar line which concern on the 1st Embodiment of this invention, (b) is shown in Fig.3 (a). It is a figure which shows the search range set on the right-eye image and left-eye image to be set. The figure which shows the outline of a DVD video format. It is a block diagram which shows an example of the encoding apparatus for performing the compression encoding process by MPEG. FIG. 6 is a block diagram illustrating a decoding apparatus that decodes and reproduces video data that has been compression-encoded by the encoding apparatus illustrated in FIG. 5. FIG. 6 is a diagram showing a schematic configuration of a hierarchical structure of an MPEG video stream VS generated by the encoding device shown in FIG. 5. (A) is a figure which shows the syntax of the sequence layer shown in FIG. 7, (b) is a figure which shows the syntax of a sequence header. It is a figure which shows the syntax of a GOP layer. It is a figure which shows the syntax of a picture layer. It is a figure which shows the syntax of a slice layer. It is a figure which shows the syntax of a Macroblock layer. It is a figure which shows the syntax of a block layer. It is a block diagram which shows schematic structure of the recording device which concerns on the 2nd Embodiment of this invention. It is a figure which shows the syntax of the system layer of an MPEG multiplexed transport stream. It is a block diagram which shows schematic structure of the recording device which concerns on the 3rd Embodiment of this invention. It is a figure which shows the outline of the DVD video format based on the 3rd Embodiment of this invention. It is a block diagram which shows schematic structure of the recording device which concerns on the 4th Embodiment of this invention. It is a block diagram which shows schematic structure of the recording device which concerns on the 5th Embodiment of this invention. It is a block diagram which shows schematic structure of the recording device which concerns on the 6th Embodiment of this invention. It is a block diagram which shows schematic structure of the reproducing | regenerating apparatus based on the 7th Embodiment of this invention. It is a figure which shows the coordinate system for demonstrating the visual field conversion system in the 7th Embodiment of this invention. It is a figure for demonstrating schematically the parallax barrier system which is an example of the three-dimensional display system concerning the 7th Embodiment of this invention. (A) is a figure for demonstrating schematically the IP system at the time of imaging | photography which is an example of the three-dimensional display system which concerns on the 7th Embodiment of this invention, (b) is an IP system at the time of a display. It is a figure for demonstrating schematically. It is a figure which shows schematic structure of the three-dimensional image display shown in FIG. 21 which applied the IP system as a three-dimensional display system based on the 7th Embodiment of this invention. It is a block diagram which shows schematic structure of the recording device which concerns on the modification of 1st-6th embodiment of this invention. It is a block diagram which shows schematic structure of the reproducing | regenerating apparatus based on the modification of the 7th Embodiment of this invention. (A) is a figure which shows schematic structure of the system containing the computer with a built-in memory in which the recording program based on the 8th Embodiment of this invention was installed, (b) is a figure in Fig.28 (a). It is a flowchart which shows the process sequence of the computer shown schematically. (A) is a figure which shows schematic structure of the system containing the computer with a built-in memory in which the program for reproduction | regeneration which concerns on the 9th Embodiment of this invention was installed, (b) is a figure which shows (a) in FIG. It is a flowchart which shows the process sequence of the computer shown schematically.

Explanation of symbols

1A, 1B Imaging camera 2, 2A Video compressor 3 Parallax vector extractor 4 Depth information calculator 5, 5A Depth information formatter 6 Microphone 7 Audio compressor 8, 8A, 8B Information multiplexer 9 Recording medium 10 Recorder 11 Control unit 91 Computer 100, 100A Reproducing apparatus 101 Reproducing unit 102 Information separating unit 103 Video decoder 104 Depth information extracting unit 105 Stereo image display unit 106 Field of view converter 107 Audio decoder 108 Speaker 109 Controller 130 Communication (broadcasting) packet Generator 131 Communication (broadcast) network 140 Communication (broadcast) packet release unit 150, 160 Computer 150a, 160a Memory 200, 200A-200F Recording device

Claims (6)

A first bit stream recorded on a recording medium formatted in accordance with a predetermined standard and formed by compressing and encoding a two-dimensional image using a compression encoding format conforming to the format of the recording medium; A playback device for playing back a multiplexed stream obtained by multiplexing a second bit stream formed by formatting information relating to the depth of the two-dimensional image obtained for each predetermined unit,
Means for separating the first bit stream from the multiplexed stream and reproducing the two-dimensional image;
Means for separating the second bitstream from the multiplexed stream and reproducing information relating to the depth of the two-dimensional image;
Means for generating a three-dimensional image based on the reproduced two-dimensional image and information relating to the depth of the two-dimensional image;
A playback apparatus comprising: A video stream recorded on a recording medium formatted in accordance with a predetermined standard and formed by compressing and encoding a two-dimensional image according to a compression encoding format conforming to the format of the recording medium, and the compression encoding in the video stream A playback device that plays back a multiplexed stream configured by multiplexing information on the depth of the two-dimensional image stored in an arbitrary use area set in a format,
Means for reproducing the two-dimensional image by separating the bit stream from the multiplexed stream;
Means for reproducing information relating to the depth of the two-dimensional image stored in the arbitrary use area in the separated bitstream;
Means for generating a three-dimensional image based on the reproduced two-dimensional image and information relating to the depth of the two-dimensional image;
A playback apparatus comprising: The two-dimensional image is a plurality of frame images constituting a two-dimensional video, and the depth information is a pixel value for each pixel as the predetermined unit in each frame image constituting the two-dimensional video,
A pixel value representing depth information for each pixel in each frame image constituting the two-dimensional video is a first compression code that is differentially encoded in each frame image by a compression coding format that conforms to the format of the recording medium. Processing, second compression encoding processing for run-length encoding, third encoding for predictive encoding from each frame image and at least one of the future and past frame images on the time axis of each frame image Compressed and encoded by at least one of the compression encoding processing and the fourth compression encoding processing encoded using orthogonal transform in each frame image, and recorded on the recording medium Has been
The means for reproducing the information on the depth is the depth for each pixel of each frame image that has been compression-encoded by at least one of the first to fourth compression-encoding processes. 2. A means for decoding pixel values representing information by a decoding process corresponding to at least one of the compression encoding processes to reproduce information relating to the depth of the two-dimensional image. Or the reproducing apparatus of 2. A first bit stream recorded on a recording medium formatted in accordance with a predetermined standard and formed by compressing and encoding a two-dimensional image using a compression encoding format conforming to the format of the recording medium; A computer-executable playback program for playing back a multiplexed stream obtained by multiplexing a second bit stream formed by formatting information about the depth of the two-dimensional image obtained for each predetermined unit,
In the computer,
Processing to separate the first bitstream from the multiplexed stream and reproduce the two-dimensional image;
Processing to separate the second bitstream from the multiplexed stream and reproduce information about the depth of the two-dimensional image;
Processing for generating a three-dimensional image based on the reproduced two-dimensional image and information on the depth of the two-dimensional image;
A reproduction program characterized in that each of the above is executed. A video stream recorded on a recording medium formatted in accordance with a predetermined standard and formed by compressing and encoding a two-dimensional image according to a compression encoding format conforming to the format of the recording medium, and the compression encoding in the video stream A reproduction program executable by a computer for reproducing a multiplexed stream configured by multiplexing information on the depth of the two-dimensional image stored in an arbitrary use area set in a format,
In the computer,
Processing to separate the bit stream from the multiplexed stream and reproduce the two-dimensional image;
Processing for reproducing information about the depth of the two-dimensional image stored in the arbitrary use area in the separated bitstream;
Processing for generating a three-dimensional image based on the reproduced two-dimensional image and information on the depth of the two-dimensional image;
A reproduction program characterized in that each of the above is executed. The two-dimensional image is a plurality of frame images constituting a two-dimensional video, and the depth information is a pixel value for each pixel as the predetermined unit in each frame image constituting the two-dimensional video,
A pixel value representing depth information for each pixel in each frame image constituting the two-dimensional video is a first compression code that is differentially encoded in each frame image by a compression coding format that conforms to the format of the recording medium. Processing, second compression encoding processing for run-length encoding, third encoding for predictive encoding from each frame image and at least one of the future and past frame images on the time axis of each frame image Compressed and encoded by at least one of the compression encoding processing and the fourth compression encoding processing encoded using orthogonal transform in each frame image, and recorded on the recording medium Has been
The process of reproducing the information about the depth is the depth for each pixel of each frame image that has been compression-encoded by at least one of the first to fourth compression-encoding processes. 5. The method according to claim 4, further comprising: a process of decoding a pixel value representing information by a decoding process corresponding to at least one of the compression encoding processes to reproduce information about the depth of the two-dimensional image. 5. The reproduction program according to 5.

Images