JP2006128816A - Recording program and reproducing program corresponding to stereoscopic video and stereoscopic audio, recording apparatus and reproducing apparatus, and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a recording/reproducing technique corresponding to stereoscopic video and stereoscopic audio capable of performing corresponding recording and reproducing of the contents of stereoscopic video and stereoscopic audio. <P>SOLUTION: In this recording technique corresponding to stereoscopic video and stereoscopic audio, stereoscopic video data and stereoscopic audio data are recorded in a recording medium, stereoscopic positional information of an object in a video image is created per time, together with stereoscopic video information, and the created information is recorded in a predetermined area capable of being referred to, when the stereoscopic video data and the stereoscopic audio data are reproduced in this recording medium. Also, in this reproducing technique corresponding to stereoscopic video and stereoscopic audio, the stereoscopic positional information of the object in the video image is detected from the stereoscopic video data, read from the recording medium or the received stereoscopic video data, and spatial localization of spatial audio is performed on the basis of the positional information. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a stereoscopic video / stereo acoustic correspondence recording program, a reproducing program, a recording apparatus, a reproducing apparatus, and a recording medium for recording and reproducing stereoscopic object motion and stereoscopic sound in association with each other.

  Conventionally, 3D (stereoscopic) video reproduction techniques have been proposed in several ways. Further, as a technique related to a 3D (stereoscopic) sound field, a technique has been proposed in which a sound is localized and reproduced as if a speaker is located at that position from a place without a speaker. However, there is a problem that it is not possible to reproduce the stereoscopic video of the content that can be displayed and reproduced by the stereoscopic video technology that can express the space and the sound by the spatial localization technology of the spatial sound in cooperation.

  For example, in Japanese Patent Laid-Open No. 2000-17355 (Patent Document 1), the present inventor discloses a phase in both directions with respect to a direction of an epipolar line connecting between corresponding pixel points of a stereo image related to a target object and a direction orthogonal thereto. A method of calculating a distance to a target object by performing a two-dimensional search on the screen so as to include both corresponding pixel points and calculating a disparity vector is disclosed.

  In Japanese Patent Laid-Open No. 7-236199 (Patent Document 2), the movements of left and right objects in a stereoscopic video are automatically detected by detecting a motion vector, and the sound fields corresponding to the motions are a main audio signal and a sub audio signal. A method of reproducing three-dimensionally with a main speaker and a surround speaker driven by the above is disclosed. However, the motion vector is the motion of the object in the time direction of the content and is not information on the depth of the 3D stereoscopic video. In this prior art, the three-dimensional sound field is also performed by simply using the surround feeling of several speakers, and does not perform localization of sound in space.

  Japanese Patent Laid-Open No. 2001-306081 (Patent Document 3) discloses a technique related to a music space configuration control device that controls the configuration of an audio space in real time, and parameters that describe a three-dimensional sound source existing in DirectX. To provide an effective music space using localization, direction, Doppler parameters, and the like. However, a parameter relating to depth, which is a space of stereoscopic video, is not disclosed, and synchronous playback of the spatial video and sound is not described.

As described above, conventionally, in response to the movement in the depth direction of a three-dimensional stereoscopic image, the sound and the sound are coordinated to reproduce the image and the sound, or even if the image exceeds the display display area. There is no known system that reproduces the sound of the content using sound localization technology and reproduces realistic video and audio.
JP 2000-17355 A JP 7-236199 A JP 2001-306081 A NHK Broadcasting Technology Laboratory, “Basics of 3D Video”, Ohm, 1995 By Jens Brauert, “Spatial Acoustics”, Kashima Press, 1985 B. Javidi, F.A. Okano Editors, “Three-Dimensional Television, Video, and Display Technologies”, Springer-Verlag (2002), P101-P123.

  The present invention has been made in view of the above-described technical problems of the prior art, and an object of the present invention is to provide a stereoscopic video / stereoacoustic recording technique capable of corresponding recording of stereoscopic video and stereoscopic audio content. .

  The present invention also provides a stereoscopic video / audio that can be reproduced in synchronization with the three-dimensional position of the object in the video and the three-dimensional localization of the sound when reproducing the stereoscopic video data and the audio data recorded on the recording medium. It aims at providing the reproduction | regeneration technology corresponding to a three-dimensional sound.

  According to the first aspect of the present invention, there is provided a recording program for stereoscopic video / stereo acoustic recording of a data stream of both stereoscopic video data and audio data on a recording medium, and an audio having itself as a sound source during reproduction in a predetermined time unit. Recording the stereo position information of the object to be subjected to the stereo localization control in the computer together with the identification information of the object in a predetermined storage area that can be referred to when reproducing the stereo video data and the audio data on the recording medium. To be executed.

  The reproduction program for stereoscopic video / stereo sound according to claim 2 reads out and reproduces stereoscopic video data and audio data recorded on a recording medium, and stereo localization of a sound source from a predetermined storage area of the recording medium. The step of reading the identification information and the stereoscopic position information of the object to be controlled, and the reproduction of the acoustic data, based on the stereoscopic position information corresponding to the identification information of the object, the stereo localization position of the sound image of the acoustic data And a step of performing sound image position control using at least two or more speakers so that the three-dimensional position is obtained.

  According to a third aspect of the present invention, there is provided a stereoscopic video / stereo sound compatible recording apparatus, a unit for recording a data stream of both stereoscopic video data and audio data on a recording medium, and an audio having itself as a sound source during reproduction in a predetermined time unit. And means for recording the stereoscopic position information of the object to be subjected to the stereo localization control together with the identification information of the object in a predetermined storage area that can be referred to when reproducing the stereoscopic video data and the audio data on the recording medium. Is.

  According to a fourth aspect of the present invention, there is provided a stereoscopic video / stereo sound-compatible playback device that reads and plays back stereoscopic video data and audio data recorded on a recording medium, and stereo localization of a sound source from a predetermined storage area of the recording medium. Means for reading the identification information and stereoscopic position information of the object to be controlled, and upon reproduction of the acoustic data, based on the stereoscopic position information corresponding to the identification information of the object, the stereo localization position of the sound image of the acoustic data is determined as the object Means for performing sound image position control using at least two or more speakers so that the three-dimensional position is obtained.

  According to the fifth aspect of the present invention, there is provided a recording medium for stereoscopic video / stereo acoustic recording both the stereoscopic video data and the audio data, and the stereoscopic position information of the object together with the identification information of the object in a predetermined time unit. It is recorded in a predetermined storage area that can be referred to when reproducing the stereoscopic video data and the sound data on the recording medium.

  In the recording technology for stereoscopic video / stereo sound of the present invention, stereoscopic video data and acoustic data are recorded on a recording medium, and stereoscopic position information of an object in the video is created in a predetermined time unit together with stereoscopic video information. Thus, it can be recorded in a predetermined storage area that can be referred to when reproducing the stereoscopic video data and the audio data on the recording medium.

  In the 3D image / stereo sound compatible playback technology of the present invention, the 3D position information of an object in the image is detected from the 3D image data read from the recording medium or the received 3D image data, and based on the position information. Spatial localization of spatial acoustics can be performed accurately.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  (First Embodiment) FIG. 1 is a block diagram of a stereoscopic video / stereo acoustic recording system according to one embodiment of the present invention, and FIG. 2 shows a photographing mode of a target object using the same. . The stereoscopic video / stereoacoustic recording system of the present embodiment includes a pair of left and right imaging cameras A1, B2, parallax vector extractor 3, depth distance calculator 4, object analyzer 5, as a stereoscopic video information recording system. A horizontal position analyzer 6, a vertical position analyzer 7, a depth direction position analyzer 8, a position information formatter 9, and a video compressor 14 are provided. This system also includes a plurality of microphone groups 11, a sound source selector 12, and an audio compressor 13 as an acoustic information recording system. The system also includes a CPU 10, an information multiplexer 15, and a recorder 16 as elements common to both systems.

  A stereoscopic video / stereo sound compatible recording operation by the stereoscopic video / stereo sound compatible recording system configured as described above will be described. The video recording system captures the left image Pl and the right image Pr by the pair of left and right imaging cameras A1 and B2, and outputs the captured image of the target object to the parallax vector extractor 3. The disparity vector extractor 3 associates each pixel according to an evaluation function such as a correlation function disclosed in Japanese Patent Laid-Open No. 9-33249. Here, the two imaging cameras A1 and B2 are arranged so that their optical axes are included in the same XZ plane. As long as this is strictly arranged correctly, the search for the corresponding point may be performed only on the scanning line which is an epipolar line, but in practice, it is rather rare that one pixel is arranged on the scanning line without any error. Therefore, the parallax vector extractor 3 performs parallax vector search calculation in the horizontal direction that is the epipolar line direction and a certain degree of vertical direction in the left image Pl and the right image Pr.

  A method of calculating the epipolar line direction will be described with reference to FIG. Here, corresponding points that have been discriminated in advance or feature points that are very easy to discriminate exist at the R point (Xl, Yl) in the left image Pl and at the S point (Xr, Yr) in the right image Pr. Suppose. The direction of the epipolar line EP is obtained by connecting these two points R and S with a straight line. Here, since the direction of the epipolar line EP is substantially parallel to the X-axis direction line, it is basically obtained by a horizontal search. For example, the position of the minimum value is obtained at the position X by using the difference sum of the pixels of the small blocks of horizontal 4 pixels and vertical 2 pixels, the square sum of the differences, or the like as the evaluation parameter. When a vertical installation error is considered in the camera, the vertical search range is expanded to the extent calculated by the product of the horizontal search range and Tan θ, assuming that the camera is inclined by an angle θ from the epipolar line.

  A search is performed within the set search range. As a result, for example, when a point R (Xl, Yl) on the left image corresponds to a point S (Xr, Yr) on the right image, the point R (Xl, Yl) The disparity vector at is expressed as V (X1-Xr, Y1-Yr). Such a disparity vector is obtained for all macroblocks on the left image. This process is performed over the entire screen, and the finally selected disparity vector V is sent to the depth distance calculator 4.

  The depth distance calculator 4 calculates the size of the parallax vector, and searches for a position by setting, for example, a small block of horizontal 4 pixels and vertical 2 pixels. The obtained horizontal position, vertical position, and depth direction are transmitted to the object analyzer 5 as (X, Y, Z). The object analyzer 5 formats each of the X, Y, and X with a predetermined number of bits in a raster order from the upper left to the lower right for each image. For example, if the image is NTSC class resolution of 720 × 480 horizontal, X is 10 bits and Y is 9 bits. For example, if the image has a horizontal 1920 × 1080 HDTV class resolution, X is 11 bits and Y is 9 bits.

  FIG. 4 shows a format example in the case of 720 × 480. One structure is transmitted in one frame. In the frame layer, the number of 4 × 2 macroblocks in one screen is the raster order, the object flag is 1 bit, the object ID is 7 bits, the horizontal position information X, the vertical position information Y, and the depth position information Z. Followed.

  The object flag describes 1 for a macroblock corresponding to the center of the object to be described later, and 0 for other macroblocks, and can be said to be a flag indicating whether or not the object is an object. The object ID information is also an object identification number to be described later. Since 7 bits are used here, 127 types of objects can be defined in the frame (0 occupies outside the object area). It becomes link information with corresponding sound source information.

  Further, since the object may not necessarily exist in the video, the ESC information of FIG. 4 is used at that time. In this structure, the number of subsequent ESC objects and the horizontal position, the vertical position, and the depth position at the center (or the same position) of the ESC object are recorded with 8 bits of NumOfObject. The ID of the object is also recorded. As a result, even for an object that disappears from the video and flies behind the viewer, for example, it is possible to reproduce a special scene in which only the sound image can be heard from behind.

  Information on these macro blocks is transmitted to the object analyzer 5. The object analyzer 5 differentiates the image data using a Laplacian operator, extracts the outline of the object, defines the large block as one object, and analyzes the position information of the outline of the object in the horizontal direction. And to the vertical position analyzer 7 and the depth position analyzer 8, respectively. The horizontal position analyzer 6 calculates a value obtained by halving the sum of the horizontal minimum and maximum values in the object area. The vertical position analyzer 7 calculates a value obtained by halving the sum of the minimum value and the maximum value in the vertical direction. The depth direction position analyzer 8 calculates the size of the disparity vector of a small block of horizontal 4 pixels and vertical 2 pixels corresponding to the center position from the object outline information. Although the center value is used here, the center of gravity in consideration of the area of the object image may be obtained.

  In the position information formatter 9, each calculated value is set as 1 for the macroblock corresponding to the center of the object, and for other macroblocks as the X, Y, Z position information of the object. Describe 0. Similarly, the next object is detected, but IDs 1 to 127 are assigned to the objects in the order of occurrence and recorded in the object ID field. 0 is described in places other than the object area. Since the object may be moving in the next frame image, it is the nearest neighbor, and if the outline information of the object is a feature point (for example, if it is a rectangle, the number of corners, the brightness inside the object, the color difference) Satisfies multiple conditions such as information such as signals that are almost similar within 10% of the signal (8-bit data) and that the detected object is the closest if the frames are temporally adjacent Are recognized as the same object, and the same ID value is recorded in the object ID information for the next frame transition and those recognized as the same object. The position information formatter 9 performs the format as shown in FIG. 4 and transmits the information to the information multiplexer 15 and simultaneously transmits the X, Y, and Z position information of the object to the CPU 10.

  Next, a method for recording audio information will be described. Audio is set with the plurality of microphone groups 11 centering on the cameras 1 and 2 so as to face all around. That is, as shown in FIG. 5, the surface of the sphere centered on the camera is directed in the normal direction of the spherical surface using a plurality of highly directional microphones. Although conceptually shown in FIG. 5, this microphone can virtually set three-dimensional coordinates in the X, Y, and Z directions with the center as the camera position, and can interpolate between the six directions of the axis and between them. As many microphones as possible are installed. However, the position of the microphone is devised so that the camera lens direction (the + Z direction microphone in FIG. 5) does not enter the field of view of the camera.

  From the position information formatter 9, position information of each detected object is input to the CPU 10. The CPU 10 transmits a selection signal to the sound source selector 12 so as to record the sound source of the microphone that faces in the direction corresponding to the position information of each object. The sound source selector 12 selects sound sources from a plurality of microphones according to the instruction signal, and transmits audio data transmitted from the sound source microphones to the audio compressor 13. At this time, the object ID is also transmitted from the position information formatter 9 to the audio compressor 13 via the CPU 10 and the sound source selector 12. The object ID is used by the audio compressor 13 to be described later to describe identifier information in audio compressed data with reference to a method for writing a Descriptor of the MPEG standard.

  On the other hand, the image of the camera B2 (or camera A1) is transmitted to the video compressor 14. The video compressor 14 inputs a two-dimensional image from the camera, depth information from the position information formatter 9 and object ID information, and performs MPEG standard image compression. Note that the depth information and the object ID information are transmitted in the user data area and the private stream so that the compatibility can be obtained in the MPEG standard, and the object ID uses the registration_descriptor's additional_identifier_info defined in the MPEG standard. . As another method, (1) a new private descriptor is defined to create an Object_ID_descriptor as shown in FIG. 19, and (2) an undefined region 11111010 in MPEG using a stream_ID reserved data stream. It is also possible to use a method of setting up to 11111110, and (3) a method of using an identification from 0x80 to 0xFF, which is a User Private Value in MPEG as Stream_Type. The Stream_ID in (2) is defined in the syntax of the PES packet of ISO13818-1. In addition to this, any user data area defined in MPEG may be used, and an identification code is provided in the area of user data allowed in AC3 or other audio syntax. It does not matter how you put it in.

  Audio data from the sound source selector 12 is subjected to MPEG audio compression (MPEG1 audio, MPEG2 audio, AAC, Dolby AC3, ATRAC, etc.) in the audio compressor 13.

  Next, a method for recording additional information while conforming to the MPEG standard will be described. In the MPEG image compression standard, user data areas are set in the picture layer and the GOP layer, respectively. These are recorded in a data packet such as user_data or private_data_byte set as a predetermined area in which data unrelated to video and audio can be embedded in MPEG syntax, or private_stream which can be arbitrarily set by the user. For example, a picture layer in an MPEG1 video is as shown in FIG. 17, and a mechanism is defined in which user_data can be recorded in units of 8 bits after sending user_data_start_code before the slice layer. Also, in the system layer of a multiplexed transport stream such as MPEG2, it is possible to clearly indicate that private_data exists by setting 1 to transport_private_data_flag as shown in FIG. 18, and there is a restriction that the data length does not protrude from the transport packet. Thus, private_data having the data length set in transport_private_data_length can be transmitted. In addition to this, there are several mechanisms for recording user-specific data in the MPEG system, such as sending private data by declaring a dedicated packet by setting private_stream to stream_id. Information of the structure of FIG. 4 such as information and object ID can be recorded for each picture in these areas. Any mechanism may be used, but in this embodiment, user_data of MPEG1 video is used.

  In this embodiment, user_data_start_code is defined in MPEG as 0x000001B2 before the slice layer. After sending the code, a code of, for example, 0x0f0f0f0f2417fda, which is a uniquely identifiable code indicating the presence of the function value used for authentication of the present invention in the user data area, is transmitted. This code is recorded for the purpose of identification when user_data is used in another application, and the value of the code has no particular meaning. After the code, the structure of one macroblock 32 bits (4 bytes) in FIG. 4 is recorded in the picture layer in raster order for each picture of MPEG. If the pixel is 720 × 480, the information amount is 720 × 480 × 4 / (4 × 2) = 17 KB. If the information is large, this information is compressed. For example, the depth information is often the same as the adjacent macroblock, but there is often only a slight difference, so when it has not changed, it can be skipped by setting a 4-byte skip code (for example, 0x00000000 for all 0s). During this time, the macro block information can be reduced from 4 bytes to 1 byte. Or, in the case of a slight difference (for example, in the case of plus or minus 1 or less), it becomes 0. In addition to the skip method as described above, any method may be used for compression, such as a DPCM method in which only the change is taken and the entropy is reduced.

  The procedure of the stereoscopic video / stereo acoustic correspondence recording process by the above stereoscopic video / stereo acoustic correspondence recording system will be described with reference to the flowchart of FIG. The image data from the cameras A1 and B2 and the acoustic data from the plurality of microphone groups 11 are input for a predetermined time and stored in the storage device (step S1). Next, a parallax vector of the image is extracted (step S2). Here, the parallax vector search calculation between the horizontal direction which is the epipolar line direction and a certain vertical direction is obtained for all macroblocks on the image. Next, the depth distance of the macroblock is calculated. That is, the magnitude of the parallax vector is calculated (step S3).

  Next, it is determined whether or not the object exists in the image (step S4). In this case, the image data is differentiated by using a simple Laplacian operator or the like to determine whether or not the contour of the object exists. If the determination result is YES, full-scale object detection is performed (step S5). That is, the object analyzer differentiates the image data using a Laplacian operator, extracts the outline of the object, defines the large block as one object, and determines the area of the object. Next, the determined object is extracted (defined) to determine which macroblock is included in the object (step S6). Next, the horizontal and vertical position information of the object is analyzed (step S7). In this process, since the depth information has already been obtained for each macroblock, the macroblock applied to the object area is designated and the center position thereof is set to the object position. Next, the object ID and position information are formatted (step S8). That is, the obtained horizontal position, vertical position, and depth direction are set to the format shown in FIG. 4 with the position information, the object flag, and the object ID as (X, Y, Z). Next, a sound source to be linked to the position information of the object is selected (step S9).

  On the other hand, if the determination in step S4 is NO, it is determined in step S15 whether or not the object motion is extended in the scene such as the previous picture to represent an object that does not exist in the video data. If the determination is yes, the process jumps to step S7. In the case of NO, the above-described position information is formatted with the object ID set to 0 (step S16). When it does not exist in the video, the number of subsequent ESC objects, the horizontal position, the vertical position, the depth position, and the object ID at the center (or the same position) of the ESC object are determined using the ESC information of FIG. Record.

  Next, a plurality of audio data linked with video data and objects are compressed (step S10). Next, the same identification information as the ID of the video object linked to the audio stream is described (step S11). Next, MPEG multiplexing is performed (step S12), and when recording or transmission on a medium in a predetermined unit, packetization specific to the transmission path is performed and transmitted (step S13). Next, it is determined whether there is still input image data (step S14). If there is (YES), the process jumps to step S1. If not (NO), the process is terminated.

  As a result, in the stereoscopic video / stereoacoustic recording system of the present embodiment, the stereoscopic position information of the object is described in predetermined time units using the MPEG-specified user data area or private stream, or an information body in another area. Then, by describing the object identification information linked with the identification information of the streams of both the stereoscopic video data and the audio data in a predetermined time unit, the stereoscopic video data can be recorded on the recording medium together with the audio data. .

  (Second Embodiment) Next, with reference to FIG. 7, a description will be given of a stereoscopic video / stereoacoustic recording system according to a second embodiment of the present invention based on CG (computer graphics). Since CG does not require a camera or a microphone for imaging, a 3D image and 3D sound are created by the processing of the CPU 20 by executing a program. For this purpose, there are an image signal generator 21 and a sound source signal generator 22 in the CPU 20, each of which is activated by dedicated software. The CG image data basically includes position information and depth information in advance for a small-sized image such as a polygon. Therefore, the position information of the polygon portion corresponding to the 4 × 2 macroblock described above can be easily calculated. The audio can also be easily created by creating predetermined sound source data at the object position of the CG image with a synthesizer and using the sound source. The video is input to the video compressor 23 and the audio is input to the audio compressor 24 to be compressed. The compressed data is multiplexed by the information multiplexer 26 together with the information formatted by the position information formatter 25 in the same manner as the system of FIG. 1, and is recorded on the recording medium 17 by the recorder 27.

  As a result, in the stereoscopic video / stereoacoustic recording system of the present embodiment, the stereoscopic position information of the object is described in predetermined time units using the MPEG-specified user data area or private stream, or an information body in another area. Then, by describing the object identification information linked with the identification information of both the stereoscopic video data and the audio data in a predetermined time unit, the stereoscopic video data can be recorded on the recording medium together with the audio data.

  (Third Embodiment) FIG. 8 shows a stereoscopic video / stereoscopic sound recording system according to a third embodiment of the present invention. In the first embodiment shown in FIG. 1 and the second embodiment shown in FIG. 7, the final information is recorded on the recording medium 17. However, in the system of this embodiment, communication and broadcasting are specific. Packetized and transmitted to a broadcast or communication network. Therefore, the system according to the present embodiment includes a communication (broadcast) packetizer 18, and the multiplexed data of the stereoscopic video data and the stereoscopic audio data from the information multiplexer 15 is converted into packet data for communication (broadcast). Packetize and send to communication network or broadcast network. In the system of the present embodiment, the same reference numerals are assigned to the device elements common to the system of the first embodiment shown in FIG.

  The stereoscopic video / stereo sound correspondence recording method by the system of the present embodiment is based on the flowchart of FIG. 6 as in the first embodiment. As a result, in the system according to the present embodiment, as in the first embodiment, the stereoscopic position information of the object is converted into the MPEG-specified user data area or private stream, or the information body of another area in a predetermined time unit. By describing and linking the object identification information with the identification information of the streams of both the stereoscopic video data and the audio data in a predetermined time unit, the stereoscopic video data is multiplexed with the audio data and packetized. Can be sent.

  (Fourth Embodiment) Next, a stereoscopic video for reproducing the stereoscopic video / stereoacoustic recording information created by the stereoscopic video / stereoacoustic recording system of the above-described embodiment and recorded on the recording medium 17 is reproduced. A stereophonic sound compatible playback system will be described with reference to FIG. The reproduction system according to the present embodiment includes a reproduction unit 31, an information separator 32, a video decoder 33, a position information extraction unit 34, an audio decoder 35, a visual field converter 36, a stereoscopic image display 37, a sound source selector 38, A sound image position controller 39 and a speaker array 40 are provided.

In this stereoscopic video / stereoscopic reproduction system, the data multiplexed from the recording medium 17 is read by the reproduction device 31 and transmitted to the information separator 32. The information separator 32 separates the video signal and audio signal packets, and transmits the video signal to the video decoder 33 and the audio signal to the audio decoder 35. The video decoder 33 decodes the video and simultaneously transmits the user data of the video picture layer to the position information extractor 34. The position information extractor 34 extracts depth information, object position information, and object ID information recorded in the format of FIG. 4 from user data. The object position information and the object ID information are transmitted to the sound source selector 38. The depth information and the decoded video signal are transmitted to the visual field converter 36. The visual field converter 36 generates a parallax image corresponding to the stereoscopic display method of the stereoscopic image display 37 from the depth information and the decoded two-dimensional image. When generating the parallax image, a visual field conversion method is used as a conversion method of the coordinate system in CG. This is to obtain an image with a different viewpoint from the expression for conversion to the viewpoint coordinate system. If there is depth information, it can be generated from any viewpoint. For example, the coordinates of the viewpoint _{(x i, y i, z} i), the coordinates of the gazing point _{(x a, y a, z} a) and. In addition, the distance between the viewpoint and the gazing point is (x _f , y _f , z _f ).

First, move the position of the origin by parallel movement. This conversion is assumed to be T1. The transformation T1 is simply a transformation that translates (−x _a , −y _a , −z _a ). Next, the direction of the coordinate value is changed by rotation. Vector from point O _a of the point O _f direction as shown in FIG. 10 is rotated by the y-axis is first α angle vector of the z-axis from the point O _a, then allowed to rotate in only x-axis β corner. The direction of rotation is reversed so actually move the coordinates of the point O _f.

It becomes. Here, since α is the angle of the foot and O _a obtained by projecting the O _f the xy plane,

It becomes. The β length between _O a _{O f}

And the length y _f between O _f O _{f ′}

It becomes.

Last conversion is performed z-axis from the coordinate system such that the front with respect to the xy plane, the eye direction relative to the xy plane, i.e. the transformation T ₄ to allow the other side is positive. This is simply z → -z. By multiplying these four conversion matrices T _{1 to} T _4, the viewpoint coordinate conversion matrix is

It becomes.

  In IP (Integral Photography), an element image corresponding to a plurality of lens arrays is captured by a camera corresponding to the lens position. At the same time, it is calculated and generated using the conversion matrix of the viewpoint coordinates. The stereoscopic image data generated in this way is transmitted to the stereoscopic image display 37 to perform stereoscopic image reproduction. Note that a binocular stereoscopic display method using a parallax barrier may be employed as the stereoscopic display method of the stereoscopic image display 37. In that case, α that can be set according to the viewing distance is substituted into the above equation, and β and γ are set to 0, so that an image having parallax for the right eye and the left eye can be generated.

  Here, a typical parallax barrier method and an IP method among the stereoscopic image display methods will be described. The parallax barrier method can be realized by liquid crystal. As shown in FIG. 11, this is a laminate of two liquid crystal panels 101 and 102. One liquid crystal panel 101 has a thin slit-like opening, and an appropriate interval is provided on the liquid crystal panel 102 on the back side. In this case, images for the left (L) and right (R) two eyes can be alternately arranged, and the images separated from the right eye and the left eye can be perceived when viewed from the predetermined viewpoints 103L and 103R through the slit-shaped opening. Is. As a result, different images can be input to the right eye and the left eye, and can be perceived as a stereoscopic image. A backlight 104 is provided to illuminate the image on the liquid crystal panel 102.

  However, in the case of the parallax barrier system, the eye is always focused on the liquid crystal screen, but the image appears to be different from this position, which is accompanied by physiological unnaturalness. In recent years, it has been pointed out that the user is prone to fatigue and video sickness. In recent years, there are four physiological factors of stereoscopic vision, congestion adjustment contradiction (contradiction of convergence point and focus position) (binocular parallax) = When looking at an object, the human right and left eyes capture two different images seen from different directions, focus adjustment = "The thickness of the lens is controlled as the distance from the object is changed" The nature of changing the thickness of the lens, convergence = distant, close change, the nature of the eyeball moving inward or outward, motion parallax = user himself Such a manner as to satisfy the property) to see the difference in the image due to changing the angle of view Italy has also been proposed. Among them as promising, method Lippmann announced in 1908 is an IP system.

  In the IP method, depth information of an object is acquired using a two-dimensionally arranged lens array (also referred to as a fly-eye lens, a fly-eye lens, or a compound eye lens). In the 1990s, a technology for generating moving images by the IP method was developed by replacing electronic recording with a conventional photographic dry plate. Further, a refractive index distribution lens array (GRIN) was developed by a researcher of the same document. (It is also referred to as a lens array) and a high-vision camera is used to capture a subject and acquire an element image group, and each image is transmitted and displayed on a liquid crystal display in real time and is imaged in space by a fly-eye lens. Successful, the feasibility of 3D television broadcasting by the IP system was shown (Non-Patent Document 3). FIG. 12 illustrates the principle of this IP system. As shown in FIG. 12A, a GRIN lens array 110 in which a large number of minute element lenses are arranged at the time of photographing is used, and the light of each minute element lens of the GRIN lens array 110 is used. The light is collected by the condensing lens 111 and one pixel of the micro camera 112 captures an image of light in one direction. Then, when reproducing, the image 120 from the camera is reproduced on a display 121 such as an LCD as shown in FIG. 12B, and one point of all the minute cameras are gathered to form a reproduced image viewed from one direction as a whole. .

  By using the lens array 122 in which micro lenses are arranged two-dimensionally, it is possible to create horizontal and vertical motion parallax, and it is also possible to have parallax only in the horizontal direction if arranged in the horizontal direction. In this method, a plurality of element images seen through a plurality of lenses are created by converting the viewpoints of the element images based on the depth information, and the element images are arranged, as if they were captured in FIG. 12A. As described above, stereoscopic image reproduction is realized by displaying the element image array on the LCD 121.

  On the other hand, audio is transmitted to the sound source selector 38 after the audio decoder 35 decodes the compressed audio data of a plurality of sound sources. The sound source selector 38 receives the object position information and the object ID information from the position information extractor 34 described above, and links to the object ID in accordance with the object position information existing in the image. The decoded sound source is selected, and the selected sound source data and object position information are paired and transmitted to the sound image position controller 39. The sound image position controller 39 controls the localization of an audio sound source linked to each sound source according to the position of the image object, using a sound image localization control method using a speaker array 40 described later. Audio data corresponding to a plurality of speakers as a result of localization control for each sound source is obtained as a sound source of a plurality of objects corresponding to one speaker. All of these are linearly added, the gain is adjusted, and the audio data output from one speaker is transmitted to the speaker array 40 as one. The speaker array 40 outputs the transmitted audio data.

  Here, the sound image localization control method will be described. Here, using a speaker ray, the delay according to the difference between the route from the center of the speaker ray to the focal point and the route from each speaker to the focal point so as to locally increase the sound pressure near a focal point in space. Sound image localization is realized by a reproduction signal given a quantity. The principle will be described with reference to FIG. First, the speaker array 40 is constructed by assembling speakers in an array as shown in FIG. A delay circuit 131 is provided for each speaker. When the delay is set by using the delay circuit 131 in the known method to focus on the vicinity of the listening position, the sound pressure component generated at the focal point is much higher than the direct sound from the speaker at the listening position. It is possible to play back. By using this principle and continuously controlling in real time, the localization of the sound image is controlled by linking to the position of the object of the stereoscopic moving image.

  Through this series of data processing, the image is reproduced as a three-dimensional image, and the object of the image in the image is linked to the three-dimensional movement to control the position of the sound source of the audio data, as if it were three-dimensionally viewed. The sound is localized and reproduced as if the sound emitted by the object is heard from the space. That is, as shown in FIG. 14, a stereoscopic video 120 is reproduced using the LCD 121, and a stereoscopic video is constructed from the element images using, for example, the IP lens array 122, and then the speaker array 40 is set. The sound image localization is performed so that the sound source of the object is heard from the position of the object in a form linked to a stereoscopic image object such as a car or airplane that is perceived as coming out of the display 121. And play.

  Next, reproduction processing by the stereoscopic video / stereoscopic reproduction system using the IP system will be described with reference to the flowchart of FIG. First, the multiplexed data is read from the recording medium 17 (step R1). Next, MPEG separation processing is performed (step R2). Of the separated information, video data is subjected to video data decoding processing, and audio data is subjected to audio data decoding processing (steps R3 and R9). Next, the object ID and position information are separated from the user data of the decoded video data (step R4). Next, depth information for each macroblock is detected from the separated information (step R5). Further, the ID of the object is detected (step R6).

  On the other hand, the object ID described in the stream ID of the audio data is detected from the audio data decoded in step R9 (step R10). Next, the audio data ID linked to the video object is selected by collating the IDs of the plurality of audio objects with the video ID (step R11).

  Next, the video uses the depth information held for the pixels in each macroblock to perform visual field conversion of the video image (step R7). For this, an image in which the viewpoint is changed by a conversion formula to the viewpoint coordinate system is obtained. Next, stereoscopic display of video is performed by the IP method (step R8). As this stereoscopic display method, a binocular stereoscopic display method using a parallax barrier can also be used.

  On the other hand, for each of a plurality of sound sources in which the selected link relation is clarified, the audio is controlled by the sound image position controller 39 using, for example, a sound image localization control method using a speaker array 40, for each sound source. In accordance with the position of the image object, the localization of the audio sound source linked to the image object is controlled (step R12). Next, the audio data corresponding to a plurality of speakers as a result of localization control for each sound source by the speaker array 40 is linearly added, and the gain is adjusted so that one audio data is output from one speaker. Output from the speaker array 40 (step R13).

  In this manner, in the stereoscopic video / stereoscopic playback system of the present embodiment, the stereoscopic video 120 is reproduced using the LCD 121, and the stereoscopic image is formed from the element image using the IP lens array 122. Then, the sound source of the object is linked to a stereoscopic image object such as a car or an airplane that is perceived as coming out of the display 121 by the speaker array 40 set after that, as if the sound source of the object is The sound image can be localized and reproduced so that it can be heard from the position.

(Fifth embodiment)
Next, a stereoscopic video / stereoscopic reproduction system according to a fifth embodiment of the present invention will be described with reference to FIG. Although the final information is reproduced from the recording medium 17 in the stereoscopic image / stereoscopic reproduction system of the fourth embodiment shown in FIG. 9, the reproduction system of the present embodiment Receives packet information that has been packetized peculiar to broadcasting, and receives and reproduces 3D video / audio data packets from broadcasting and communication networks via a communication (broadcasting) packet releaser 31 'as shown in FIG. It is characterized by doing. The data whose packet has been canceled by the packet canceller 31 'is the same as the reproduction data from the recording medium 17 in FIG. 9, and therefore the components after the information separator 32 and their processing functions are shown in FIG. This is the same as that of the fourth embodiment.

  As a result, even in the stereoscopic video / stereoscopic playback system of the fifth embodiment, the stereoscopic video image received via the communication network or broadcast is played back using the LCD 121 shown in FIG. A three-dimensional image is constructed from element images using the array 122, and then linked to a three-dimensional object such as a car or airplane that is perceived as appearing in front of the display 121 by the speaker array 40 set thereafter. In this way, the sound source of the object can be reproduced with sound image localization so that it can be heard from the position of the object.

  In addition, this invention is not limited to said embodiment, The following deformation | transformation aspects are possible. In the above-described embodiment, the stereoscopic video method is described as the IP method. However, the stereoscopic video method may be a parallax barrier method, a lenticular lens method, a super multi-view method, a binocular method using deflecting glasses, an anaglyph, or the like. Whatever. In addition, the sound image position control method has been described with the speaker ray method as the sound image localization control method, but even a method that can realize a virtual sound field space, for example, a binaural transoral method, is represented by a wave represented by Kirchoff-Helmhotz differential equations. A method using a sound field control method using acoustic theory may be used.

  In addition, it is possible to transmit such data via any transmission media such as communication and broadcasting without recording stereoscopic video / audio data on the recording media. It can also be used. The playback device can also be used as a receiving device.

  The recording medium on which the signal data of the present invention is recorded has a media-specific effect that the object position information and the object ID information are recorded, and preferably realizes a system for reproducing a three-dimensional video or a three-dimensional sound field. Can do. In addition, the definition of “media” in a recording medium includes not only a narrowly-defined medium that can record data, but also electromagnetic waves and light for transmitting signal data. The information recorded on the recording medium includes data itself such as an electronic file in a state where it is not recorded.

  Furthermore, in the above-described embodiment, the video depth information has been described as being recorded for each picture, but it may be about every 0.5 seconds or about every 1 second. In that case, it can be realized by using user data of the GOP layer of MPEG. The object ID may be 8 bits or 16 bits. The object may be a video object or a region. Some algorithms may contain errors in detecting objects, but they may be ignored. Furthermore, the region may not be specified by a closed curve. In addition, if the sound localization has a localization effect even slightly compared to normal stereo reproduction, it is considered that the localization is controlled.

  In addition, although the depth information and the object ID information have been described using MPEG video user data in the above embodiment, they may be recorded in a private stream defined by the MPEG system layer. In this case, since data is synchronized with time for each picture or for each of a plurality of GOPs, it is synchronized with images and sound data using PTS (Presentation Time Stamp), so that the private stream stream format of the private stream defined by MPEG is used. Is desirable. In addition, the information in FIG. 4 is an encoded audiovisual data program in the same area as the audiovisual data, that is, as a separate file on the recording medium, separately from the MPEG stream, with the structure of FIG. 4 as it is. Each may be recorded in a named file. In that case, it is also possible to record in the picture (field or image) order of the reproduction order (= input image order) or the picture (field or image) order of the encoding order in MPEG. The file name may be a number or an ASCII character as long as the program can be identified. Further, each file may be a single file that combines the entire media, or a playlist that combines several programs.

  Furthermore, the functions of the apparatus described above may be realized by a computer by a program. The program may be recorded on a recording medium, read from the recording medium, and loaded into a computer, or transmitted through a communication network into the computer.

1 is a functional block diagram of a recording apparatus for stereoscopic video / stereoscopic sound according to a first embodiment of the present invention. The block diagram of the principal part of the recording apparatus corresponding to the three-dimensional video / stereo sound of the first embodiment. Explanatory drawing of the epipolar constraint conditions for calculating | requiring the general parallax of 2 screens. The format explanatory drawing of depth information, object ID information, etc. FIG. Explanatory drawing of the installation method of the microphone which records the whole circumference. The flowchart figure which shows the stereo image / stereo sound correspondence recording process by the stereo image / stereo sound correspondence recording apparatus of the 1st Embodiment of this invention. The block diagram which shows the three-dimensional video / stereo sound corresponding recording apparatus of the 2nd Embodiment of this invention. The block diagram which shows the transmission apparatus corresponding to the stereo image / stereo sound of the 3rd Embodiment of this invention. The block diagram which shows the reproduction | regeneration apparatus corresponding to the three-dimensional video / stereo sound of the 4th Embodiment of this invention. Explanatory drawing of general viewpoint conversion. Explanatory drawing of a general parallax barrier system. Explanatory drawing of a general IP system. (1) An explanatory diagram of a general IP system. (2) Explanatory drawing of the array speaker used in the 4th Embodiment of this invention. The block diagram of the system of a speaker array and IP stereoscopic video system by the said 4th Embodiment. The flowchart figure which shows the three-dimensional video and three-dimensional sound reproduction | regeneration processing by the said 4th Embodiment. The block diagram which shows the reproduction | regeneration system corresponding to the three-dimensional video / stereo sound of the 5th Embodiment of this invention. FIG. 3 is an explanatory diagram of an MPEG video stream video layer. FIG. 2 is an explanatory diagram of an MPEG multiplexed transport stream system layer. (1) An explanatory diagram of an MPEG multiplexed transport stream system layer. (2) Explanatory drawing which shows Object_ID_Descriptor which describes new object ID employ | adopted by the 1st Embodiment of this invention.

Explanation of symbols

1 Camera A
2 Camera B
3 Disparity vector extractor 4 Depth distance calculator 5 Object analyzer 6 Horizontal position analyzer 7 Vertical position analyzer 8 Depth position analyzer 9 Position information formatter 10 CPU
DESCRIPTION OF SYMBOLS 11 Multiple microphone group 12 Sound source selector 13 Audio compressor 14 Video compressor 15 Information multiplexer 16 Recorder 17 Recording medium 31 Regenerator 32 Information separator 33 Video decoder 34 Position information extractor 35 Audio decoder 36 View conversion 37 Stereoscopic image display 38 Sound source selector 39 Sound image position controller 40 Speaker layout

Claims (5)

Recording both stereoscopic video data and audio data on a recording medium;
Refers to the stereo position information of the object for which stereo localization control is performed using the sound source as a sound source at the time of reproduction, together with the identification information of the object, at the time of reproduction of the stereoscopic video data and audio data on the recording medium in a predetermined time unit. A recording program for stereoscopic video / stereo sound that causes a computer to execute the step of recording in a predetermined storage area. Reading and playing back stereoscopic video data and audio data recorded on a recording medium;
Reading object identification information and stereo position information for performing stereo localization control of the sound source from a predetermined storage area of the recording medium;
When reproducing the acoustic data, based on the three-dimensional position information corresponding to the identification information of the object, the sound image is obtained using at least two or more speakers so that the three-dimensional localization position of the sound image of the acoustic data becomes the three-dimensional position of the object. A reproduction program for stereoscopic video and stereophonic sound that causes a computer to execute the step of performing position control. Means for recording both stereoscopic video data and audio data on a recording medium;
Refers to the stereo position information of the object for which stereo localization control is performed using the sound source as a sound source at the time of reproduction, together with the identification information of the object, at the time of reproduction of the stereoscopic video data and audio data on the recording medium in a predetermined time unit. A stereoscopic video / stereoscopic recording device comprising: means for recording in a predetermined predetermined storage area. Means for reading and playing back stereoscopic video data and audio data recorded on a recording medium;
Means for reading identification information and stereoscopic position information of an object for performing stereo localization control of a sound source from a predetermined storage area of the recording medium;
When reproducing the acoustic data, based on the three-dimensional position information corresponding to the identification information of the object, the sound image is obtained using at least two or more speakers so that the three-dimensional localization position of the sound image of the acoustic data becomes the three-dimensional position of the object. A reproduction apparatus for stereoscopic video and stereophonic sound, comprising: means for performing position control. Both the stereoscopic video data and the audio data are recorded, and the stereo position information of the object for which the stereo localization control of the sound source is controlled in a predetermined time unit, together with the identification information of the object, the stereo video data and the audio data of the recording medium. A recording medium for stereoscopic video / stereo sound recorded in a predetermined storage area that can be referred to during playback.

Images