JP2006128818A - Recording program and reproducing program corresponding to stereoscopic video and 3d audio, recording apparatus, reproducing apparatus and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a recording/reproducing technique corresponding to stereoscopic video and stereoscopic audio, capable of performing corresponding recording and reproducing of the contents of stereoscopic video and stereoscopic audio. <P>SOLUTION: This system consists of a recording system for describing the stereoscopic positional information of an object in specific time units, by using user data region defined in MPEG, a private stream, or an information body of another region, describing the identification information of the object in specific time units so that the information is linked with the identification information of the steams of both stereoscopic video data and stereoscopic audio data, and recording the stereoscopic video data in a recording medium, together with the audio data; and a reproducing system for detecting the stereoscopic positional information of the object from the recording medium, detecting the identification information of the object from the streams of both the stereoscopic video data and stereoscopic audio data, and controlling sound image localization position, by using a speaker array so that the stereoscopic localization position of the sound image of the audio data, corresponding to the identification information of the object of the video image, becomes the stereoscopic position of the object. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a stereoscopic video / stereo acoustic correspondence recording program, a reproduction program, a recording apparatus, a reproducing apparatus, and an audiovisual recording medium for recording and reproducing stereoscopic object motion and stereoscopic sound in association with each other.

  Conventionally, 3D (stereoscopic) video reproduction techniques have been proposed in several ways. Further, as a technique related to a 3D (stereoscopic) sound field, a technique has been proposed in which a sound is localized and reproduced as if a speaker is located at that position from a place without a speaker. However, there is a problem that it is not possible to reproduce the stereoscopic video of the content that can be displayed and reproduced by the stereoscopic video technology that can express the space and the sound by the spatial localization technology of the spatial sound in cooperation.

  For example, in Japanese Patent Laid-Open No. 2000-17355 (Patent Document 1), the present inventor discloses a phase in both directions with respect to a direction of an epipolar line connecting between corresponding pixel points of a stereo image related to a target object and a direction orthogonal thereto. A method of calculating a distance to a target object by performing a two-dimensional search on the screen so as to include both corresponding pixel points and calculating a disparity vector is disclosed.

  In Japanese Patent Laid-Open No. 7-236199 (Patent Document 2), the movements of left and right objects in a stereoscopic video are automatically detected by detecting a motion vector, and the sound fields corresponding to the motions are a main audio signal and a sub audio signal. A method of reproducing three-dimensionally with a main speaker and a surround speaker driven by the above is disclosed. However, the motion vector is the motion of the object in the time direction of the content and is not information on the depth of the 3D stereoscopic video. In this prior art, the three-dimensional sound field is also performed by simply using the surround feeling of several speakers, and does not perform localization of sound in space.

  Japanese Patent Laid-Open No. 2001-306081 (Patent Document 3) discloses a technique related to a music space configuration control device that controls the configuration of an audio space in real time, and parameters that describe a three-dimensional sound source existing in DirectX. To provide an effective music space using localization, direction, Doppler parameters, and the like. However, a parameter relating to depth, which is a space of stereoscopic video, is not disclosed, and synchronous playback of the spatial video and sound is not described.

As described above, conventionally, in response to the movement in the depth direction of a three-dimensional stereoscopic image, the sound and the sound are coordinated to reproduce the image and the sound, or even if the image exceeds the display display area. There is no known system that reproduces the sound of the content using sound localization technology and reproduces realistic video and audio.
JP 2000-17355 A JP 7-236199 A JP 2001-306081 A NHK Broadcasting Technology Laboratory, “Basics of 3D Video”, Ohm, 1995 By Jens Brauert, “Spatial Acoustics”, Kashima Press, 1985 B. Javidi, F.A. Okano Editors, “Three-Dimensional Television, Video, and Display Technologies”, Springer-Verlag (2002), P101-P123.

  It is an object of the present invention to provide a stereoscopic video / stereoacoustic recording technique capable of recording corresponding stereoscopic video and stereoscopic audio content.

  Another object of the present invention is to provide a stereoscopic video / stereo acoustic compatible reproduction technique capable of reproducing a three-dimensional position of an object video and a three-dimensional sound localization in synchronization.

  According to the first aspect of the present invention, there is provided a recording program for stereoscopic video / stereo sound, which is horizontal position information of an object as stereo positional information of an object for which sound localization control is performed using the sound source as a sound source during reproduction in a predetermined time unit. Creating vertical position information and depth position information; linking the object's stereoscopic position information with the appearance timing of the object in both the stereoscopic video data and audio data streams; and the stereoscopic video data and Recording audio data on a recording medium and recording the object's 3D position information on the recording medium linked to the appearance timing of the object in both the 3D video data and audio data streams. To be executed.

  The reproduction program for stereoscopic video / stereoscopic video / stereo sound according to the invention of claim 2 includes the stereoscopic position information and the appearance timing of the object for performing stereo localization control of the sound using itself as a sound source together with the stereoscopic video data and the acoustic data. Reading out and reproducing stereoscopic video data and audio data from a recording medium on which the link information is recorded, reading out the stereoscopic position information of the object from the recording medium, and stereoscopic position information of the object from the recording medium And a sound image position control using at least two speakers so that the stereo localization position of the sound image of the acoustic data corresponding to the stereo position information of the object becomes the stereo position of the object. Do the steps and the executed on the computer Is shall.

  According to a third aspect of the present invention, there is provided a recording apparatus for stereo video / stereo sound, which is horizontal position information of an object as stereo position information of an object to be subjected to stereo stereo control using the sound source as a sound source during reproduction in a predetermined time unit. Means for creating vertical position information and depth position information, means for linking the stereoscopic position information of the object with the appearance timing of the object in both streams of stereoscopic video data and audio data, the stereoscopic video data, and Means for recording the audio data on a recording medium and recording the object's stereoscopic position information on the recording medium linked to the appearance timing of the object in the streams of both the stereoscopic video data and the audio data. Is.

  The stereoscopic video / stereo sound compatible playback device of the invention of claim 4 is the stereoscopic position information of the object and the link information of the appearance timing for performing the stereo localization control of the sound using the sound source as a sound source together with the stereoscopic video data and the acoustic data. Means for reading out and reproducing stereoscopic video data and audio data from a recording medium on which recording is performed, means for reading out stereoscopic position information of the object from the recording medium, and corresponding to the stereoscopic position information of the object from the recording medium Means for reading out sound data, and means for performing sound image position control using at least two or more speakers so that the stereo localization position of the sound image of the acoustic data corresponding to the stereo position information of the object becomes the stereo position of the object. It is equipped with.

  The stereoscopic video / stereo sound compatible recording medium of the invention of claim 5 records the stereoscopic video data and the acoustic data, and attempts to perform the stereo localization control of the sound using itself as a sound source during reproduction created in a predetermined time unit. The stereoscopic position information of the object is recorded by being linked to the appearance timing of the object in the streams of both the stereoscopic video data and the audio data.

  According to a sixth aspect of the present invention, there is provided a three-dimensional video / stereo sound recording program for reproducing information as a sound source and an object to be displayed as a sound source and an object to be reproduced as a sound source but not displayed as a video in a predetermined time unit. Creating horizontal position information, vertical position information, and depth position information for each object, setting identification information for each object, and identifying each object for both stereoscopic video data and audio data. Linking to the appearance timing of the object in the stream, recording the stereoscopic video data and audio data on a recording medium, and the stereoscopic position information of the object in both the stereoscopic video data and audio data streams Appearance timing of the object Grayed and is linked is intended to execute a step of recording on the recording medium to the computer.

  The reproduction program for stereoscopic video / stereo sound of the invention of claim 7 is reproduced as a sound source at the time of reproduction together with the stereoscopic video data and the audio data, and is reproduced as an object and a sound source to be displayed on the video, but is not displayed on the video. 3D position information with respect to the object and information indicating the appearance timing corresponding to the identification information of each object are recorded and read out from the recording medium, and the 3D video data and the sound data are reproduced and displayed on the video from the recording medium A step of reading identification information and stereoscopic position information of an object to be displayed and an object not to be displayed, a step of reading acoustic data corresponding to the identification information of each object from the recording medium, and a corresponding to identification information of each object of the stereoscopic video data Acoustic data The three-dimensional localization position of the image so that the stereoscopic positions of the object, in which and a step of performing sound image position control by using at least two or more speakers in the computer.

  The stereoscopic video / stereoacoustic recording apparatus according to claim 8 is configured as stereoscopic position information of an object that is reproduced as a sound source and displayed on the video and an object that is reproduced as the sound source but not displayed on the video in a predetermined time unit. Means for creating horizontal position information, vertical position information, and depth position information of each object, means for setting identification information for each of the objects, and identification information for each of the objects, streams of both stereoscopic video data and audio data Means for linking to the appearance timing of the object in the medium, and recording the stereoscopic video data and the acoustic data on a recording medium, and the stereoscopic position information of the object in the stream of both the stereoscopic video data and the acoustic data Linked to the appearance timing of It is obtained by a means for recording on the media.

  According to the ninth aspect of the present invention, the 3D video / stereo sound compatible playback apparatus plays back the 3D video data and the audio data as a sound source at the time of playback, and plays back the object and sound source displayed in the video but does not display them in the video. Means for reading out and reproducing stereoscopic video data and audio data from a recording medium in which information indicating stereoscopic position information and appearance timing of the object is recorded in association with identification information of each object, and displayed on the video from the recording medium Means for reading identification information and stereoscopic position information of objects to be displayed and objects not to be displayed, means for reading acoustic data corresponding to the identification information of each object from the recording medium, and corresponding to identification information of each object of the stereoscopic video data Position of sound image of sound data As will become the object each stereoscopic position, in which a means for performing sound image position control by using at least two or more speakers.

  According to the tenth aspect of the present invention, there is provided a recording medium for stereo video / stereo sound that records stereo video data and audio data, reproduces a sound source created in a predetermined time unit, and reproduces an object and a sound source that display images. However, the stereoscopic position information and the identification information of an object that does not display video are recorded by being linked with the appearance timings of both the stereoscopic video data and the audio data of the object.

  According to the present invention, the recording system can create stereoscopic position information of an object in a stereoscopic video in a predetermined time unit and record it on a recording medium together with stereoscopic video data and acoustic data.

  Further, according to the present invention, on the reproduction system side, it is possible to detect the spatial position information of the object in the stereoscopic video and accurately reproduce the spatial localization of the spatial sound based on the positional information. Further, by controlling the sound image position using at least two speakers so that the stereo position of the sound image of the corresponding acoustic data becomes the stereo position of the object based on the stereo position information of the object, It is possible to accurately reproduce the three-dimensional position of the object and the spatial localization of the spatial sound.

  According to the invention, in the recording system, stereoscopic position information and object identification information of an object to be displayed and an object not to be displayed are created in a predetermined time unit, and the stereoscopic video data is recorded on the recording medium together with the audio data. be able to.

  Further, according to the present invention, on the playback system side, the 3D video / stereo sound compatible playback program and playback device detect the 3D position information and the object identification information of the object to be displayed and the object not to be displayed in the video, and Sound image position control is performed using at least two speakers so that the stereo localization position of the sound image of the audio data corresponding to the identification information becomes the stereo position of the object, and the stereo position of the object in the stereoscopic video and the spatial acoustic Even in the case of a scene where the spatial localization is accurately reproduced and the object disappears from the video (moves to the outside of the screen), for example, the sound image position is localized in the space on the extension line where the object moves. It is possible to perform playback with a more realistic feeling.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  (First Embodiment) FIG. 1 is a block diagram of a stereoscopic video / stereo acoustic recording system according to one embodiment of the present invention, and FIG. 2 shows a photographing mode of a target object using the same. . The stereoscopic video / stereoacoustic recording system of the present embodiment includes a pair of left and right imaging cameras A1, B2, parallax vector extractor 3, depth distance calculator 4, object analyzer 5, as a stereoscopic video data recording system. A horizontal position analyzer 6, a vertical position analyzer 7, a depth direction position analyzer 8, a position information formatter 9, and a video compressor 14 are provided. This system also includes a plurality of microphone groups 11, a sound source selector 12, and an audio compressor 13 as an acoustic data recording system. The system also includes a CPU 10, an information multiplexer 15, and a recorder 16 as elements common to both systems.

  A stereoscopic video / stereo sound compatible recording operation by the stereoscopic video / stereo sound compatible recording system configured as described above will be described. The video recording system captures the left image Pl and the right image Pr by the pair of left and right imaging cameras A1 and B2, and outputs the captured image of the target object to the parallax vector extractor 3. The disparity vector extractor 3 associates each pixel according to an evaluation function such as a correlation function disclosed in Japanese Patent Laid-Open No. 9-33249. Here, the two imaging cameras A1 and B2 are arranged so that their optical axes are included in the same XZ plane. As long as this is strictly arranged correctly, the search for the corresponding point may be performed only on the scanning line which is an epipolar line, but in practice, it is rather rare that one pixel is arranged on the scanning line without any error. Therefore, the parallax vector extractor 3 performs parallax vector search calculation in the horizontal direction that is the epipolar line direction and a certain degree of vertical direction in the left image Pl and the right image Pr.

  A method of calculating the epipolar line direction will be described with reference to FIG. Here, corresponding points that have been discriminated in advance or feature points that are very easy to discriminate exist at the R point (Xl, Yl) in the left image Pl and at the S point (Xr, Yr) in the right image Pr. Suppose. The direction of the epipolar line EP is obtained by connecting these two points R and S with a straight line. Here, since the direction of the epipolar line EP is substantially parallel to the X-axis direction line, it is basically obtained by a horizontal search. For example, the position of the minimum value is obtained at the position X by using the difference sum of the pixels of the small blocks of horizontal 4 pixels and vertical 2 pixels, the square sum of the differences, or the like as the evaluation parameter. When a vertical installation error is considered in the camera, the vertical search range is expanded to the extent calculated by the product of the horizontal search range and Tan θ, assuming that the camera is inclined by an angle θ from the epipolar line.

  A search is performed within the set search range. As a result, for example, when a point R (Xl, Yl) on the left image corresponds to a point S (Xr, Yr) on the right image, the point R (Xl, Yl) The disparity vector at is expressed as V (X1-Xr, Y1-Yr). Such a disparity vector is obtained for all macroblocks on the left image. This process is performed over the entire screen, and the finally selected disparity vector V is sent to the depth distance calculator 4.

  The depth distance calculator 4 calculates the size of the parallax vector, and searches for a position by setting, for example, a small block of horizontal 4 pixels and vertical 2 pixels. The obtained horizontal position, vertical position, and depth direction are transmitted to the object analyzer 5 as (X, Y, Z). The object analyzer 5 formats each of the X, Y, and X with a predetermined number of bits in a raster order from the upper left to the lower right for each image. For example, if the image is NTSC class resolution of 720 × 480 horizontal, X is 10 bits and Y is 9 bits. For example, if the image has a horizontal 1920 × 1080 HDTV class resolution, X is 11 bits and Y is 9 bits.

  FIG. 4 shows a format example in the case of 720 × 480. One structure is transmitted in one frame. In the frame layer, the number of 4 × 2 macroblocks in one screen is the raster order, the object flag is 1 bit, the object ID is 7 bits, the horizontal position information X, the vertical position information Y, and the depth position information Z. Followed.

  The object flag describes 1 for a macroblock corresponding to the center of the object to be described later, and 0 for other macroblocks, and can be said to be a flag indicating whether or not the object is an object. The object ID information is also an object identification number to be described later. Since 7 bits are used here, 127 types of objects can be defined in the frame (0 occupies outside the object area). It becomes link information with corresponding sound source information.

  Further, since the object may not necessarily exist in the video, the ESC information of FIG. 4 is used at that time. In this structure, the number of subsequent ESC objects and the horizontal position, the vertical position, and the depth position at the center (or the same position) of the ESC object are recorded with 8 bits of NumOfObject. The ID of the object is also recorded. As a result, even for an object that disappears from the video and flies behind the viewer, for example, it is possible to reproduce a special scene in which only the sound image can be heard from behind.

  Information on these macro blocks is transmitted to the object analyzer 5. The object analyzer 5 differentiates the image data using a Laplacian operator, extracts the outline of the object, defines the large block as one object, and analyzes the position information of the outline of the object in the horizontal direction. And to the vertical position analyzer 7 and the depth position analyzer 8, respectively. The horizontal position analyzer 6 calculates a value obtained by halving the sum of the horizontal minimum and maximum values in the object area. The vertical position analyzer 7 calculates a value obtained by halving the sum of the minimum value and the maximum value in the vertical direction. The depth direction position analyzer 8 calculates the size of the disparity vector of a small block of horizontal 4 pixels and vertical 2 pixels corresponding to the center position from the object outline information. Although the center value is used here, the center of gravity in consideration of the area of the object image may be obtained.

  In the position information formatter 9, each calculated value is set as 1 for the macroblock corresponding to the center of the object, and for other macroblocks as the X, Y, Z position information of the object. Describe 0. Similarly, the next object is detected, but IDs 1 to 127 are assigned to the objects in the order of occurrence and recorded in the object ID field. 0 is described in places other than the object area. Since the object may be moving in the next frame image, it is the nearest neighbor, and if the outline information of the object is a feature point (for example, if it is a rectangle, the number of corners, the brightness inside the object, the color difference) Satisfies multiple conditions such as information such as signals that are almost similar within 10% of the signal (8-bit data) and that the detected object is the closest if the frames are temporally adjacent Are recognized as the same object, and the same ID value is recorded in the object ID information for the next frame transition and those recognized as the same object. The position information formatter 9 performs the format as shown in FIG. 4 and transmits the information to the information multiplexer 15 and simultaneously transmits the X, Y, and Z position information of the object to the CPU 10.

  Next, a method for recording audio information will be described. Audio is set with the plurality of microphone groups 11 centering on the cameras 1 and 2 so as to face all around. That is, as shown in FIG. 5, the surface of the sphere centered on the camera is directed in the normal direction of the spherical surface using a plurality of highly directional microphones. Although conceptually shown in FIG. 5, this microphone can virtually set three-dimensional coordinates in the X, Y, and Z directions with the center as the camera position, and can interpolate between the six directions of the axis and between them. As many microphones as possible are installed. However, the position of the microphone is devised so that the camera lens direction (the + Z direction microphone in FIG. 5) does not enter the field of view of the camera.

  From the position information formatter 9, position information of each detected object is input to the CPU 10. The CPU 10 transmits a selection signal to the sound source selector 12 so as to record the sound source of the microphone that faces in the direction corresponding to the position information of each object. The sound source selector 12 selects sound sources from a plurality of microphones according to the instruction signal, and transmits audio data transmitted from the sound source microphones to the audio compressor 13. At this time, the object ID is also transmitted from the position information formatter 9 to the audio compressor 13 via the CPU 10 and the sound source selector 12. The object ID is used by the audio compressor 13 to be described later to describe identifier information in audio compressed data with reference to a method for writing a Descriptor of the MPEG standard.

  On the other hand, the image of the camera B2 (or camera A1) is transmitted to the video compressor 14. The video compressor 14 inputs a two-dimensional image from the camera, depth information from the position information formatter 9 and object ID information, and performs MPEG standard image compression. Note that the depth information and the object ID information are transmitted in the user data area and the private stream so that the compatibility can be obtained in the MPEG standard, and the object ID uses the registration_descriptor's additional_identifier_info defined in the MPEG standard. . As another method, (1) a new private descriptor is defined to create an Object_ID_descriptor as shown in FIG. 19, and (2) an undefined region 11111010 in MPEG using a stream_ID reserved data stream. It is also possible to use a method of setting up to 11111110, and (3) a method of using an identification from 0x80 to 0xFF, which is a User Private Value in MPEG as Stream_Type. The Stream_ID in (2) is defined in the syntax of the PES packet of ISO13818-1. In addition to this, any user data area defined in MPEG may be used, and an identification code is provided in the area of user data allowed in AC3 or other audio syntax. It does not matter how you put it in.

  Audio data from the sound source selector 12 is subjected to MPEG audio compression (MPEG1 audio, MPEG2 audio, AAC, Dolby AC3, ATRAC, etc.) in the audio compressor 13.

  Next, a method for recording additional information while conforming to the MPEG standard will be described. In the MPEG image compression standard, user data areas are set in the picture layer and the GOP layer, respectively. These are recorded in a data packet such as user_data or private_data_byte set as a predetermined area in which data unrelated to video and audio can be embedded in MPEG syntax, or private_stream which can be arbitrarily set by the user. For example, a picture layer in an MPEG1 video is as shown in FIG. 17, and a mechanism is defined in which user_data can be recorded in units of 8 bits after sending user_data_start_code before the slice layer. Also, in the system layer of a multiplexed transport stream such as MPEG2, it is possible to clearly indicate that private_data exists by setting 1 to transport_private_data_flag as shown in FIG. 18, and there is a restriction that the data length does not protrude from the transport packet. Thus, private_data having the data length set in transport_private_data_length can be transmitted. In addition to this, there are several mechanisms for recording user-specific data in the MPEG system, such as sending private data by declaring a dedicated packet by setting private_stream to stream_id. Information of the structure of FIG. 4 such as information and object ID can be recorded for each picture in these areas. Any mechanism may be used, but in this embodiment, user_data of MPEG1 video is used.

  In this embodiment, user_data_start_code is defined in MPEG as 0x000001B2 before the slice layer. After sending the code, a code of, for example, 0x0f0f0f0f2417fda, which is a uniquely identifiable code indicating the presence of the function value used for authentication of the present invention in the user data area, is transmitted. This code is recorded for the purpose of identification when user_data is used in another application, and the value of the code has no particular meaning. After the code, the structure of one macroblock 32 bits (4 bytes) in FIG. 4 is recorded in the picture layer in raster order for each picture of MPEG. If the pixel is 720 × 480, the information amount is 720 × 480 × 4 / (4 × 2) = 17 KB. If the information is large, this information is compressed. For example, the depth information is often the same as the adjacent macroblock, but there is often only a slight difference, so when it has not changed, it can be skipped by setting a 4-byte skip code (for example, 0x00000000 for all 0s). During this time, the macro block information can be reduced from 4 bytes to 1 byte. Or, in the case of a slight difference (for example, in the case of plus or minus 1 or less), it becomes 0. In addition to the skip method as described above, any method may be used for compression, such as a DPCM method in which only the change is taken and the entropy is reduced.

  The procedure of the stereoscopic video / stereo acoustic correspondence recording process by the above stereoscopic video / stereo acoustic correspondence recording system will be described with reference to the flowchart of FIG. The image data from the cameras A1 and B2 and the acoustic data from the plurality of microphone groups 11 are input for a predetermined time and stored in the storage device (step S1). Next, a parallax vector of the image is extracted (step S2). Here, the parallax vector search calculation between the horizontal direction which is the epipolar line direction and a certain vertical direction is obtained for all macroblocks on the image. Next, the depth distance of the macroblock is calculated. That is, the magnitude of the parallax vector is calculated (step S3).

  Next, it is determined whether or not the object exists in the image (step S4). In this case, the image data is differentiated by using a simple Laplacian operator or the like to determine whether or not the contour of the object exists. If the determination result is YES, full-scale object detection is performed (step S5). That is, the object analyzer differentiates the image data using a Laplacian operator, extracts the outline of the object, defines the large block as one object, and determines the area of the object. Next, the determined object is extracted (defined) to determine which macroblock is included in the object (step S6). Next, the horizontal and vertical position information of the object is analyzed (step S7). In this process, since the depth information has already been obtained for each macroblock, the macroblock applied to the object area is designated and the center position thereof is set to the object position. Next, the object ID and position information are formatted (step S8). That is, the obtained horizontal position, vertical position, and depth direction are set to the format shown in FIG. 4 with the position information, the object flag, and the object ID as (X, Y, Z). Next, a sound source to be linked to the position information of the object is selected (step S9).

  On the other hand, if the determination in step S4 is NO, it is determined in step S15 whether or not the object motion is extended in the scene such as the previous picture to represent an object that does not exist in the video data. If the determination is yes, the process jumps to step S7. In the case of NO, the above-described position information is formatted with the object ID set to 0 (step S16). When it does not exist in the video, the number of subsequent ESC objects, the horizontal position, the vertical position, the depth position, and the object ID at the center (or the same position) of the ESC object are determined using the ESC information of FIG. Record.

  Next, a plurality of audio data linked with video data and objects are compressed (step S10). Next, the same identification information as the ID of the video object linked to the audio stream is described (step S11). Next, MPEG multiplexing is performed (step S12), and when recording or transmission on a medium in a predetermined unit, packetization specific to the transmission path is performed and transmitted (step S13). Next, it is determined whether there is still input image data (step S14). If there is (YES), the process jumps to step S1. If not (NO), the process is terminated.

  As a result, in the stereoscopic video / stereoacoustic recording system of the present embodiment, the stereoscopic position information of the object is described in predetermined time units using the MPEG-specified user data area or private stream, or an information body in another area. Then, by describing the object identification information linked with the identification information of the streams of both the stereoscopic video data and the audio data in a predetermined time unit, the stereoscopic video data can be recorded on the recording medium together with the audio data. .

  (Second Embodiment) Next, with reference to FIG. 7, a description will be given of a stereoscopic video / stereoacoustic recording system according to a second embodiment of the present invention based on CG (computer graphics). Since CG does not require a camera or a microphone for imaging, a 3D image and 3D sound are created by the processing of the CPU 20 by executing a program. For this purpose, there are an image signal generator 21 and a sound source signal generator 22 in the CPU 20, each of which is activated by dedicated software. The CG image data basically includes position information and depth information in advance for a small-sized image such as a polygon. Therefore, the position information of the polygon portion corresponding to the 4 × 2 macroblock described above can be easily calculated. The audio can also be easily created by creating predetermined sound source data at the object position of the CG image with a synthesizer and using the sound source. The video is input to the video compressor 23 and the audio is input to the audio compressor 24 to be compressed. The compressed data is multiplexed by the information multiplexer 26 together with the information formatted by the position information formatter 25 in the same manner as the system of FIG. 1, and is recorded on the recording medium 17 by the recorder 27.

  As a result, in the stereoscopic video / stereoacoustic recording system of the present embodiment, the stereoscopic position information of the object is described in predetermined time units using the MPEG-specified user data area or private stream, or an information body in another area. Then, by describing the object identification information linked with the identification information of both the stereoscopic video data and the audio data in a predetermined time unit, the stereoscopic video data can be recorded on the recording medium together with the audio data.

  (Third Embodiment) FIG. 8 shows a stereoscopic video / stereoscopic sound recording system according to a third embodiment of the present invention. In the first embodiment shown in FIG. 1 and the second embodiment shown in FIG. 7, the final information is recorded on the recording medium 17. However, in the system of this embodiment, communication and broadcasting are specific. Packetized and transmitted to a broadcast or communication network. Therefore, the system according to the present embodiment includes a communication (broadcast) packetizer 18, and the multiplexed data of the stereoscopic video data and the stereoscopic audio data from the information multiplexer 15 is converted into packet data for communication (broadcast). Packetize and send to communication network or broadcast network. In the system of the present embodiment, the same reference numerals are assigned to the device elements common to the system of the first embodiment shown in FIG.

  The stereoscopic video / stereo sound correspondence recording method by the system of the present embodiment is based on the flowchart of FIG. 6 as in the first embodiment. As a result, in the system according to the present embodiment, as in the first embodiment, the stereoscopic position information of the object is converted into the MPEG-specified user data area or private stream, or the information body of another area in a predetermined time unit. By describing and linking the object identification information with the identification information of the streams of both the stereoscopic video data and the audio data in a predetermined time unit, the stereoscopic video data is multiplexed with the audio data and packetized. Can be sent.

  (Fourth Embodiment) Next, a stereoscopic video for reproducing the stereoscopic video / stereoacoustic recording information created by the stereoscopic video / stereoacoustic recording system of the above-described embodiment and recorded on the recording medium 17 is reproduced. A stereophonic sound compatible playback system will be described with reference to FIG. The reproduction system according to the present embodiment includes a reproduction unit 31, an information separator 32, a video decoder 33, a position information extraction unit 34, an audio decoder 35, a visual field converter 36, a stereoscopic image display 37, a sound source selector 38, A sound image position controller 39 and a speaker array 40 are provided.

In this stereoscopic video / stereoscopic reproduction system, the data multiplexed from the recording medium 17 is read by the reproduction device 31 and transmitted to the information separator 32. The information separator 32 separates the video signal and audio signal packets, and transmits the video signal to the video decoder 33 and the audio signal to the audio decoder 35. The video decoder 33 decodes the video and simultaneously transmits the user data of the video picture layer to the position information extractor 34. The position information extractor 34 extracts depth information, object position information, and object ID information recorded in the format of FIG. 4 from user data. The object position information and the object ID information are transmitted to the sound source selector 38. The depth information and the decoded video signal are transmitted to the visual field converter 36. The visual field converter 36 generates a parallax image corresponding to the stereoscopic display method of the stereoscopic image display 37 from the depth information and the decoded two-dimensional image. When generating the parallax image, a visual field conversion method is used as a conversion method of the coordinate system in CG. This is to obtain an image with a different viewpoint from the expression for conversion to the viewpoint coordinate system. If there is depth information, it can be generated from any viewpoint. For example, the coordinates of the viewpoint _{(x i, y i, z} i), the coordinates of the gazing point _{(x a, y a, z} a) and. In addition, the distance between the viewpoint and the gazing point is (x _f , y _f , z _f ).

First, move the position of the origin by parallel movement. This conversion is assumed to be T1. The transformation T1 is simply a transformation that translates (−x _a , −y _a , −z _a ). Next, the direction of the coordinate value is changed by rotation. Vector from point O _a of the point O _f direction as shown in FIG. 10 is rotated by the y-axis is first α angle vector of the z-axis from the point O _a, then allowed to rotate in only x-axis β corner. The direction of rotation is reversed so actually move the coordinates of the point O _f.

It becomes. Here, since α is the angle of the foot and O _a obtained by projecting the O _f the xy plane,

It becomes. The β length between _O a _{O f}

And the length y _f between O _f O _{f ′}

It becomes.

Last conversion is performed z-axis from the coordinate system such that the front with respect to the xy plane, the eye direction relative to the xy plane, i.e. the transformation T ₄ to allow the other side is positive. This is simply z → -z. By multiplying these four conversion matrices T _{1 to} T _4, the viewpoint coordinate conversion matrix is

It becomes.

  In IP (Integral Photography), an element image corresponding to a plurality of lens arrays is captured by a camera corresponding to the lens position. At the same time, it is calculated and generated using the conversion matrix of the viewpoint coordinates. The stereoscopic image data generated in this way is transmitted to the stereoscopic image display 37 to perform stereoscopic image reproduction. Note that a binocular stereoscopic display method using a parallax barrier may be employed as the stereoscopic display method of the stereoscopic image display 37. In that case, α that can be set according to the viewing distance is substituted into the above equation, and β and γ are set to 0, so that an image having parallax for the right eye and the left eye can be generated.

  Here, a typical parallax barrier method and an IP method among the stereoscopic image display methods will be described. The parallax barrier method can be realized by liquid crystal. As shown in FIG. 11, this is a laminate of two liquid crystal panels 101 and 102. One liquid crystal panel 101 has a thin slit-like opening, and an appropriate interval is provided on the liquid crystal panel 102 on the back side. In this case, images for the left (L) and right (R) two eyes can be alternately arranged, and the images separated from the right eye and the left eye can be perceived when viewed from the predetermined viewpoints 103L and 103R through the slit-shaped opening. Is. As a result, different images can be input to the right eye and the left eye, and can be perceived as a stereoscopic image. A backlight 104 is provided to illuminate the image on the liquid crystal panel 102.

  However, in the case of the parallax barrier method, although the focus of the eyes is always on the liquid crystal screen, the image is felt in a place different from this position, which is accompanied by physiological unnaturalness. However, it has been pointed out that users are prone to fatigue and video sickness. In recent years, there are four physiological factors of stereoscopic vision, convergence contradiction (contradiction of convergence point and focus position) (binocular parallax) = When looking at an object, the human right and left eyes capture two different images seen from different directions, focus adjustment = “The thickness of the lens is controlled as the distance from the subject changes. The nature of changing the thickness of the lens, convergence = distant, close change, accompanied by the movement of the eyeball rotating inward or outward, motion parallax = the user himself Such a manner as to satisfy the property) to see the difference in the image due to changing the angle of view Italy has also been proposed. Among them as promising, method Lippmann announced in 1908 is an IP system.

  In the IP method, depth information of an object is acquired using a two-dimensionally arranged lens array (also referred to as a fly-eye lens, a fly-eye lens, or a compound eye lens). In the 1990s, a technology for generating moving images by the IP method was developed by replacing electronic recording with a conventional photographic dry plate. Further, a refractive index distribution lens array (GRIN) was developed by a researcher of the same document. (It is also referred to as a lens array) and a high-vision camera is used to capture a subject and acquire an element image group, and each image is transmitted and displayed on a liquid crystal display in real time and is imaged in space by a fly-eye lens. Successful, the feasibility of 3D television broadcasting by the IP system was shown (Non-Patent Document 3). FIG. 12 illustrates the principle of this IP system. As shown in FIG. 12A, a GRIN lens array 110 in which a large number of minute element lenses are arranged at the time of photographing is used, and the light of each minute element lens of the GRIN lens array 110 is used. The light is collected by the condensing lens 111 and one pixel of the micro camera 112 captures an image of light in one direction. Then, when reproducing, the image 120 from the camera is reproduced on a display 121 such as an LCD as shown in FIG. 12B, and one point of all the minute cameras are gathered to form a reproduced image viewed from one direction as a whole. .

  By using the lens array 122 in which micro lenses are arranged two-dimensionally, it is possible to create horizontal and vertical motion parallax, and it is also possible to have parallax only in the horizontal direction if arranged in the horizontal direction. In this method, a plurality of element images seen through a plurality of lenses are created by converting the viewpoints of the element images based on the depth information, and the element images are arranged, as if they were captured in FIG. 12A. As described above, stereoscopic image reproduction is realized by displaying the element image array on the LCD 121.

  On the other hand, audio is transmitted to the sound source selector 38 after the audio decoder 35 decodes the compressed audio data of a plurality of sound sources. The sound source selector 38 receives the object position information and the object ID information from the position information extractor 34 described above, and links to the object ID in accordance with the object position information existing in the image. The decoded sound source is selected, and the selected sound source data and object position information are paired and transmitted to the sound image position controller 39. The sound image position controller 39 controls the localization of an audio sound source linked to each sound source according to the position of the image object, using a sound image localization control method using a speaker array 40 described later. Audio data corresponding to a plurality of speakers as a result of localization control for each sound source is obtained as a sound source of a plurality of objects corresponding to one speaker. All of these are linearly added, the gain is adjusted, and the audio data output from one speaker is transmitted to the speaker array 40 as one. The speaker array 40 outputs the transmitted audio data.

  Here, the sound image localization control method will be described. Here, using a speaker ray, the delay according to the difference between the route from the center of the speaker ray to the focal point and the route from each speaker to the focal point so as to locally increase the sound pressure near a focal point in space. Sound image localization is realized by a reproduction signal given a quantity. The principle will be described with reference to FIG. First, the speaker array 40 is constructed by assembling speakers in an array as shown in FIG. A delay circuit 131 is provided for each speaker. When the delay is set by using the delay circuit 131 in the known method to focus on the vicinity of the listening position, the sound pressure component generated at the focal point is much higher than the direct sound from the speaker at the listening position. It is possible to play back. By using this principle and continuously controlling in real time, the localization of the sound image is controlled by linking to the position of the object of the stereoscopic moving image.

  Through this series of data processing, the image is reproduced as a three-dimensional image, and the object of the image in the image is linked to the three-dimensional movement to control the position of the sound source of the audio data, as if it were three-dimensionally viewed. The sound is localized and reproduced as if the sound emitted by the object is heard from the space. That is, as shown in FIG. 14, a stereoscopic video 120 is reproduced using the LCD 121, and a stereoscopic video is constructed from the element images using, for example, the IP lens array 122, and then the speaker array 40 is set. The sound image localization is performed so that the sound source of the object is heard from the position of the object in a form linked to a stereoscopic image object such as a car or airplane that is perceived as coming out of the display 121. And play.

  Next, reproduction processing by the stereoscopic video / stereoscopic reproduction system using the IP system will be described with reference to the flowchart of FIG. First, the multiplexed data is read from the recording medium 17 (step R1). Next, MPEG separation processing is performed (step R2). Of the separated information, video data is subjected to video data decoding processing, and audio data is subjected to audio data decoding processing (steps R3 and R9). Next, the object ID and position information are separated from the user data of the decoded video data (step R4). Next, depth information for each macroblock is detected from the separated information (step R5). Further, the ID of the object is detected (step R6).

  On the other hand, the object ID described in the stream ID of the audio data is detected from the audio data decoded in step R9 (step R10). Next, the audio data ID linked to the video object is selected by collating the IDs of the plurality of audio objects with the video ID (step R11).

  Next, the video uses the depth information held for the pixels in each macroblock to perform visual field conversion of the video image (step R7). For this, an image in which the viewpoint is changed by a conversion formula to the viewpoint coordinate system is obtained. Next, stereoscopic display of video is performed by the IP method (step R8). As this stereoscopic display method, a binocular stereoscopic display method using a parallax barrier can also be used.

  On the other hand, for each of a plurality of sound sources in which the selected link relation is clarified, the audio is controlled by the sound image position controller 39 using, for example, a sound image localization control method using a speaker array 40, for each sound source. In accordance with the position of the image object, the localization of the audio sound source linked to the image object is controlled (step R12). Next, the audio data corresponding to a plurality of speakers as a result of localization control for each sound source by the speaker array 40 is linearly added, and the gain is adjusted so that one audio data is output from one speaker. Output from the speaker array 40 (step R13).

  In this manner, in the stereoscopic video / stereoscopic playback system of the present embodiment, the stereoscopic video 120 is reproduced using the LCD 121, and the stereoscopic image is formed from the element image using the IP lens array 122. Then, the sound source of the object is linked to a stereoscopic image object such as a car or an airplane that is perceived as coming out of the display 121 by the speaker array 40 set after that, as if the sound source of the object is The sound image can be localized and reproduced so that it can be heard from the position.

(Fifth embodiment)
Next, a stereoscopic video / stereoscopic reproduction system according to a fifth embodiment of the present invention will be described with reference to FIG. Although the final information is reproduced from the recording medium 17 in the stereoscopic image / stereoscopic reproduction system of the fourth embodiment shown in FIG. 9, the reproduction system of the present embodiment Receives packet information that has been packetized peculiar to broadcasting, and receives and reproduces packets of stereoscopic video / audio data from the broadcast or communication network via the communication (broadcast) packet release unit 31 'as shown in FIG. To do. The data whose packet has been canceled by the packet canceller 31 'is the same as the reproduction data from the recording medium 17 in FIG. 9, and therefore the components after the information separator 32 and their processing functions are shown in FIG. This is the same as that of the fourth embodiment.

  As a result, even in the stereoscopic video / stereoscopic playback system of the fifth embodiment, the stereoscopic video image received via the communication network or broadcast is played back using the LCD 121 shown in FIG. A three-dimensional image is constructed from element images using the array 122, and then linked to a three-dimensional object such as a car or airplane that is perceived as appearing in front of the display 121 by the speaker array 40 set thereafter. In this way, the sound source of the object can be reproduced with sound image localization so that it can be heard from the position of the object.

  In addition, this invention is not limited to said embodiment, The following deformation | transformation aspects are possible. In the above-described embodiment, the stereoscopic video method is described as the IP method. However, the stereoscopic video method may be a parallax barrier method, a lenticular lens method, a super multi-view method, a binocular method using deflecting glasses, an anaglyph, or the like. Whatever. In addition, the sound image position control method has been described with the speaker ray method as the sound image localization control method, but even a method that can realize a virtual sound field space, for example, a binaural transoral method, is represented by a wave represented by Kirchoff-Helmhotz differential equations. A method using a sound field control method using acoustic theory may be used.

  In addition, it is possible to transmit such data via any transmission media such as communication and broadcasting without recording stereoscopic video / audio data on the recording media. It can also be used. The playback device can also be used as a receiving device.

  The recording medium on which the signal data of the present invention is recorded has a media-specific effect that the object position information and the object ID information are recorded, and preferably realizes a system for reproducing a three-dimensional video or a three-dimensional sound field. Can do. In addition, the definition of “media” in a recording medium includes not only a narrowly-defined medium that can record data, but also electromagnetic waves and light for transmitting signal data. The information recorded on the recording medium includes data itself such as an electronic file in a state where it is not recorded.

  Furthermore, in the above-described embodiment, the video depth information has been described as being recorded for each picture, but it may be about every 0.5 seconds or about every 1 second. In that case, it can be realized by using user data of the GOP layer of MPEG. The object ID may be 8 bits or 16 bits. The object may be a video object or a region. Some algorithms may contain errors in detecting objects, but they may be ignored. Furthermore, the region may not be specified by a closed curve. In addition, if the sound localization has a localization effect even slightly compared to normal stereo reproduction, it is considered that the localization is controlled.

  In addition, although the depth information and the object ID information have been described using MPEG video user data in the above embodiment, they may be recorded in a private stream defined by the MPEG system layer. In this case, since data is synchronized with time for each picture or for each of a plurality of GOPs, it is synchronized with images and sound data using PTS (Presentation Time Stamp), so that the private stream stream format of the private stream defined by MPEG is used. Is desirable. In addition, the information in FIG. 4 is an encoded audiovisual data program in the same area as the audiovisual data, that is, as a separate file on the recording medium, separately from the MPEG stream, with the structure of FIG. 4 as it is. Each may be recorded in a named file. In that case, it is also possible to record in the picture (field or image) order of the reproduction order (= input image order) or the picture (field or image) order of the encoding order in MPEG. The file name may be a number or an ASCII character as long as the program can be identified. Further, each file may be a single file that combines the entire media, or a playlist that combines several programs.

  Furthermore, the functions of the apparatus described above may be realized by a computer by a program. The program may be recorded on a recording medium, read from the recording medium, and loaded into a computer, or transmitted through a communication network into the computer.

1 is a functional block diagram of a recording apparatus for stereoscopic video / stereoscopic sound according to a first embodiment of the present invention. The block diagram of the principal part of the recording apparatus corresponding to the three-dimensional video / stereo sound of the first embodiment. Explanatory drawing of the epipolar constraint conditions for calculating | requiring the general parallax of 2 screens. The format explanatory drawing of depth information, object ID information, etc. FIG. Explanatory drawing of the installation method of the microphone which records the whole circumference. The flowchart figure which shows the stereo image / stereo sound correspondence recording process by the stereo image / stereo sound correspondence recording apparatus of the 1st Embodiment of this invention. The block diagram which shows the three-dimensional video / stereo sound corresponding recording apparatus of the 2nd Embodiment of this invention. The block diagram which shows the transmission apparatus corresponding to the stereo image / stereo sound of the 3rd Embodiment of this invention. The block diagram which shows the reproduction | regeneration apparatus corresponding to the three-dimensional video / stereo sound of the 4th Embodiment of this invention. Explanatory drawing of general viewpoint conversion. Explanatory drawing of a general parallax barrier system. Explanatory drawing of a general IP system. (1) An explanatory diagram of a general IP system. (2) Explanatory drawing of the array speaker used in the 4th Embodiment of this invention. The block diagram of the system of a speaker array and IP stereoscopic video system by the said 4th Embodiment. The flowchart figure which shows the three-dimensional video and three-dimensional sound reproduction | regeneration processing by the said 4th Embodiment. The block diagram which shows the reproduction | regeneration system corresponding to the three-dimensional video / stereo sound of the 5th Embodiment of this invention. FIG. 3 is an explanatory diagram of an MPEG video stream video layer. FIG. 2 is an explanatory diagram of an MPEG multiplexed transport stream system layer. (1) An explanatory diagram of an MPEG multiplexed transport stream system layer. (2) Explanatory drawing which shows Object_ID_Descriptor which describes new object ID employ | adopted by the 1st Embodiment of this invention.

Explanation of symbols

1 Camera A
2 Camera B
3 Disparity vector extractor 4 Depth distance calculator 5 Object analyzer 6 Horizontal position analyzer 7 Vertical position analyzer 8 Depth position analyzer 9 Position information formatter 10 CPU
DESCRIPTION OF SYMBOLS 11 Multiple microphone group 12 Sound source selector 13 Audio compressor 14 Video compressor 15 Information multiplexer 16 Recorder 17 Recording medium 31 Regenerator 32 Information separator 33 Video decoder 34 Position information extractor 35 Audio decoder 36 View conversion 37 Stereoscopic image display 38 Sound source selector 39 Sound image position controller 40 Speaker layout

Claims (10)

Creating horizontal position information, vertical position information, and depth position information of the object as the three-dimensional position information of the object for which sound localization control is performed using the sound source as a sound source during reproduction in a predetermined time unit;
Linking the stereoscopic position information of the object with the appearance timing of the object in the stream of both stereoscopic video data and audio data;
The stereoscopic video data and the acoustic data are recorded on a recording medium, and the stereoscopic position information of the object is recorded on the recording medium by linking with the appearance timing of the object in both the stereoscopic video data and the acoustic data streams. A recording program for stereoscopic video / stereo sound that causes a computer to execute steps. 3D image data and audio data are read out from the recording medium that records the 3D position information and the link information of the appearance timing of the object that performs stereo localization control of the sound using itself as a sound source, together with the 3D image data and audio data. Step to play and
Reading three-dimensional position information of the object from the recording medium;
Reading out acoustic data corresponding to the three-dimensional position information of the object from the recording medium;
A three-dimensional image that causes a computer to execute sound image position control using at least two or more speakers so that the three-dimensional position of the sound image of the acoustic data corresponding to the three-dimensional position information of the object becomes the three-dimensional position of the object. Video / stereo sound compatible playback program. Means for creating horizontal position information, vertical position information, and depth position information of the object as the three-dimensional position information of the object for which sound localization control is performed using the sound source as a sound source during reproduction in a predetermined time unit;
Means for linking the stereoscopic position information of the object with the appearance timing of the object in the stream of both stereoscopic video data and audio data;
The stereoscopic video data and the acoustic data are recorded on a recording medium, and the stereoscopic position information of the object is recorded on the recording medium by linking with the appearance timing of the object in both the stereoscopic video data and the acoustic data streams. And a stereoscopic video / stereoscopic recording device. 3D image data and audio data are read out from the recording medium that records the 3D position information and the link information of the appearance timing of the object that performs stereo localization control of the sound using itself as a sound source, together with the 3D image data and audio data. And means to play
Means for reading out the three-dimensional position information of the object from the recording medium;
Means for reading out acoustic data corresponding to the three-dimensional position information of the object from the recording medium;
A stereoscopic image comprising: means for performing sound image position control using at least two or more speakers so that the stereo localization position of the sound image of the acoustic data corresponding to the stereo position information of the object becomes the stereo position of the object; Stereo sound compatible playback device.   3D image data and audio data are recorded, and the 3D position information of the object for which stereo localization control is performed using the sound source as a sound source at the time of reproduction created in a predetermined time unit is stored in both the 3D image data and the audio data. A stereoscopic video / stereo sound compatible recording medium recorded by linking with the appearance timing of the object in the stream. The horizontal position information, vertical position information, and depth position information of each object as three-dimensional position information of an object that is played back as a sound source and displayed on the video and an object that is played back as the sound source but not displayed on the video in a predetermined time unit. The steps of creating
Setting identification information for each of the objects;
Linking the identification information of each object to the appearance timing of the object in the stream of both stereoscopic video data and audio data;
The stereoscopic video data and the acoustic data are recorded on a recording medium, and the stereoscopic position information of the object is recorded on the recording medium by linking with the appearance timing of the object in both the stereoscopic video data and the acoustic data streams. A recording program for stereoscopic video / stereo sound that causes a computer to execute steps. Along with the 3D video data and the audio data, the 3D position information and the information indicating the appearance timing of the object that is reproduced as a sound source at the time of reproduction and the object that is displayed as the sound source and the object that is reproduced as the sound source but not displayed in the video Reading and reproducing stereoscopic video data and audio data from a recording medium recorded in correspondence with the identification information; and
Reading identification information and stereoscopic position information of an object to be displayed and an object not to be displayed on the video from the recording medium;
Reading out acoustic data corresponding to the identification information of each object from the recording medium;
Performing sound image position control using at least two or more speakers so that the stereo localization position of the sound image of the acoustic data corresponding to the identification information of each object of the stereoscopic video data becomes the stereo position of each object. 3D / Sound reproduction program to be executed. The horizontal position information, vertical position information, and depth position information of each object as stereoscopic position information of an object that is played back as a sound source and displayed on the video and an object that is played back as the sound source but not displayed on the video in a predetermined time unit. And a means to create
Means for setting identification information for each of the objects;
Means for linking the identification information of each object to the appearance timing of the object in the stream of both stereoscopic video data and audio data;
The stereoscopic video data and the acoustic data are recorded on a recording medium, and the stereoscopic position information of the object is recorded on the recording medium by linking with the appearance timing of the object in both the stereoscopic video data and the acoustic data streams. And a stereoscopic video / stereoscopic recording device. Along with the 3D video data and the audio data, the 3D position information and the information indicating the appearance timing of the object that is reproduced as a sound source at the time of reproduction and the object that is displayed as the sound source and the object that is reproduced as the sound source but not displayed in the video Means for reading and reproducing stereoscopic video data and audio data from a recording medium recorded in correspondence with the identification information;
Means for reading identification information and three-dimensional position information between an object to be displayed on the video and an object not to be displayed from the recording medium;
Means for reading out acoustic data corresponding to identification information of each object from the recording medium;
Means for performing sound image position control using at least two or more speakers so that the stereo localization position of the sound image of the acoustic data corresponding to the identification information of each object of the stereoscopic video data becomes the stereo position of each object. 3D video / stereo sound compatible playback device. Three-dimensional video data and audio data are recorded, and the three-dimensional position information and identification information of the object that reproduces the sound source and displays the video and the sound source that reproduces the sound source but does not display the video, created in a predetermined time unit, A stereoscopic video / stereo acoustic compatible recording medium recorded by linking with the appearance timing in the stream of both the stereoscopic video data and the acoustic data of each object.

Images