JP4955718B2 - Stereoscopic display control apparatus, stereoscopic display system, and stereoscopic display control method - Google Patents

Description

  The present invention relates to a stereoscopic display control apparatus, a stereoscopic display system, and a stereoscopic display control method that allow an observer to clearly grasp where a portion that appears to pop out in a stereoscopic image.

  In Patent Literature 1, when switching from planar display to stereoscopic display, when switching from stereoscopic display to planar display, etc., via a stereoscopic display mode image created from a planar image or a planar display mode image created from a stereoscopic image, A configuration for performing transition processing is described.

  In Patent Document 2, when switching the display from a 3D image to a 3D image, or when switching the display from a 3D image to a 3D image, the pixel value ratio of the currently displayed image data is gradually decreased over a predetermined time to finally In other words, a configuration of 0% is disclosed.

  Patent Document 3 is intended for moving images having sound source position information of the original sound field, and is composed of a flat display and a group of transparent flat speakers arranged on the front surface thereof. The speaker is designated based on the sound source position information of the original sound field. A configuration for performing sound reproduction is disclosed.

  In Patent Document 4, the size, position of a person displayed on the screen, the illuminance in the room, and the position where the sound is output are controlled based on the control of the display, camera, speaker, microphone, etc. in the bidirectional communication system. A configuration is disclosed in which playback is performed by matching the location and the playback location.

JP 2005-136541 A JP 2005-109568 A JP 2007-27846 A JP 2008-5122 A

  The configurations of Patent Documents 1 and 2 can suppress the viewer's uncomfortable feeling when suddenly being displayed in 3D by gradually switching from flat display to 3D display over time by transition processing. As a result of such a transition process, there is a problem that the observer cannot grasp where the three-dimensional image appears to pop out. That is, during stereoscopic display, the viewer may not be able to clearly grasp the near side of the stereoscopic image, which may cause a problem of lack of stereoscopic effect. In the worst case, the stereoscopic image is not recognized as a stereoscopic image.

  Further, in the configuration of Patent Document 1, it is necessary to create a stereoscopic display mode image or a flat display mode image.

  In addition, the configurations of Patent Documents 3 and 4 are intended for moving images with sound source position information of the original sound field, and by using the sound source position information of the original sound field as it is, the realism to the original sound field is produced. Therefore, when switching from planar display to stereoscopic display, the effect of causing the observer to clearly understand where the stereoscopic image appears to pop out cannot be expected.

  The present invention has been made in view of such circumstances, and a stereoscopic display control apparatus, a stereoscopic display system, and a stereoscopic display control capable of allowing an observer to clearly grasp where a portion that appears to pop out of a stereoscopic image is present. It aims to provide a method.

  In order to achieve the above object, the present invention provides a stereoscopic display control device that performs stereoscopic display with sound reproduction using a display device and a plurality of sound reproducing devices, and obtains stereoscopic images and sound data. And a parallax amount detecting means for detecting a parallax amount of each region of the stereoscopic image, and a sound source position for determining a sound source position of a reproduction sound field on the display device based on the parallax amount of each region of the stereoscopic image The display unit stereoscopically displays the stereoscopic image on the display device, and sets the real or virtual sound source position of the reproduction sound field at the sound source position on the display device determined by the sound source position determination unit, There is provided a stereoscopic display control device comprising: a reproduction control means for reproducing the sound data as sound by the sound reproduction device.

  According to the present invention, since the sound source position of the reproduction sound field is set on the display device based on the parallax amount of each region of the stereoscopic image, it is clear to the observer where the portion that appears to pop out of the stereoscopic image. It becomes possible to grasp.

  In one aspect of the present invention, the sound source position determination unit divides the stereoscopic image into a region with a large amount of parallax and a region with a small amount of parallax, and corresponds to the region with a large amount of parallax of the stereoscopic image. The position on the display device to be determined is determined as the sound source position.

  According to this configuration, the observer can recognize a region where the stereoscopic image appears to pop out.

  In the aspect of the present invention, the reproduction control unit moves the sound source position on the display device when the stereoscopic image includes a plurality of regions having a large amount of parallax.

  According to this configuration, it is easy to recognize a region with a large amount of parallax due to movement of the sound source position.

  In the aspect of the invention, the sound source position determination unit may determine a region having the largest amount of parallax among regions obtained by dividing the stereoscopic image into a plurality of regions, and determine the region having the largest amount of parallax of the stereoscopic image. The corresponding position on the display device is determined as the sound source position.

  According to this invention, the observer can recognize the most visible area of the stereoscopic image.

  Further, in one aspect of the present invention, the reproduction control unit reproduces sound continuously before and after the switching when switching from planar display to stereoscopic display and when switching a stereoscopic image to be stereoscopically displayed, and for a certain period of time, Sound reproduction is performed using the sound source position determined based on the parallax amount of each region of the stereoscopic image.

  According to this configuration, it is possible to allow the observer to recognize a portion that appears to pop out of the stereoscopic image without stopping sound output.

  Moreover, in one aspect of the present invention, the reproduction control means sets the volume of the sound reproduction device corresponding to the sound source position on the display device among the plurality of sound reproduction devices more than the volume of other sound reproduction devices. Enlarge.

  In one embodiment of the present invention, the plurality of sound reproducing devices are configured by a group of sound reproducing devices arranged two-dimensionally on the front surface or the back surface of the display device.

  In one embodiment of the present invention, the sound reproducing device is a transparent planar sound reproducing device, and is disposed on the front surface of the display device.

  According to this configuration, it becomes easier for the observer to specify the position of the sound source in the reproduction sound field, compared to the case where the sound reproduction device is arranged on the back surface of the display device. Moreover, since it is planar, the display device and the sound reproducing device can be integrated and manufactured thinly.

  In one embodiment of the present invention, the plurality of sound reproducing devices are arranged around the display device.

  In one aspect of the present invention, the plurality of sound reproduction devices are arranged separately from the display device, and the reproduction control unit performs sound image localization using the sound source position as a virtual sound source position of a reproduction sound field. .

  Further, in one aspect of the present invention, the plurality of sound reproduction devices are configured by headphones or earphones worn on a human head, and the reproduction control means uses the sound source position as a virtual sound source position of a reproduction sound field. Perform sound image localization.

  The present invention also provides a stereoscopic display system comprising the stereoscopic display control device, the display device, and the sound reproduction device.

  The present invention is also a stereoscopic display control method for performing stereoscopic display with sound reproduction using a display device and a plurality of sound reproducing devices, a data acquisition step for acquiring a stereoscopic image and sound data, A parallax amount detecting step of detecting a parallax amount of each region; a sound source position determining step of determining a sound source position of a reproduction sound field on the display device based on the parallax amount of each region of the stereoscopic image; and the display The stereoscopic image is stereoscopically displayed by a device, and a real or virtual sound source position of a reproduction sound field is set to the sound source position on the display device determined by the sound source position determination unit, and the sound reproduction device performs the And a playback step of playing back sound data as sound.

  According to the present invention, it is possible to make an observer clearly understand where a portion of a stereoscopic image that appears to pop out is.

Block diagram showing an example of a stereoscopic display system to which the present invention is applied Explanatory drawing which shows the example of a structure of the display apparatus in which a three-dimensional display is possible Explanatory drawing used to explain the relationship between subject distance and parallax amount (A) is an explanatory diagram when there is no difference in the position of corresponding points between the right eye image and the left eye image, and (B) is an explanation when there is a large difference in the position of corresponding points between the right eye image and the left eye image. (C) is an explanatory diagram showing the amount of parallax Explanatory drawing which shows the right eye image and left eye image which were obtained by imaging a cup Explanatory drawing which shows the captured image and distance image of a cup (A) and (B) are a front view and a side view of a display device and a sound reproduction device when a two-dimensional array of sound reproduction devices is arranged on the back surface of the display device and a region with a large amount of parallax is set as a sound source position. (A) and (B) are a front view and a side view of the display device and the sound reproducing device when a two-dimensional array of sound reproducing devices is arranged on the back surface of the display device and the region having the largest amount of parallax is set as the sound source position. (A) and (B) are a front view and a side view of a display device and a sound reproduction device when a two-dimensional array of sound reproduction devices is arranged on the front surface of the display device and a region with a large amount of parallax is set as a sound source position. The flowchart which shows the flow of the stereoscopic display control process to which this invention is applied. (A) and (B) are a front view and a side view of a display device and a sound reproducing device in a planar display. (A)-(C) is explanatory drawing which shows the case where the sound reproduction apparatus is arrange | positioned around a display apparatus. (A) is a front view showing an example of a single display, (B) is a front view showing an example of a multi-display.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a block diagram showing an example of a stereoscopic display system to which the present invention is applied.

  In FIG. 1, the stereoscopic display system 100 includes a data storage unit 22, a display device 24, a sound reproduction unit 25 (speaker group), and a stereoscopic display control device 30.

  The data storage unit 22 is storage means for storing data such as images (image data) and sound data. For example, a non-volatile storage device such as a memory card, hard disk, or optical disk is used.

  The display device 24 is configured by a display device that can stereoscopically display a stereoscopic image obtained by imaging from a plurality of viewpoints. For example, a display device such as an LCD (Liquid Crystal Display Device), a plasma display device, or an organic EL display device is used. The mechanism of stereoscopic display will be described later.

  The sound reproduction device 25 includes a plurality of sound reproduction devices. A specific example and arrangement of the sound reproducing device 25 will be described later.

  The stereoscopic display control device 30 according to the present embodiment includes an operation unit 31, a data acquisition unit 32, a display image memory 33, a display driver 34, a sound reproduction driver 35, and a CPU 40.

  The operation unit 31 is configured by a known input device such as a keyboard and a mouse.

  In this example, the data acquisition unit 32 acquires image and sound data from the data storage unit 22. In this example, acquisition is performed from the data storage unit 22, but image and sound data may be acquired via a communication network such as the Internet. Moreover, you may acquire an image and sound data directly with a known camera and microphone.

  The display image memory 33 is a display image-dedicated storage unit that temporarily stores image data to be displayed on the display device 24. For example, a high-speed accessible storage device such as a RAM is used.

  The display driver 34 is configured by a circuit that drives the display device 24.

  The sound reproduction driver 35 is configured by a circuit that drives the sound reproduction device 25.

  A CPU (Central Processing Unit) 40 controls each part of the stereoscopic display control device 30 and performs stereoscopic display with sound reproduction using the display device 24 and the sound reproducing device 25. Details of the functions of the CPU 40 will be described later. FIG. 2 is an explanatory diagram showing a structural example that enables stereoscopic display on the display device 24. This example is a lenticular method, and uses a lenticular lens 24a having a semi-cylindrical lens group. The lenticular lens 24a is arranged on the front surface of the display device 24 (observer's viewpoint, that is, the z-axis direction in which the left eye EL and the right eye ER are present), and a plurality of cylindrical convex lenses are connected in the x-axis direction in FIG. It consists of The display area of the stereoscopic image displayed on the display device 24 includes a right eye strip image display area 24R and a left eye strip image display area 24L. The strip image display area 24R for the right eye and the strip image display area 24L for the left eye each have an elongated strip shape in the y-axis direction in FIG. 2 of the screen, and are alternately arranged in the x-axis direction in FIG. Each convex lens constituting the lenticular lens 24a has a strip image display area including a pair of right eye strip image display areas 24R and a left eye strip image display area 24L with reference to a given observation point of the observer. It is formed at a position corresponding to the region 24c. In FIG. 2, the right eye strip image displayed in the right eye strip image display area 24 </ b> R of the display device 24 is incident on the observer's right eye ER by the light refraction action of the lenticular lens 24 a. Further, the left eye strip image displayed in the left eye strip image display area 24L of the display device 24 is incident on the left eye EL of the observer due to the photorefractive action of the lenticular lens 24a. Therefore, the observer's right eye sees only the right-eye strip image, and the observer's left eye sees only the left-eye strip image. Stereoscopic viewing is possible by the left-right parallax of the left-eye image that is a set of ophthalmic strip images.

  Although the case where the lenticular method is used as an example of the structure of the display device 24 for stereoscopic display has been described with reference to FIG. 2, the present invention is not particularly limited to the lenticular method.

  For example, a light direction control system (time-division light direction control back) that controls the direction of the backlight that illuminates the back surface of the LCD (liquid crystal display device) constituting the display device 24 in the right eye direction and the left eye direction of the observer in a time division manner. 3D display may also be performed using a light method. For the light direction control system, Kentaro Toyooka, Tetsuya Miyashita, Tatsuo Uchida, “Three-dimensional display using time-division light direction control backlight”, 2000 Proceedings of the Liquid Crystal Society of Japan, pp.137-138 (2000 ), And Japanese Patent Application Laid-Open No. 2004-20684.

  Self-emission or a method of controlling the amount of light with a separate light source may be used. In addition, a system such as a polarized light system, an anaglyph, and a naked eye system are not limited. Moreover, the system which laminated | stacked the liquid crystal and organic EL in the multilayer may be sufficient.

  The display device 24 may be any display device capable of both planar display and stereoscopic display in the present embodiment.

  Next, details of the functions of the CPU 40 will be described.

  In FIG. 1, the CPU 40 includes a main control unit 41, a parallax amount detection unit 42, a sound source position determination unit 43, an image reproduction control unit 44, and a sound reproduction control unit 45.

  The main control unit 41 controls the respective units inside and outside the CPU 40.

  The parallax amount detection unit 42 detects the parallax amount for each divided region obtained by dividing the stereoscopic image. In this example, a superposed image of the right eye image and the left eye image is used as a stereoscopic image, and a representative amount of parallax is calculated in each of the divided areas (for example, 7 × 7 divided areas) obtained by dividing the stereoscopic image. .

  First, the amount of parallax will be described with reference to FIGS. 3 and 4.

  In FIG. 3, the distance (base length) between the optical axes and the light between the right-eye imaging optical system 14 </ b> R and the left-eye imaging optical system 14 </ b> L in the camera 10 (imaging device) (not shown in FIG. 1). The angle (convergence angle) formed by the axes is assumed to be fixed. A subject image is formed on the right-eye imaging device 134R by the right-eye imaging optical system 14R, and a right-eye image is generated by the imaging device 134R. In addition, a subject image is formed on the left-eye imaging element 134L by the left-eye imaging optical system 14L, and a left-eye image is generated by the imaging element 134L. 4A to 4C, the right eye image and the left eye image have the same magnification.

  In FIG. 4A, the farther the subject is with respect to the plurality of photographing optical systems 14R and 14L (actually, the closer the subject is to the intersection of the optical axes of the two photographing optical systems 14R and 14L), FIG. As shown, the position of the corresponding point 200R of the subject in the right-eye image matches the position of the corresponding point 200L of the subject in the left-eye image (which is a point corresponding to the corresponding point 200R of the right-eye image). Come. Actually, when the subject is located at the intersection of the optical axes of the plurality of photographing optical systems 14R and 14L, the positions of the corresponding points 200R and 200L coincide in the right eye image and the left eye image. On the other hand, as shown in FIG. 4B, the closer the subject is to the plurality of photographing optical systems 14R and 14L in FIG. 3, the position of the corresponding point 200R of the subject in the right-eye image, The position of the corresponding point 200L of the subject in the left eye image is separated.

  As shown in FIG. 4C, the difference d between the positions of the corresponding points 200R and 200L between the right eye image and the left eye image corresponds to a “parallax amount” indicating the magnitude of binocular parallax in the stereoscopic image. . That is, as described above, when the baseline length and the convergence angle are fixed and the image magnification is equal, the “parallax amount” is determined by the subject distance. This “parallax amount” corresponds to an amount (amount of pop-out) that appears to be popped out by the observer when stereoscopic display is performed.

  The parallax amount detection unit 42 of the present example first determines the correspondence between the positions (coordinates) of the corresponding points 200R and 200L in the captured images (right eye image and left eye image) of each viewpoint and the binocular parallax amount d. Ask. Next, the coordinates (intermediate coordinates) of the intermediate position between the corresponding points 200R and 200L are obtained and associated with the parallax amount d.

  For example, when the cup 210 is imaged as shown in FIG. 5, the coordinates of the corresponding points (221R, 222R, etc.) on the right eye image 220R and the respective points on the left eye image 220L for at least the entire cup 210 as the main subject. Differences d1 and d2 from the coordinates of corresponding points (221L, 222L, etc.) are calculated. Next, the coordinates of the intermediate positions (the intermediate positions of 221R and 221L and the intermediate positions of 222R and 222L) are obtained from the combinations of the corresponding points (221R and 221L, 222R and 222L), and the coordinates and parallax of these intermediate positions are obtained. The quantities d1 and d2 are associated with each other. Then, based on the coordinates of the intermediate position, for each divided region (for example, a 7 × 7 divided region) on the superimposed image (stereoscopic image) obtained by superimposing the right eye image and the left eye image, the parallax amount is representative. A value (for example, the maximum value) is obtained.

  In FIG. 5, for convenience of illustration, only two sets of corresponding points 221R and 222R and corresponding points 221L and 222L are shown, but actually, the parallax amount is obtained over at least the entire cup 210 as the main subject.

  As a first mode for detecting the amount of parallax, there is a mode of calculating the amount of parallax from a stereoscopic image. For example, in the right eye image and the left eye image constituting the stereoscopic image, an edge whose color and / or luminance changes is detected, respectively, and based on the edge detected in the right eye image and the edge detected in the left eye image Then, the pixels on the right eye image and the pixels on the left eye image are associated with each other, and a position difference d between the associated pixels is calculated. Moreover, there exists an aspect which acquires a distance image as a 2nd aspect of parallax amount detection. Specifically, a distance image is acquired using a well-known distance image sensor at the time of imaging, and the stereoscopic image and the distance image are associated with each other and stored in the data storage unit 22. And at the time of reproduction | regeneration of a stereo image, a distance image is acquired from the data storage part 22, and the distance image is used as parallax amount information. For example, irradiate an infrared light pulse to an imaging target range, and correspond to the delay time from the irradiation of the infrared light pulse (infrared echo) reflected in the imaging target range to each pixel of the right eye image and the left eye image In addition, the subject distance corresponding to each pixel of the stereoscopic image is calculated by associating the measured delay time with the subject distance. That is, a distance image sensor generates a so-called distance image. FIG. 6 shows an example of a captured image 220 (right eye image or left eye image) of a flower-patterned cup and a distance image 230 corresponding to the captured image 220. The captured image 220 obtained by the imaging elements 134R and 134L is an image composed of pixels having pixel values (pixel information) indicating the color and luminance of each point on the subject, and corresponds to an image visually recognized by a person. On the other hand, the distance image 230 is an image composed of pixels having pixel values (distance information) indicating the distance to each point on the subject. Here, the pixel value (distance information) of the distance image is treated as the amount of parallax. In other words, the distance image is used as an image (parallax amount image) composed of pixels having pixel values indicating the parallax amount. In the distance image of FIG. 6, for the sake of convenience, the subject distance (parallax amount) is divided into three stages, but it goes without saying that it may be divided more finely. In addition, since the subject image differs between the right eye image and the left eye image due to different viewpoints, the distance image sensor corresponding to the right eye imaging optical system 14R and the left eye imaging optical system 14L are used. By providing the corresponding distance image sensor in the camera 10, a distance image corresponding to each captured image (right eye image, left eye image) is accurately obtained. Note that it is not necessary to associate the pixels of the distance image and the pixels of the stereoscopic image one by one. What is necessary is just to obtain | require a to-be-photographed object distance as a parallax amount for every two-dimensional block which consists of several pixels (for example, 3 pixels x 3 pixels) of a stereo image. Moreover, you may acquire a distance image using the electromagnetic waves (for example, ultrasonic wave) of another wavelength instead of infrared rays. When performing stereoscopic display, the parallax amount detection unit 42 acquires the distance image associated with the stereoscopic image (right eye image and left eye image) from the data storage unit 22 and refers to the pixel value of the distance image, thereby obtaining The amount of parallax for each pixel of the stereoscopic image is detected.

  In the first aspect described above, there is an advantage that the cost of the hardware of the camera can be reduced by acquiring the parallax amount based on the stereoscopic image itself. In the second aspect, the distance image sensor is used. By creating a distance image, there is an advantage that the amount of parallax can be easily acquired. The sound source position determination unit 43 determines the sound source position of the reproduction sound field on the display device 24 based on the amount of parallax for each divided region of the stereoscopic image. There are various modes for determining the sound source position, which will be described later.

  The image reproduction control unit 44 causes the display device 24 to display an image via the display driver 34. The image reproduction control unit 44 of this example can perform planar display of a planar image and stereoscopic display of a stereoscopic image. You may comprise so that switching between the three-dimensional display of a stereo image and planar display (only the image of a single viewpoint is displayed) is possible.

  The sound reproduction control unit 45 causes the sound reproduction device 25 to reproduce sound via the sound reproduction driver 35. When the sound reproduction device 25 is an analog signal input, the sound data is converted from a digital signal to an analog signal in the input format of the sound reproduction device 25 and output to the sound reproduction device 25. When the sound reproducing device 25 is a digital signal input, the sound data is converted into a digital signal in the input format of the sound reproducing device 25.

  The sound reproduction control unit 45 of the present embodiment sets the real or virtual sound source position of the reproduction sound field at the sound source position on the display device 24 determined by the sound source position determination unit 43, and the sound reproduction device 25 performs the sound reproduction. Play data as sound. Hereinafter, the setting of the real or virtual sound source position in the reproduction sound field may be referred to as “sound image localization”. There are various arrangements of the sound reproduction device 25 and sound image localization suitable for the arrangement, which will be described in detail later.

  In addition, when switching from flat display to stereoscopic display or when switching a stereoscopic image to be stereoscopically displayed, the sound reproduction control unit 45 of the present embodiment continuously reproduces sound before and after the switching, and for a certain period of time ( For example, for 3 seconds, the sound source position determination unit 43 sets the sound source position on the display device 24 determined based on the parallax amount of each divided area of the stereoscopic image as the real or virtual sound source position of the reproduction sound field, Perform playback.

  In the actual sound source position setting, when the sound reproduction device 25 is arranged on the front surface, the back surface, or the periphery of the display device 24, the sound source position on the display device 24 determined by the sound source position determination unit 43 or a position near the sound source position is determined. The sound is actually reproduced from the sound reproducing device 25. In the virtual sound source position setting, for example, when the sound reproduction device 25 is arranged away from the display device 24, if a sound source is present at the sound source position on the display device 24 determined by the sound source position determination unit 43, a stereoscopic image is displayed. Sound reproduction is performed so that the observer (who is the listener of the sound) feels. In this specification, both actual sound source position setting and virtual sound source position setting in the reproduction sound field are referred to as “sound image localization”.

  There are various arrangements of the sound reproducing device 25. First, there is a mode in which sound reproduction devices are arranged in a two-dimensional manner on the front surface or the back surface of the display device 24 to perform sound image localization. Secondly, there is an aspect in which sound image localization is performed by arranging a plurality of sound reproducing devices in a one-dimensional or two-dimensional manner around the display device 24. Third, there is an aspect in which a plurality of sound reproducing devices are arranged apart from the display device 24 to perform sound image localization. Fourthly, there is an aspect in which sound image localization is performed by wearing headphones or earphones on a person's head.

  The front view of FIG. 7A and the side view of FIG. 7B are a display device 24 and a group of sound playback devices 25 when a plurality of two-dimensional array of sound playback devices 25 are arranged on the back side of the display device 24. Indicates. The sound source position determination unit 43 divides a stereoscopic image (right-eye image and left-eye image) into a divided region where the amount of parallax is larger than the threshold and a divided region where the amount of parallax is equal to or smaller than the threshold. The position on the display device 24 corresponding to the divided area hatched in FIG. 7A is determined as the sound source position of the reproduction sound field. The sound reproduction control unit 45 is a sound that is hatched in the sound reproduction device 25 (FIG. 7B) located at or near the sound source position among the sound reproduction devices 25 two-dimensionally arranged on the back surface of the display device 24. Sound is output from the playback device.

  In this example, the case where the stereoscopic image is divided into 7 × 7 is shown as an example, but the number of divided regions is arbitrary. For example, the maximum value of the parallax amount in the divided area is compared with a threshold value.

  In this example, the display device 24 has a large number of holes. For example, in the display device 24 shown in FIG. 2, a hole is provided between the strip image regions 24R and 24L. Through these holes, sound generated by the sound reproducing device 25 arranged on the back surface of the display device 24 is propagated to the front surface side of the display device 24.

  8A and the side view (B) of FIG. 8, the divided area having the largest amount of parallax is determined from among the divided areas of the stereoscopic image. In this example, the heel stereoscopic image is divided into 7 × 7, and the divided area where the ridge scissors are present (the divided area hatched in FIG. 8A) is determined to have the maximum parallax amount. The position on the display device 24 corresponding to the divided area is determined as the sound source position, and the sound reproduction device 25 located at or near the sound source position among the sound reproduction devices 25 two-dimensionally arranged on the back surface of the display device 24. A sound is output from (a sound reproducing device with a hatched line in FIG. 8B).

  The front view of FIG. 9A and the side view of FIG. 9B show the display device and the sound reproduction device group when the two-dimensional array of sound reproduction device group is arranged on the front side of the display device.

  In this example, the sound reproduction device 25 group is a transparent and planar sound reproduction device 25, and an image on the display device 24 can be seen through. For example, a transparent diaphragm is provided on the front side of the display device 24, and the diaphragm is vibrated by an electric signal to output sound. Here, not all of the sound reproducing device 25 may be transparent, but at least a portion (for example, half or more) may be transparent. A known transparent speaker can be used as such a sound reproducing device.

  FIG. 10 is a flowchart showing a flow of an example of a stereoscopic display control process to which the present invention is applied. This process is executed according to the program by the CPU 40 of FIG.

  In step S <b> 1, the main control unit 41 detects an operation on the operation unit 31.

  In step S2, the main control unit 41 determines whether or not to perform reproduction based on the operation content. If the reproduction is not performed, the process returns to step S1, and if the reproduction is performed, the process proceeds to step S3.

  In step S3, the data acquisition unit 32 acquires image and sound data.

  In step S4, the main control unit 41 determines whether the acquired image is a stereoscopic image or a planar image. If it is a stereoscopic image, the process proceeds to step S6. If it is a planar image (if it is not a stereoscopic image), the process proceeds to step S5.

  If the image is a flat image, image reproduction by the image reproduction control unit 44 and sound reproduction by the sound reproduction control unit 45 are performed in step S9. In this example, all sound playback devices are selected. That is, as shown in the plan view of FIG. 11A and the side view of FIG. 11B, a planar image is displayed on the display device 24 and sounds (for example, background sounds) are reproduced from all the sound reproduction devices 25. . In this example, the sound reproducing devices 25 are two-dimensionally arranged on the back surface of the display device 24, and the sound reproducing device 25 on the back surface of the shaded area (all regions) on the display device 24 in FIG. Sound is output.

  If it is a stereoscopic image, the parallax amount of each divided area of the stereoscopic image is detected by the parallax amount detection unit 42 in step S6. For example, the distance image is acquired from the data storage unit 22 and the amount of parallax is detected. Alternatively, corresponding points corresponding to each other are detected in the captured images (right eye image and left eye image) of each viewpoint constituting the stereoscopic image, and the difference between the positions of the detected corresponding points on the stereoscopic image is calculated as the parallax amount. To do. Then, a representative value of the parallax amount is determined for each divided region.

  In step S7, the sound source position determination unit 43 determines a protruding portion of the stereoscopic image. For example, the divided area where the representative value of the parallax amount is maximum is determined, and the position on the display device 24 corresponding to the divided area is determined as the sound source position. A divided area where the representative value of the parallax amount is larger than the threshold value may be determined, and a position on the display device 24 corresponding to the divided area may be determined as the sound source position.

  In step S8, the sound reproduction control unit 45 sets a real or virtual sound source position in the reproduction sound field. In this example, the sound reproduction device 25 located at the sound source position (or the sound reproduction device 25 located near the sound source position) is selected.

  In step S <b> 9, the image reproduction control unit 44 displays a stereoscopic image on the display device 24, and the sound reproduction control unit 45 reproduces sound from the sound reproduction device 25. Here, sound is output only from the sound reproducing device 25 selected in step S8 for a certain period of time after the stereoscopic image is displayed.

  Immediately before the image switching, as shown in FIGS. 11A and 11B, the sound is output from the whole sound reproduction device 25 of the two-dimensionally arranged sound reproduction device group. As shown in FIGS. 8A and 8B, the sound is output only from the sound reproducing device 25 arranged at a location where the amount of parallax is large. FIGS. 8A and 8B show a case where sound is output only from the sound reproducing device 25 arranged at the place where the parallax amount is the largest, but the cases shown in FIGS. 7A and 7B are shown. As described above, sound may be output from a plurality of sound reproducing devices 25 arranged at a location where the parallax amount is larger than the threshold value.

  Thereby, the observer can recognize the stereoscopic effect of the switched stereoscopic image while the background sound (so-called BGM) is flowing.

  The flow of the reproduction control process has been described with reference to FIGS. 8 and 11, taking as an example the case where the sound reproduction device 25 is arranged on the back of the display device 24, but as shown in FIGS. 9A and 9B. In addition, even when the transparent planar sound reproducing device 25 is arranged in front of the display device 24, the reproduction control process may be performed in the same manner.

  When the sound reproducing device group is arranged on the back surface or the front surface of the display device 24 as described above, the sound reproduction control unit 45 determines the sound source position of the display device 24 (or a position in the vicinity thereof) determined by the sound source position determining unit 43. Sound reproduction is performed from the sound reproduction device 25 arranged in the. When the actual sound source position on the display device 24 is controlled in this way, it is possible to directly show a region where the parallax amount of the stereoscopic image is large to the viewer of the stereoscopic image. The sound output level (volume) of the sound reproducing device 25 corresponding to the sound source position may be made larger than that of other sound reproducing devices.

  Further, by integrating the display device 24 and the sound reproduction device 25, the indoor space can be effectively used.

  As shown in FIGS. 12A to 12C, a plurality of sound reproducing devices 25 may be arranged around the display device 24. The array may be one-dimensional or two-dimensional. In this case, the sound reproduction control unit 45 performs sound reproduction from the sound reproduction device closest to the sound source position of the display device 24 determined by the sound source position determination unit 43.

  There are various types of sound image localization, and a plurality of sound reproducing devices 25 may be arranged at positions separated from the display device 24 without being arranged on the front surface, the back surface, or the periphery of the display device 24. The sound reproduction control unit 45 performs sound image localization using the sound source position on the display device 24 determined by the sound source position determination unit 43 as a virtual sound source position. For such sound image localization, a well-known sound image localization technique used in a known sound field reproduction system can be used.

  For example, a person listens to two-channel sound with both left and right ears, and in a two-channel (stereo) reproduction sound field corresponding to this, sound image localization can be performed using two sound reproduction apparatuses. The reproduction sound field may be limited to the vicinity of the ears of the listener, and sound image localization may be performed using headphones or earphones worn on a person's head. Of course, a speaker may be used.

  In a multi-channel reproduction sound field, sound image localization can be easily performed to the side or rear of the listener. For example, in a 5.1 channel surround system, five sound reproducing devices arranged at the front right, front center, front left, rear right and rear left so as to surround the listener, and one sound reproducing device dedicated to the bass, The sound image can be localized well from the side of the listener to the rear. By using an indoor transmission path (for example, echo), it is also possible to localize the sound image behind the listener without arranging the sound reproducing device behind the listener.

  Specifically, sound image localization is performed by controlling a sound level difference, a time difference, a phase difference, and the like.

  As described above, according to the present invention, when switching from planar display to stereoscopic display, and when switching a stereoscopic image to be stereoscopically displayed, by using sound image localization, where a portion of the stereoscopic image appears to pop out. It is possible to make the observer know if there is any.

  However, the stereoscopic image displayed on the display device 24 may be gradually completed together with the sound image localization. Specifically, when switching from planar display to stereoscopic display and when switching a stereoscopic image to be stereoscopically displayed, the stereoscopic image is sequentially from a region with a large amount of parallax to a region with a small amount of parallax, or a stereoscopic image. Among them, information of a stereoscopic image to be given to the display device 24 is added in order from an area with a small amount of parallax to an area with a large amount of parallax.

  For example, the sound reproduction control unit 45 causes the display device 24 to sequentially display missing portions of the stereoscopic image from the state where the entire stereoscopic image is not displayed on the display device 24 or the state where the stereoscopic image is partially displayed. Go.

  In addition, for example, the sound reproduction control unit 45 causes the display device 24 to stereoscopically display a stereoscopic image in a single color, and then sequentially adds color information of the stereoscopic image to be given to the display device 24 to thereby convert the stereoscopic image. A single color (for example, monochrome) stereoscopic display is gradually changed to a color stereoscopic display.

  In addition, for example, the sound reproduction control unit 45 causes the display device 24 to stereoscopically display the stereoscopic image with a single luminance, and then sequentially adds luminance information of the stereoscopic image to be given to the display device 24, so that the stereoscopic image is displayed. Is gradually changed from a single luminance stereoscopic display to a multi-gradation luminance stereoscopic display.

  For example, the sound reproduction control unit 45 causes the display device 24 to display only a single viewpoint image constituting the stereoscopic image, and then sequentially adds other viewpoint images to be given to the display device 24. Thus, the stereoscopic image is gradually changed from the planar display to the stereoscopic display.

  For example, the sound reproduction control unit 45 causes the display device 24 to roughly display a stereoscopic image, and then changes each area of the stereoscopic image from the coarse display to the precise display.

  Note that the sound reproducing device is not particularly limited to the above-described example. For example, when arranged on the front surface or the back surface of the display device 24, the shape of the sound reproducing device 25 is not particularly limited to a planar shape. Moreover, when arrange | positioning around the display apparatus 24 and arrange | positioning away from the display apparatus 24, the shape and arrangement | sequence shape of a sound reproduction apparatus are not specifically limited. Further, for example, the headphones are not particularly limited to those worn on the ear, and for example, bone conduction headphones may be used.

  Further, the sound source position may be moved. For example, when the stereoscopic image includes a plurality of divided regions having a parallax amount larger than the threshold value, the sound source position is moved in the order of decreasing parallax amount or increasing parallax amount. That is, using the auditory characteristics of a person who can easily recognize the sound source position by moving the sound source position, the sound source position determination unit 43 moves the sound source position from the divided area where the parallax amount is small toward the divided area where the parallax amount is large. Alternatively, the sound source position may be moved from a divided region having a large amount of parallax toward a divided region having a small amount of parallax.

  In this case, the user can recognize a plurality of spots that appear to pop out (a plurality of divided areas with a parallax amount equal to or greater than a threshold) without the user recognizing the place that appears to pop out the most (divided areas with the largest amount of parallax). It only has to be made.

  The sound source position in the reproduction sound field may be moved by switching the sound reproduction device 25 that reproduces the sound and moving the actual sound source position or moving the virtual sound source position.

  In addition, the present invention does not exclude sound image localization of the reproduction sound field based on the sound source position of the original sound field, but obtains sound source position information of the original sound field from the data storage unit 22 and displays it for a certain time (several times). Sound image localization based on the amount of parallax only for a second) may be performed, and then sound reproduction may be performed by switching to sound image localization based on sound source position information of the original sound field.

  Further, the present invention is not particularly limited to the case where stereoscopic display is performed by one display device 24 as shown in FIG. As shown in FIG. 13B, a plurality of display devices 24a to 24d may be used to perform stereoscopic display using a multi-display equivalent to the single display shown in FIG. For example, a multi-display is used when displaying a large image in an exhibition hall or the like.

  The display control device is not particularly limited when controlling the reproduction of images and sounds. The present invention can be applied to various electronic devices such as a television and a personal computer as well as various portable terminals such as a digital camera and a mobile phone.

  The present invention is not limited to the examples described in the present specification and the examples illustrated in the drawings, and various design changes and improvements may be made without departing from the spirit of the present invention.

  DESCRIPTION OF SYMBOLS 22 ... Data storage part, 24 ... Display apparatus, 25 ... Sound reproduction apparatus, 30 ... Stereoscopic display control apparatus, 32 ... Data acquisition part, 40 ... CPU, 42 ... Parallax amount detection part, 43 ... Sound source position determination part, 44 ... Image reproduction control unit, 45 ... sound reproduction control unit, 100 ... stereoscopic display system

Claims (13)

A stereoscopic display control device that performs stereoscopic display with sound reproduction using a display device and a plurality of sound reproduction devices,
Data acquisition means for acquiring stereoscopic image and sound data;
Parallax amount detection means for detecting the parallax amount of each region of the stereoscopic image;
Sound source position determining means for determining a sound source position of a reproduction sound field on the display device based on the parallax amount of each region of the stereoscopic image;
The three-dimensional image is three-dimensionally displayed by the display device, and a real or virtual sound source position of a reproduction sound field is set to the sound source position on the display device determined by the sound source position determining means, and the sound reproduction device Reproduction control means for reproducing the sound data as sound,
A stereoscopic display control apparatus comprising:   The sound source position determination unit divides the stereoscopic image into a region where the amount of parallax is large and a region where the amount of parallax is small, and determines a position on the display device corresponding to the region where the amount of parallax of the stereoscopic image is large. The stereoscopic display control device according to claim 1, wherein the stereoscopic display control device is determined as a sound source position.   The stereoscopic display control apparatus according to claim 2, wherein the reproduction control unit moves the sound source position on the display apparatus when the stereoscopic image includes a plurality of regions having a large amount of parallax.   The sound source position determining means determines a region having the largest amount of parallax among regions obtained by dividing the stereoscopic image into a plurality of regions, and determines a position on the display device corresponding to the region having the largest amount of parallax of the stereoscopic image. The stereoscopic display control apparatus according to claim 1, wherein the stereoscopic display control apparatus determines the sound source position.   When switching from flat display to stereoscopic display or switching a stereoscopic image to be stereoscopically displayed, the reproduction control means continuously reproduces sound before and after the switching, and the parallax of each area of the stereoscopic image for a certain period of time. 5. The stereoscopic display control device according to claim 1, wherein sound reproduction is performed based on the sound source position determined based on a quantity.   The playback control means makes the volume of the sound playback device corresponding to the sound source position on the display device out of the plurality of sound playback devices larger than the volume of other sound playback devices. The stereoscopic display control apparatus according to any one of 1 to 4.   The plurality of sound reproducing devices are configured by a group of sound reproducing devices arranged in a two-dimensional manner on the front surface or the back surface of the display device. 3D display control device.   The stereoscopic display control device according to claim 7, wherein the sound reproduction device is a transparent planar sound reproduction device, and is disposed in front of the display device.   The three-dimensional display control device according to claim 1, wherein the plurality of sound reproduction devices are arranged around the display device. The plurality of sound reproducing devices are arranged apart from the display device,
6. The stereoscopic display control apparatus according to claim 1, wherein the reproduction control means performs sound image localization using the sound source position as a virtual sound source position of a reproduction sound field. The plurality of sound reproducing devices are configured by headphones or earphones worn on a person's head,
6. The stereoscopic display control apparatus according to claim 1, wherein the reproduction control means performs sound image localization using the sound source position as a virtual sound source position of a reproduction sound field.   A 3D display system comprising the 3D display control device according to any one of claims 1 to 11, the display device, and the sound reproduction device. A stereoscopic display control method for performing stereoscopic display with sound reproduction using a display device and a plurality of sound reproducing devices,
A data acquisition step for acquiring stereo image and sound data;
A parallax amount detection step of detecting a parallax amount of each region of the stereoscopic image;
A sound source position determining step for determining a sound source position of a reproduction sound field on the display device based on the parallax amount of each region of the stereoscopic image;
The three-dimensional image is three-dimensionally displayed by the display device, and a real or virtual sound source position of a reproduction sound field is set to the sound source position on the display device determined by the sound source position determining means, and the sound reproduction device A reproduction step of reproducing the sound data as sound by:
3D display control method characterized by including.