JP5037713B1 - Stereoscopic image display apparatus and stereoscopic image display method - Google Patents

Abstract

A stereoscopic image display apparatus capable of simultaneously providing a stereoscopic image suitable for a plurality of users is provided.
Display means for displaying a plurality of images in order in time division, synchronization means for outputting a synchronization signal in synchronization with display on the display means, receiving the synchronization signal from the synchronization means, and receiving the synchronization signal A shielding unit that shields the right eye or the left eye according to the image and selectively transmits the plurality of images to the right eye and the left eye, and an inclination detection unit that detects an inclination of the shielding unit, the shielding unit including the inclination A set of images to be transmitted through the right eye and the left eye is selected from the plurality of images according to the inclination detected by the detecting means.
[Selection] Figure 1

Description

  The present invention relates to a stereoscopic image display device.

  Humans have the ability to grasp a space from the difference in images obtained by each of two eyes having a fixed interval. The shift of corresponding points in an image obtained from different viewpoints by the left and right eyes is called parallax, and the positional relationship of the object is grasped in three dimensions using parallax as one of the cues. By utilizing this, a means for displaying a right-eye image and a left-eye image on the right eye is provided to provide a right-eye image and an image with parallax as a left-eye image. Is known to be possible. Here, a plurality of images with parallax intended for stereoscopic viewing are referred to as stereoscopic images.

  In stereoscopic vision, it is said that humans associate the angle formed by the optical axes of both eyes according to parallax, that is, the magnitude of convergence with the object. Therefore, when the right-eye image is shifted to the right and the left-eye image is relatively shifted to the left to show the parallaxed image, the display object can be perceived farther than the actual display surface. However, if too much parallax is added at this time and the parallax exceeds the distance between the eyes of the user, more precisely the distance between the pupils when looking at infinity, the situation cannot occur in the natural world, and stereoscopic vision is not possible. Even if it is possible or even if it can be stereoscopically viewed, it will impose a heavy burden on the human body. Similarly, the extreme parallax causes an unnatural positional relationship even on the near side from the actual display surface, and the user is forced to have an extreme cross-over, so that comfortable stereoscopic viewing cannot be performed. In addition, as the parallax increases, the difference between the convergence and the focus adjustment of the eyes increases, resulting in an unnatural state, which causes a sense of discomfort. As described above, stereoscopic viewing can be comfortably performed within a certain range of the parallax amount of the stereoscopic image. However, when the parallax amount is large, the images of both eyes are not fused and stereoscopic viewing is impossible.

  Here, the stereoscopic image is generally produced on the premise that the stereoscopic image is observed horizontally with respect to the stereoscopic image display device. Since the human eyes are arranged at positions separated from each other on the left and right, An image with parallax added is produced as a stereoscopic image. However, in an actual viewing environment, an image is not necessarily observed horizontally with respect to the stereoscopic image display device. When an image is observed not horizontally with respect to the stereoscopic image display device, the right and left eyes occur in nature. An image from an unnatural viewpoint that cannot be obtained may be input, and stereoscopic viewing may be impossible, or the human body may be burdened.

  Therefore, Patent Document 1 discloses a stereoscopic image display device that provides a viewpoint image in accordance with the inclination of a user.

JP 2006-84963 A

  According to the method of Patent Document 1, it is possible to provide a viewpoint image that matches the inclination of the user. However, in a one-system system, the number of users who can handle the tilt is limited to one, and a plurality of users cannot be handled.

  In view of such circumstances, the present invention provides a stereoscopic image display apparatus that can simultaneously provide stereoscopic images suitable for a plurality of users.

  The stereoscopic image display apparatus according to the present invention includes a display unit that sequentially displays a plurality of images in time division, a synchronization unit that outputs a synchronization signal in synchronization with the display on the display unit, and the synchronization signal received from the synchronization unit And a shielding unit that shields the right eye or the left eye according to the synchronization signal and selectively transmits the plurality of images to the right eye and the left eye, and an inclination detection unit that detects an inclination of the shielding unit. The means is characterized by selecting a set of images to be transmitted through the right eye and the left eye from the plurality of images according to the inclination detected by the inclination detecting means.

  The plurality of images may include images obtained by capturing subjects from different viewpoints, and may include two or more sets of images that are transmitted through the right eye and the left eye.

  The sets of images transmitted through the right eye and the left eye may have parallaxes corresponding to different inclinations.

  The sets of images that are transmitted through the right eye and the left eye may have different vertical parallaxes.

  At least one of the plurality of images may be included in a set of images that are transmitted through the plurality of right eyes and left eyes.

  The plurality of images may include a two-dimensional display image.

  The plurality of images may be generated in consideration of display timing when each of the images is displayed on the display unit in a time division manner.

  When the inclination of the shielding means cannot be obtained from the inclination detection means, the shielding means may transmit a predetermined set of images in the plurality of images to the right eye and the left eye.

  When the inclination of the shielding means cannot be obtained from the inclination detection means, the shielding means may transmit the same image in the plurality of images to the right eye and the left eye.

  When there is no set of images to be transmitted through the right eye and the left eye corresponding to the inclination of the shielding means detected by the inclination detecting means, the shielding means determines a predetermined set of images in the plurality of images as the right eye and the left eye. It may be permeated through.

  When there is no set of images to be transmitted through the right eye and the left eye corresponding to the inclination of the shielding means detected by the inclination detection means, the shielding means transmits the same image in the plurality of images to the right eye and the left eye. Also good.

  The synchronization signal may include a signal indicating a correspondence between the inclination of the shielding unit detected by the inclination detection unit and a set of images transmitted through the right eye and the left eye.

  The synchronization signal may include a signal representing a set of images that the shielding means transmits to the right eye and the left eye when the inclination of the shielding means cannot be obtained from the inclination detection means.

  The synchronization signal represents a set of images that the shielding unit transmits to the right eye and the left eye when there is no set of images that are transmitted through the right eye and the left eye corresponding to the inclination of the shielding unit detected by the tilt detection unit. A signal may be included.

  The different viewpoints in each of the plurality of images may be set in association with preset inclinations of the shielding means.

  The different viewpoints in each of the plurality of images may be set in association with the inclination of the shielding means selected by the user.

  The different viewpoints in each of the plurality of images may be set according to a setting value associated with each of the plurality of images.

The synchronization signal may be output by infrared rays.
The synchronization signal may be output by radio waves.

The shielding means may be a glasses type.
The shielding means may be a helmet type.
The shielding means may have a surface shape covering the user's face.

  Image adjustment means may be provided for adjusting the parallax in the vertical direction by relatively shifting the entire or part of the screen area of the image of a certain viewpoint in the vertical direction and generating the plurality of images.

  An image adjusting unit may be provided that generates an image that can be regarded as an image obtained by capturing a subject from different viewpoints from an image of a certain viewpoint, and generates the plurality of images.

  According to the present invention, a stereoscopic image suitable for a plurality of users can be provided simultaneously.

It is a block diagram which shows an example of a structure of the stereo image display apparatus by the 1st Embodiment of this invention. 3 is a block diagram illustrating an example of a configuration of a parallax adjustment unit 101. FIG. 2 is a block diagram illustrating an example of a configuration of shutter glasses 107. FIG. (A)-(d) is a figure explaining the relationship between parallax and depth display. (A)-(c) is a figure explaining the difference in the parallax by the relative inclination of a stereo image display apparatus and a user's viewpoint. It is a figure explaining a some viewpoint. It is a figure explaining a some viewpoint. It is a figure which shows an example of the presentation method of a viewpoint image. It is a block diagram which shows an example of a structure of the parallax adjustment part 101 by the 2nd Embodiment of this invention. It is a block diagram which shows an example of a structure of the stereo image display apparatus by the 3rd Embodiment of this invention. It is a figure which shows a mode that the dotted | punctate object is discharged from the cylindrical object.

First Embodiment (Generating Multiple Parallaxes by Shifting Process)
Hereinafter, a stereoscopic image display device according to a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating an example of the configuration of the stereoscopic image display apparatus according to the present embodiment. As shown in FIG. 1, the stereoscopic image display apparatus according to the present embodiment has an input unit 10 that receives image data, and display data that can be displayed in 3D by processing the input image data (hereinafter, stereoscopic image data). 3D image processing unit 100 that performs image processing for generating image, parallax adjustment unit 101 that adjusts the parallax of the image, display control unit 102 that controls display by matching the image with the display unit, and display that displays the image Unit 103, system control unit 104 that controls the entire system, user input unit 105 that the user inputs, glasses synchronization unit 106 that synchronizes shutter glasses 107, and shutter glasses 107 that the user wears.

  FIG. 2 is a block diagram illustrating an example of the configuration of the parallax adjustment unit 101. The parallax adjustment unit 101 includes a communication / control unit 1011, a parallax calculation unit 1012, and shift processing units 1013 a to 1013 c.

  FIG. 3 is a block diagram illustrating an example of the configuration of the shutter glasses 107. The shutter glasses 107 include a glasses synchronization reception unit 1071, a shutter control unit 1072, an inclination detection unit 1073, and shutter units 1074a and 1074b. The shutter units 1074a and 1074b correspond to a right-eye shutter and a left-eye shutter, respectively.

  Next, the operation of each unit will be described. The input unit 10 transmits the image data input to the stereoscopic image display device to the stereoscopic image processing unit 100, and the stereoscopic image processing unit 100 expands the left-eye data and the right-eye data according to the input format. Here, the input image data may be any data such as data based on a broadcast wave, data read electronically from a recording medium, or data acquired by communication. That is, the input unit 10 may be a semiconductor memory reading device, or may have a communication function with an optical disk or magnetic disk reading device, a radio wave receiver, or a network. In short, any data can be used as long as it can input data that can be interpreted as a stereoscopic image. Further, the right-eye image data and the left-eye image data may be created from a single piece of image data. That is, it may be a multi-viewpoint image synthesized from image data and depth data or parallax data, or a multi-viewpoint image created by estimating depth information. It may be a multi-view image for multi-view display. If there is additional information in the input image data, the stereoscopic image processing unit 100 extracts the additional information and transmits it to the system control unit 104. The additional information may be parameters at the time of shooting, parallax information, parallax amount correction information, depth data, and the like.

  The stereoscopic image processing unit 100 transmits the developed left-eye image data and right-eye image data to the parallax adjustment unit 101, and the parallax adjustment unit 101 recombines the left-eye image data and the right-eye image data into a plurality of viewpoint images. To do. Within the parallax adjustment unit 101, a communication / control unit 1011 that communicates with the system control unit 104 controls each unit. The parallax calculation unit 1012 calculates the parallax from the shift of corresponding points of the left and right images in the image data input to the parallax adjustment unit 101, and the calculated parallax data is transmitted to the system control unit 104 via the communication / control unit 1011. Transmit to. As described above, when depth data is input or when depth data is created, it may be used as parallax data. The parallax data may be, for example, the maximum amount and the minimum amount of parallax in the image. Here, the disparity on the far side of the display screen is expressed as a positive value and the near side is expressed as a negative value, so the maximum value of the disparity is the disparity of the farthest point on the screen, and the minimum value is the most distant on the screen. Represents the parallax of a near point.

  The system control unit 104 obtains an image shift amount appropriate for each viewpoint image based on the parallax data received from the parallax adjustment unit 101, and transmits the image shift amount to the parallax adjustment unit 101.

  The image shift processing units 1013a to 1013c of the parallax adjustment unit 101 receive pixel shift amounts from the system control unit 104 via the communication / control unit 1011 and perform processing for shifting the entire right eye image vertically and horizontally, respectively. , Adjust the parallax and output. The parallax adjustment unit 101 includes right-eye data in which parallax is not adjusted, right-eye image data in which parallax is adjusted by each of the image shift processing units 1013a to 1013c, and left-eye image data in which parallax is not adjusted. , And output as a plurality (four in this example) viewpoint images.

  Although the parallax is adjusted by shifting only the right-eye image here, the left-eye image may be shifted or both the left-right eye image may be shifted. In short, it is sufficient that the parallax can be adjusted by relatively shifting the corresponding points of the plurality of viewpoint images vertically and horizontally.

  In addition, when a parallax range can be assumed in advance, the image shift amount may be a value calculated in advance without using the calculation result by the parallax calculation unit 1012. In this case, the parallax calculation unit 1012 can be omitted, and the processing amount of the system control unit 104 can be reduced. The parallax range may be attached to the image data or may be determined by agreement.

  The display control unit 102 receives a plurality of viewpoint images adjusted by the parallax output from the parallax adjustment 101, performs display control in accordance with the display unit 103, and simultaneously transmits the plurality of viewpoint images to the glasses synchronization unit 106. Signals indicating the timing to display and the timing to display the right-eye image and the left-eye image are sent.

  The glasses synchronization unit 106 sends a synchronization signal to the shutter glasses 107 worn by the user based on the signal received from the display control unit 102, and performs synchronization processing with the display unit 103. Specifically, for example, in the case of a system in which four viewpoint images are displayed in a time-sharing manner using a liquid crystal display panel on the display unit 103 and stereoscopic viewing is performed in synchronization with the shutter glasses 107 worn by the user, display control is performed. The unit 102 sequentially outputs four viewpoint images to the display unit 103. The output frequency is, for example, 60 images each second corresponding to each viewpoint. The display unit 103 displays the image sent from the display control unit 102 at any time. At that time, the display unit 103 and the shutter glasses 107 are synchronized, and the left-eye image is displayed on any of the four viewpoint images sequentially displayed on the display unit 103. By opening the shutter and the shutter for the right eye in synchronization, the corresponding viewpoint images are presented to the left eye and the right eye, thereby realizing stereoscopic viewing. Specifically, the shutter glasses 107 are based on a synchronization signal that the glasses synchronization receiving unit 1071 receives from the glasses synchronization unit 106 and a tilt detection signal that the tilt detection unit 1073 detects and outputs the tilt of the shutter glasses 107. The shutter control unit 1072 controls the timing of the shutter open / close signal and drives the shutter units 1074a and 1074b to present the viewpoint image to the user in synchronization with the display unit 103.

  Note that the user's burden can be reduced by transmitting the synchronization signal wirelessly without connecting the glasses synchronization unit 106 and the shutter glasses 107 by wire. When the synchronization signal is transmitted by infrared rays, the glasses synchronization unit 106 and the glasses synchronization reception unit 1071 are provided at a position where the user can observe the display screen by providing the glasses synchronization unit 106 at a position that can be seen by the user near the display unit 103. There is no obstacle between them and an appropriate distance is maintained, and the shutter glasses 107 can receive the synchronization signal. When the synchronization signal is transmitted by radio waves, it is not necessary to provide the glasses synchronization unit 103 at a position visible to the user near the display unit 103, and the shutter glasses 107 do not need to face the glasses synchronization unit 103. Even if there is an obstacle between the shutter glasses 107 and the shutter glasses 107, synchronization is possible.

  FIG. 4 is a diagram illustrating the relationship between parallax and depth display. FIG. 4 is a top view of the user and the display.

  FIG. 4A shows a state in which the corresponding points of the right-eye image and the left-eye image are at the same position on the display during the stereoscopic image display, which is the same as in the normal two-dimensional display state. In this case, the corresponding point is perceived as being on the display.

  FIG. 4B shows a state where the corresponding point of the right-eye image is shifted to the right and the corresponding point of the left-eye image is shifted to the left on the display. In this state, the corresponding point is perceived by the user behind the display surface.

  FIG. 4C shows a state in which the corresponding point of the right-eye image is shifted to the left and the corresponding point of the left-eye image is shifted to the right on the display. In this state, the user perceives the corresponding point in front of the display surface.

  FIG. 4D is a diagram summarizing FIGS. 4A to 4C. As described above, on the display, when the corresponding point of the right-eye image is displayed on the right, the corresponding point of the left-eye image is shifted to the left, and the distance between the corresponding points of the left-right image is equal to the binocular distance, Corresponding points are perceived at infinity, but if the distance between corresponding points exceeds the binocular distance, the line of sight does not face the divergent direction and cannot be fused. Similarly, when the corresponding point of the image for the right eye is greatly shifted to the left and the corresponding point of the image for the left eye is greatly shifted to the right on the display, the line of sight becomes an extreme crossed state and cannot be merged. Therefore, the depth range in which stereoscopic viewing can be comfortably performed, that is, the comfortable fusion range shown in FIG. 4D is on the inner side of the display surface than these fusion ranges. By applying these, the left and right eye images are relatively shifted to the left and right to increase or decrease the shift of corresponding points in the left and right eye images. , You can be in the foreground.

  FIG. 5 is a diagram for explaining a difference in parallax due to the relative inclination between the stereoscopic image display device and the user's viewpoint. As shown in FIG. 5A, it is assumed that the left-right parallax necessary for stereoscopic viewing is d in a state where the user's viewpoint is not inclined with respect to the stereoscopic image display device. When the user tilts with respect to the stereoscopic image display device, as shown in FIG. 5B, the parallax dv ′ in the vertical direction is added to the user and should not be parallax according to the tilt. The parallax is reduced from d to dh ′, and the natural stereoscopic effect is reduced. Therefore, as shown in FIG. 5C, the vertical parallax dv is added to the display image in accordance with the relative inclination between the stereoscopic image display device and the user's viewpoint, and the horizontal parallax is adjusted to dh. Therefore, by making the parallax d ′ in the left-right direction equal to d, natural stereoscopic vision can be prevented from being hindered. Specifically, the right-eye image and the left-eye image are relatively shifted up and down and left and right, thereby adding necessary vertical and horizontal parallaxes.

  Note that, in a range where the inclination angle is small, the amount of displacement of the lateral parallax is not large, and therefore the process of adjusting the lateral parallax may be omitted. Also, in the case of vertical parallax, when the distance to the subject is sufficiently large relative to the vertical displacement, the displacement of the field of view due to the movement of the viewpoint is relative to the individual subject due to the movement of the viewpoint. Since it is larger than the parallax change, it can be easily realized by shifting the image in the vertical direction.

  Next, the shutter glasses 107 will be described. The shutter glasses 107 select a combination that matches the tilt angle measured by the shutter glasses 107 from a plurality of viewpoint images in synchronization with the synchronization signal sent from the glasses synchronization unit 106, and provide the user with a viewpoint image with an appropriate amount of parallax. Presenting enables stereoscopic viewing. 6 and 7 are diagrams illustrating a plurality of viewpoints. For example, as shown in FIG. 6, the plurality of viewpoint images may have combinations according to angles, but as shown in FIG. 7, a plurality of stereoscopic images can be obtained by combining any of the viewpoints and combining these viewpoints. As a result, the number of viewpoint images can be reduced. Strictly speaking, the displacement of the tilt of the left and right eyes does not correspond to the displacement of the image at this time, but does not significantly affect the distance from the displayed image. In this embodiment, by adopting the viewpoint positions shown in FIGS. 7A to 7D, a set of three different left and right images can be obtained from the four viewpoint images, and a stereoscopic image having parallax according to the left and right inclinations. Can be presented. As described above, since the parallax in the vertical direction is simply realized by the shift process, it is preferable that the tilt of the viewpoint, that is, the angle cad and the angle bda that are angles formed by the viewpoint in FIG.

  Since the tilt of the user's viewpoint can be approximated by the tilt of the shutter glasses 107 worn, natural stereoscopic viewing can be achieved by selecting the parallax in the vertical direction according to the tilt of the shutter glasses 107. The inclination of the shutter glasses 107 can be easily detected by a small device such as a piezoelectric gyro sensor. Note that the tilt detection unit 1073 does not necessarily have to be built in the shutter glasses 107, and a method of detecting the tilt by recognizing the shutter glasses 107 from an image from the display unit 103 side may be used. In short, the relative inclination between the display unit 103 and the individual shutter glasses 107 may be detected and transmitted to the individual shutter glasses 107.

  The correspondence between the tilt and the viewpoint image combination may be determined in advance, but may be superimposed on the synchronization signal sent from the glasses synchronization unit 106 and transmitted. Accordingly, it is possible to present to the user an appropriate set of images corresponding to various stereoscopic images, observation environments, and purposes. In addition, when the user's movement is intense or when the posture is not appropriate, the correct inclination may not be detected. Even if the tilt itself can be detected correctly, the tilt range may be exceeded. In such a case, an image with an unnatural parallax can be prevented from being presented to the user by presenting a predetermined combination of images. In addition, a signal representing a combination of images to be presented when the tilt cannot be detected or exceeds the range of the tilt that can be handled may be superimposed on the synchronization signal sent from the glasses synchronization unit. The set of images to be presented can be changed according to the scene and purpose. In such a case, if a two-dimensional image is presented, the correspondence between the inclination and the parallax can be avoided, and an image with an unnatural parallax can be prevented from being presented to the user.

  In this way, a set of a plurality of images corresponding to a plurality of tilt angles can be presented according to the tilt of the shutter glasses 107 worn by the user. That is, a stereoscopic image with a parallax amount corresponding to an assumed inclination can be obtained from the combination of the respective viewpoint images, and a stereoscopic image can be presented with an appropriate parallax amount to a plurality of users. Since the shutter glasses 107 used by each user select an appropriate combination of viewpoint images according to the respective inclinations, the number of users can be increased as much as possible for one stereoscopic image display device. The shutter glasses 107 worn by each user are not significantly different from general sunglasses, and the visual discomfort is not great, so that the shutter glasses 107 are less likely to hinder communication between users.

  In this embodiment, as shown in FIG. 2, there are images that are not processed by the shift processing units 1013 a to 1013 c, that is, images that are sent to the display unit 103 without modifying the input image. When the processing by the shift processing units 1013a to 1013c is performed, the image is moved vertically and horizontally due to the influence of parallax adjustment. Therefore, when the same image is observed with the left and right eyes among the images that have been subjected to the shift process, it is observed as a two-dimensional image because no parallax is added, but the image is preferably moved up and down and left and right as the parallax is adjusted. Absent. Therefore, by presenting to both eyes an image that does not pass through the shift processing unit 1013, the input image itself can be favorably observed as a two-dimensional image. The set of viewpoint images selected by the shutter glasses may include a combination that presents the same viewpoint image on the left and right as described above.

  Further, an image different from the stereoscopic display image may be prepared in advance as an image to be two-dimensionally displayed. Since a stereoscopic image is produced on the premise that it is observed with both eyes, when one of the stereoscopic images is extracted, it may not always be an appropriate viewpoint. This is particularly noticeable when the distance to the subject is close to the baseline length. In order to avoid this phenomenon, if an image premised on two-dimensional display is prepared, a good two-dimensional image can be presented.

  FIG. 8 is a diagram illustrating an example of a method for presenting four viewpoint images including an image for two-dimensional display. The viewpoint images 1-3 are images from different viewpoints. A set 1 of images that presents the viewpoint image 1 to the left eye and a viewpoint image 2 to the right eye, a set 2 of images that presents the viewpoint image 2 to the left eye and presents the viewpoint image 3 to the right eye, and a viewpoint image 1 to the left eye The set 3 of images that are presented and the viewpoint image 3 is presented to the right eye is a parallax image to which different parallaxes are added. In the case of two-dimensional display, for example, the viewpoint image 1 that is also included in the parallax image can be presented as the left eye and the right eye as in the image set 4, but the viewpoint image 4 that is an image for two-dimensional display is displayed in the left eye. And can be presented to the right eye.

Second Embodiment (Multiple Parallax Generation by Shifting Each Pixel)
Hereinafter, a stereoscopic image display apparatus according to a second embodiment of the present invention will be described with reference to the drawings. A configuration example of the stereoscopic image display apparatus according to the present embodiment is shown in FIG. 1 as in the first embodiment. As shown in FIG. 1, the stereoscopic image display apparatus according to the present embodiment has an input unit 10 and a stereoscopic image processing unit 100 that processes input image data and performs image processing for generating stereoscopic image data. A parallax adjustment unit 101 that adjusts the parallax of the image, a display control unit 102 that controls display by matching the image with the display unit 103, a display unit 103 that displays an image, and a system control unit 104 that controls the entire system. The user input unit 105 that the user inputs, the glasses synchronization unit 106 that synchronizes the shutter glasses 107, and the shutter glasses 107 that the user wears.

  FIG. 9 is a block diagram illustrating an example of the configuration of the parallax adjustment unit 101 according to the present embodiment. The parallax adjustment unit 101 includes a communication / control unit 1011, a parallax calculation unit 1012, a parallax correction unit 1014, a shielding compensation unit 1015, and image processing units 1016 a to 1016 e.

  Since the configuration of the shutter glasses 107 is the same as that shown in FIG. 3 of the first embodiment, the description thereof is omitted.

  Next, the operation of each unit will be described. The input unit 10 transmits the image data input to the stereoscopic image display device to the stereoscopic image processing unit 100, and the stereoscopic image processing unit 100 expands the left-eye data and the right-eye data according to the input format. At the same time, if the input image data includes additional information, the stereoscopic image processing unit 100 extracts the additional information and transmits it to the system control unit 104. Here, the input image data may be any data such as data based on broadcast waves, data read electronically from a recording medium, or data acquired through communication. Further, the right-eye image data and the left-eye image data may be created from a single piece of image data. That is, it may be a multi-viewpoint image synthesized from image data and depth data or parallax data, or a multi-viewpoint image created by estimating depth information. Of course, it may be a multi-viewpoint image input as multi-viewpoint image data.

  The stereoscopic image processing unit 100 sends the developed left-eye image data and right-eye image data to the parallax adjustment unit 101, and the parallax adjustment unit 101 reconverts the received left-eye image data and right-eye image data into a plurality of viewpoint images. Synthesize. In the parallax adjustment unit 101, a communication / control unit 1011 that communicates with the system control unit 104 controls each unit.

  The parallax calculation unit 1012 calculates the parallax from the shift between corresponding points of the left and right images in the image data input to the parallax adjustment unit 101, and sends the calculated parallax data to the parallax correction unit 1014. For example, the parallax data may be a so-called parallax map that represents the parallax for each pixel.

  The parallax correction unit 1014 selects parameters necessary for system control from the received parallax data, and transmits the parameters to the system control unit 104 via the communication / control unit 1011.

  Based on the parameter received from the parallax adjustment unit 101 and the set viewpoint position, the system control unit 104 obtains an appropriate correction parameter for each viewpoint image and transmits the correction parameter to the parallax adjustment unit 101. Here, each viewpoint position is set according to the assumed inclination. The assumed inclination may be set to a specific value in advance, or a method obtained from a set value recorded in advance in the image information, a method input by the user, or the like may be used. If the image information is recorded in advance, an inclination corresponding to the scene can be set. In addition, when the method is input by the user, it is possible to prepare an inclination image in accordance with the characteristics of each user, for example, the strength of a heel that inclines the body according to the image.

  The communication / control unit 1011 transmits the correction parameter transmitted from the system control unit 104 to the parallax adjustment unit 101 to the parallax correction unit 1014. The parallax correction unit 1014 obtains a plurality of corrected parallax data corresponding to the assumed viewpoint positions based on the correction parameters received from the communication / control unit 1011 and the parallax data received from the parallax calculation unit 1012. It is calculated and output to the shielding compensation unit 1015 and the image processing unit 1016 corresponding to each viewpoint image. The plurality of corrected parallax data at this time may also be so-called parallax maps.

  When the occlusion compensation unit 1015 re-synthesizes an image based on a plurality of parallax data corrected by the parallax correction unit 1014, the occlusion compensation unit 1015 changes image data regarding a portion where the occlusion relation changes and the image data disappears due to a change in viewpoint. Compensation image data to be compensated is generated. The compensation image data can be generated using, for example, images of other viewpoints having different shielding relations among the input images. A plurality of pieces of compensation image data are output corresponding to the number of image processing units 1016.

  The image processing units 1016a to 1016e use the plurality of corrected parallax data and the compensated image data to re-synthesize a new viewpoint image that matches each assumed viewpoint position. More specifically, based on the corrected plurality of parallax data, the pixel data of the target region is moved, and the pixel data of the corresponding portion of the compensation image data is supplemented in the region where the corresponding pixel disappears due to the movement. . In this way, a viewpoint image that is not included in the input stereoscopic data is generated.

  Here, the parallax is calculated from the left and right images, but when the input is a combination of image data and depth data, the input depth data may be converted into parallax data and used. In FIG. 9, left and right images are input to the image processing units 1016a to 1016e, but one of the right and left images may be input. By doing so, it is possible to generate a viewpoint image in which the camera position is virtually shifted, and to adjust the parallax of the image corresponding to the tilt of the viewpoint.

  The parallax adjustment unit 101 sends the generated plurality of viewpoint images to the display control unit 102. The display control unit 102 performs display control in accordance with the display unit 103 and simultaneously sends a signal to the glasses synchronization unit 106.

  The glasses synchronization unit 106 sends a synchronization signal to the shutter glasses 107 worn by the user based on the signal received from the display control unit 102, and performs synchronization processing with the display unit 103. Specifically, for example, in the case of a system in which five viewpoint images are displayed in a time-sharing manner using a liquid crystal display panel on the display unit 103 and stereoscopic viewing is performed in synchronization with the shutter glasses 107 worn by the user, display control is performed. The unit 102 sequentially outputs five viewpoint images to the display unit 103. The output frequency is, for example, 60 images each second corresponding to each viewpoint. The display unit 103 displays the image sent from the display control unit 102 at any time. At this time, the display unit 103 and the shutter glasses 107 are synchronized, and the left-eye image is displayed on one of the five viewpoint images displayed in order on the display unit 103. By opening the shutter and the shutter for the right eye in synchronization, the corresponding viewpoint images are presented to the left eye and the right eye, thereby realizing stereoscopic viewing.

  As described above, since the parallax is caused by the relative shift between the left and right viewpoints, it is necessary to add the parallax according to the inclination. Therefore, as shown in FIG. 6, two images with relatively appropriate viewpoint deviation amounts are extracted from images from a plurality of viewpoints, and each of them is used as a right-eye image and a left-eye image. Select a set of images with parallax. Further, by adjusting the relative shift amount of each viewpoint at this time, it is possible to obtain a plurality of viewpoint images by changing the way of selecting a combination. FIG. 7 shows an example in which images with five parallaxes are obtained from viewpoint images from five viewpoints. Needless to say, the more viewpoint images that are elements, the more combinations can be obtained.

  The shutter glasses 107 select and present a combination of viewpoint images corresponding to the inclination of each shutter glasses 107 from the plurality of viewpoint images. In this way, a stereoscopic image can be presented with an appropriate amount of parallax to a plurality of users.

  Note that, here, a method of obtaining five viewpoint images from two viewpoint images by conversion is shown, but it goes without saying that images may be acquired by using an actual camera for five viewpoints. The arrangement of the viewpoints may be such that the optical axes are installed in parallel, the optical axes are converging, or the viewpoints are arranged on the same straight line so that the distance to the convergence point of the optical axis is equidistant. It does n’t matter. Although an example with five viewpoints is shown, the number of viewpoints is not limited to five, and should be adjusted according to the frame rate of the image and the response speed of the display device.

  In the present embodiment, images of five viewpoints with the camera position virtually moved are displayed. In addition, a viewpoint image to be displayed as a two-dimensional image is added, and the same image is observed as a two-dimensional image with the left and right eyes. You may be able to do it. Since a stereoscopic image is produced on the premise that it is observed with both eyes, when one of the stereoscopic images is extracted, it may not always be an appropriate viewpoint. Such an inappropriate viewpoint is particularly noticeable when the distance to the subject is close to the baseline length. Further, if a viewpoint image to be displayed as a two-dimensional image is added, it is possible to avoid alteration of the image due to viewpoint synthesis. By the two-dimensional display, it is possible to safely observe images including users who are not suitable for viewing stereoscopic images due to their constitution and physical condition. Similarly to the three-dimensional image, the two-dimensional image can be presented using shutter glasses.

  In the present embodiment, the system has been described by taking a parallax map representing parallax for each pixel as an example. However, the parallax map for a reduced image is not necessarily required for each pixel. In this case, when processing is actually performed, the parallax map having a low resolution may be used as it is, or a parallax map having a higher resolution by filter processing or the like may be used.

<Third Embodiment> (Multi-parallax generation by CG)
Hereinafter, a stereoscopic image display apparatus according to a third embodiment of the present invention will be described with reference to the drawings. FIG. 10 is a block diagram illustrating an example of the configuration of the stereoscopic image display apparatus according to the present embodiment. As shown in FIG. 10, the stereoscopic image display apparatus according to the present embodiment includes a user operation unit 20, a game control unit 201, a display control unit 202, a display unit 203, glasses synchronization unit 206, and shutter glasses 207. Is provided. The game control unit 201 includes a system control unit 2011, a program storage unit 2012, and a stereoscopic image generation unit 2013.

  The user operation unit 20 may be a game controller. The system control unit 2011 may include I / O with each of the user operation unit 20, the program storage unit 2012, the stereoscopic image generation unit 2013, and the display control unit 202, a CPU, a memory, and the like. The program storage unit 2012 may be a hard disk or an optical disk that stores game software and the like. The stereoscopic image generation unit 2013 may include a GPU (Graphic Processing Unit), a memory, and the like.

  The display unit 203 may be a liquid crystal display, a plasma display, an organic EL display, or the like, but is preferably a display device with a high response speed. For example, if an LED display or the like is used, a high-speed response can be obtained. The glasses synchronization unit 206 sends a synchronization signal to the shutter glasses 207 by infrared rays or radio waves. There may be a plurality of user operation units 20 and shutter glasses 207.

  The configuration of the shutter glasses 207 is the same as the configuration of the shutter glasses 107 shown in the first embodiment and FIG.

  Next, the operation of each unit will be described. The system control unit 2011 receives an operation from the user operation unit 20, reads the program from the program storage unit 2012 and executes it. As described above, the program may be stored in the program storage unit 2012 that is an optical disk or a hard disk. Of course, the program storage unit 2012 may be another storage device such as a solid-state memory. When the program storage unit 2012 is a hard disk, the program may be stored in the hard disk in any form such as a method of reading from the optical disk, a method of acquiring from the network, and a method of reading from the solid memory.

  The system control unit 2011 outputs a stereoscopic image model to the stereoscopic image generation unit 2013 as the program is executed. Here, the stereoscopic image model represents the shape and arrangement of an object to be displayed, the position of a light source serving as illumination, a virtual viewpoint position, and the like on three-dimensional coordinates. The three-dimensional coordinates may be an orthogonal coordinate system or a polar coordinate system. In addition, the pattern and state of the surface of the object to be displayed may be described. The stereoscopic image generation unit 2013 expands the stereoscopic image model input from the system control unit 2011 and generates individual viewpoint images. More specifically, a plurality of viewpoint images are obtained by generating a diagram in which an arrangement diagram of objects included in the stereoscopic image model is viewed from the positions of a plurality of virtual cameras included in the stereoscopic image model. The positions of the plurality of virtual cameras are positions corresponding to a plurality of viewpoints, for example, as shown in FIG. FIG. 7 shows how images with five parallaxes are obtained from five viewpoint images by five cameras. Note that the positions of the plurality of virtual cameras are determined by an assumed inclination. The assumed inclination may be input by the user or may be preset in the program. When the user inputs, a parallax image corresponding to the inclination according to the user's preference and habit can be presented, and when using a preset program, an image suitable for the scene assumed by the program can be presented .

  The stereoscopic image generation unit 2013 sends the generated plurality of viewpoint images to the display control unit 202. The display control unit 202 receives a plurality of viewpoint images from the stereoscopic image generation unit 2013, performs display control in accordance with the display unit 203, and simultaneously sends a signal to the glasses synchronization unit 206. The glasses synchronization unit 206 sends a synchronization signal to the shutter glasses 207 worn by the user based on the signal received from the display control unit 202, and performs a synchronization process with the display unit 203. Specifically, for example, in the case of a method in which five viewpoint images are displayed on the display unit 203 using a liquid crystal display panel and stereoscopic viewing is performed in synchronization with the shutter glasses 207 worn by the user, the display control unit 202 Then, five viewpoint images are sequentially output to the display unit 203. The output frequency is, for example, 60 images each second corresponding to each viewpoint. The display unit 203 displays the image sent from the display control unit 202 at any time. However, the display unit 203 and the shutter glasses 207 are synchronized, and the left-eye shutter and the right-eye are displayed on any of the five viewpoint images sequentially displayed on the display unit. By opening the shutters in synchronization, the corresponding viewpoint images are presented to the left eye and the right eye to realize stereoscopic viewing.

  At this time, since a plurality of viewpoint images are displayed in order, the stereoscopic image generation unit 2013 generates the image for each viewpoint image in consideration of the timing actually displayed in the time division on the display unit 203. A smooth movement along the time axis can be perceived, which is preferable. FIG. 11 is a diagram for explaining the situation at this time. FIG. 11 shows a point-like object being emitted from a cylindrical object. At this time, the appearance of the cylindrical object changes according to the viewpoint, and the display position of the dot-like object changes with the passage of time. Although the appearance of the point-like object also changes according to the viewpoint, since FIG. 11 is small, the viewpoint change is omitted in the representation of the figure. An image along the time axis may be generated by interpolation of time-series images.

  As described above, the shutter glasses 107 select and present a combination of viewpoint images corresponding to the inclination of each shutter glasses 107 from the plurality of viewpoint images. In this way, viewpoint images corresponding to the tilt of each user's head can be presented to a plurality of users. That is, a stereoscopic image with a parallax amount corresponding to the assumed inclination can be obtained, and a stereoscopic image can be presented to each of a plurality of users with an appropriate parallax amount from a combination of viewpoint images. The user can simultaneously enjoy a stereoscopic display game toward the same display screen. In addition, since the shutter glasses used by each user select an appropriate combination of viewpoint images, it is possible to increase the number of users who observe the screen with respect to one stereoscopic image display device.

  In the above description, shutter glasses are used as shielding means for selecting an image in synchronization with the stereoscopic image display device, but the shape of the shielding means is not limited to glasses. For example, it is good also as a shielding means which provided the shielding element, for example, a liquid-crystal shutter, in the part which covers a thing like a surface or the eyes of a helmet. In this case, by imitating a protective surface used for ball games or martial arts, or a helmet used for motor sports, etc., the effect of external light entering through the gap between the glasses and the face when using glasses can be reduced. A more immersive game device using the image display device can be configured. In addition, it is possible to reduce the slipping and tilting of glasses, which is often the case with glasses, and the close contact with the head is higher than that of glasses, so it is possible to measure the tilt of the face more accurately and estimate the appropriate parallax be able to. Furthermore, in game devices installed in public spaces, it is preferable that each component device is connected by wire, but compared to glasses, the surface and helmet support the weight of the connection line with the entire head. It can reduce the troublesomeness.

  The present invention is applicable to a stereoscopic image display device.

DESCRIPTION OF SYMBOLS 10 Input part 20 User operation part 100 Stereoscopic image processing part 101 Parallax adjustment part 102, 202 Display control part 103, 203 Display part 104, 2011 System control part 105 User input part 106, 206 Glasses synchronization part 107, 207 Shutter glasses 201 Game Control unit 1011 Communication / control unit 1012 Parallax calculation unit 1013a-c Shift processing unit 1071 Glasses synchronization reception unit 1072 Shutter control unit 1073 Inclination detection unit 1074a, b Shutter unit 1014 Parallax correction unit 1015 Shading compensation unit 1016a-e Image processing unit 2012 Program storage unit 2013 stereoscopic image generation unit

Claims (25)

Display means for sequentially displaying a plurality of images in a time-sharing manner;
Synchronization means for outputting a synchronization signal in synchronization with the display on the display means;
Shielding means for receiving the synchronization signal from the synchronization means, shielding a right eye or a left eye according to the synchronization signal, and selectively transmitting the plurality of images to the right eye and the left eye;
An inclination detecting means for detecting the inclination of the shielding means,
The shielding means selects a set of images that are transmitted through the right eye and the left eye from the plurality of images according to the inclination detected by the inclination detection means ,
The stereoscopic image display apparatus , wherein when the inclination of the shielding means cannot be obtained from the inclination detection means, the shielding means transmits a predetermined set of images in the plurality of images to the right eye and the left eye .   The stereoscopic image display apparatus according to claim 1, wherein the plurality of images include images obtained by capturing subjects from different viewpoints, and include two or more sets of images that are transmitted through the right eye and the left eye.   The stereoscopic image display apparatus according to claim 1, wherein the sets of images that are transmitted through the right eye and the left eye have parallaxes corresponding to different inclinations.   4. The stereoscopic image display device according to claim 1, wherein the sets of images transmitted through the right eye and the left eye have different vertical parallaxes. 5.   5. The stereoscopic image display device according to claim 1, wherein at least one of the plurality of images is included in a set of images that are transmitted through the plurality of right eyes and left eyes.   The stereoscopic image display apparatus according to claim 1, wherein the plurality of images include a two-dimensional display image.   The three-dimensional image according to any one of claims 1 to 6, wherein the plurality of images are generated in consideration of a display timing when each of the plurality of images is displayed on the display unit in a time division manner. Image display device.   8. The device according to claim 1, wherein when the inclination of the shielding unit cannot be obtained from the inclination detection unit, the shielding unit transmits the same image in the plurality of images to the right eye and the left eye. 9. The three-dimensional image display apparatus described in 1. When there is no set of images to be transmitted through the right eye and the left eye corresponding to the inclination of the shielding means detected by the inclination detecting means, the shielding means determines a predetermined set of images in the plurality of images as the right eye and the left eye. The stereoscopic image display device according to any one of claims 1 to 8 , wherein When there is no set of images to be transmitted through the right eye and the left eye corresponding to the inclination of the shielding means detected by the inclination detecting means, the shielding means transmits the same image in the plurality of images to the right eye and the left eye. the stereoscopic image display device according to any one of claims 1 8, characterized in. Wherein the synchronization signal, any one of claims 1 to 10, characterized in that it comprises a signal indicating the correspondence between the set of slope and the right-eye and an image to be transmitted to the left eye of the shielding means which is detected by the inclination detecting means The three-dimensional image display device according to item 1. Wherein the synchronization signal, according to claim 1, characterized in that it comprises a signal representative of a set of images that the shielding means is transmitted through the right eye and the left eye when the said inclination detecting means not tilt is obtained of the shielding means Stereoscopic image display device. The synchronization signal represents a set of images that the shielding unit transmits to the right eye and the left eye when there is no set of images that are transmitted through the right eye and the left eye corresponding to the inclination of the shielding unit detected by the tilt detection unit. The stereoscopic image display apparatus according to claim 9 , further comprising a signal.   The stereoscopic image display apparatus according to claim 2, wherein the different viewpoints in each of the plurality of images are set in association with a preset inclination of the shielding unit.   The stereoscopic image display apparatus according to claim 2, wherein the different viewpoints in each of the plurality of images are set in association with an inclination of the shielding means selected by a user.   The stereoscopic image display apparatus according to claim 2, wherein the different viewpoints in each of the plurality of images are set according to a setting value associated with each of the plurality of images. Wherein the synchronization signal stereoscopic image display apparatus according to any one of claims 1, wherein the output by the infrared 16. Wherein the synchronization signal stereoscopic image display apparatus according to any one of claims 1, wherein the output by the radio 16. It said shielding means stereoscopic image display apparatus according to any one of claims 1 to 18, characterized in that a spectacle type. It said shielding means stereoscopic image display apparatus according to any one of claims 1 to 18, characterized in that a helmet type. The stereoscopic image display apparatus according to any one of claims 1 to 18 , wherein the shielding means has a surface shape covering a user's face.   5. The image adjusting device according to claim 4, further comprising: an image adjusting unit that adjusts a parallax in a vertical direction by relatively shifting a whole or a part of a screen area of an image of a certain viewpoint in a vertical direction and generating the plurality of images. The three-dimensional image display apparatus described in 1.   The stereoscopic image display apparatus according to claim 2, further comprising: an image adjusting unit configured to generate an image that can be regarded as an image obtained by capturing a subject from different viewpoints from an image of a certain viewpoint, and generate the plurality of images. A display step in which the display means displays a plurality of images sequentially in a time-sharing manner;
A synchronization step in which the synchronization means outputs a synchronization signal in synchronization with the display in the display step;
A shielding step in which a shielding means receives the synchronization signal, shields a right eye or a left eye according to the synchronization signal, and selectively transmits the plurality of images to the right eye and the left eye;
An inclination detecting step in which the inclination detecting means detects the inclination of the shielding means,
In the shielding step, the shielding means selects a set of images to be transmitted to the right eye and the left eye from the plurality of images according to the inclination of the shielding means detected by the inclination detection means ,
A stereoscopic image display method , wherein when the inclination of the shielding means cannot be obtained from the inclination detection means, the shielding means transmits a predetermined set of images in the plurality of images to the right eye and the left eye . A program for causing a computer to execute the stereoscopic image display method according to claim 24 .

Images