JP2012015818A - Three-dimensional image display device and display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To display a three-dimensional image adjusted to the viewing posture of a viewer of the three-dimensional image.SOLUTION: Image data A to D obtained by imaging a subject from three or more different view points A to D making a plane is stored in storage means. On the basis of the image data stored in the storage means, a display unit displays an image for the left eye and an image for the right eye. A view point detection unit detects the relative inclination of a line connecting the right and left eyes of the viewer viewing the display unit and the display unit to each other. On the basis of the relative inclination, a display control unit generates the image for the left eye and the image for the right eye from the image data.

Description

  The present invention relates to a stereoscopic image display device and a display method for displaying a stereoscopic image of a subject.

  For example, stereo cameras and the like described in Patent Documents 1 to 4 have begun to spread, and anyone can easily shoot a stereoscopic image of a subject. In addition, stereoscopic image display devices such as personal computer monitor devices and television devices capable of displaying stereoscopic images have begun to spread.

  The stereoscopic image display device can display a stereoscopic image in two directions, portrait / landscape, so that a viewer can view a stereoscopic image whether viewed in portrait orientation (upright) or landscape orientation (lie down). Some are capable of stereoscopic display (for example, Patent Documents 5 and 6).

  In the prior art described in Patent Document 7, a display screen of a mobile phone or the like is a stereoscopic video device, and a change in the posture of the user relative to the display screen (change in viewpoint) is detected. 3D images can be displayed.

  However, since the stereoscopic video device of Patent Document 7 is portable and used by the user in hand, there is a limit to the relative posture change between the user and the change of the horizontal viewpoint position. Only supported.

  Further, in the prior art of Patent Document 7, all video data viewed from seven horizontal viewpoints around the display object are held, and the user's posture is detected by the gyro sensor and matched with the viewpoint. Since video data is displayed, there is a problem that the amount of video data that must be retained becomes enormous, and when there is no video data corresponding to the user's viewpoint, the display object 3D video cannot be displayed.

JP 2010-114777 A JP 2008-167066 A Japanese Patent Laid-Open No. 2003-7994 JP 10-42314 A JP-A-6-258733 JP 2006-308864 A JP 2006-174434 A

  For example, when a viewer watches a stereoscopic video on a large television device, the user's face may be upright and the left and right eyes may be horizontal, while lying on the right side and the right eye on the lower side. In some cases, the left eye lies on the left side and the left eye is on the lower side, or the left and right eyes are tilted 45 degrees to the right or 45 degrees to the left because they are lying on their arms. In some cases, it is assumed that various postures can be taken freely.

  The conventional stereoscopic image display device does not consider various posture changes of the viewer, and there is a problem that the stereoscopic image cannot be viewed if the posture is changed. Also, if all of the stereoscopic video data corresponding to the various postures of the viewer must be held, a huge amount of video data is required, and the memory for storing this data also has a large capacity.

  For this reason, when there is no video data corresponding to the posture of the viewer, there arises a problem of how to display a stereoscopic video. In addition, when there are a plurality of viewers viewing the same screen, there is a problem of how to display an appropriate stereoscopic video for each viewer.

  An object of the present invention is to continuously view a stereoscopic video even when the viewer changes the viewing posture, and to display a stereoscopic image corresponding to various postures of the viewer using a small amount of stereoscopic video data. Another object of the present invention is to provide a stereoscopic image display device and a display method capable of displaying an appropriate stereoscopic video for each of a plurality of viewers.

  The stereoscopic image display apparatus and method according to the present invention store image data obtained by imaging a subject from three or more different viewpoints forming a plane in a storage unit, and based on the image data stored in the storage unit, The display unit displays the image for the eye and the image for the right eye, and the viewpoint detection unit detects the relative inclination between the line connecting the left and right eyes of the viewer viewing the display unit and the display unit, and the relative detection is performed. The display control unit generates a left-eye image and a right-eye image from the image data based on a typical inclination.

  In the stereoscopic image display apparatus and method according to the present invention, the stored image data is image data taken from four different viewpoints with respect to the subject, and the image data having the same direction as the relative inclination is the image data. The determination unit determines whether or not the stored image data is present, and the display control unit determines the left-eye image and the right-eye image from the stored image data according to a result of the determination unit. It is characterized by generating.

  According to the present invention, even when the viewer changes his / her posture while browsing, the stereoscopic image corresponding to the change is displayed on the screen, so that the viewer can continue to appreciate the stereoscopic image.

1 is an external perspective view of a stereoscopic image capturing apparatus for describing an embodiment of the present invention. FIG. 2 is an internal functional block diagram of the stereoscopic image capturing apparatus shown in FIG. 1. It is a surface schematic diagram of the image sensor shown in FIG. It is principal part explanatory drawing of the image pick-up element shown in FIG. It is a figure which shows an example of the to-be-photographed object image | photographed with the image pick-up element of FIG. FIG. 6 is a diagram illustrating a relationship between each viewpoint in FIG. 4 and the subject in FIG. 5. It is a figure which shows the example of a to-be-photographed object seen from each viewpoint of FIG. It is an internal functional block diagram of the stereoscopic image display apparatus which concerns on one Embodiment of this invention. It is explanatory drawing which produces | generates a parallax image from the picked-up image data of the several viewpoints recorded with the stereo image imaging device. It is a figure which shows the specific example of the parallax image shown in FIG. It is a flowchart which shows the process sequence which produces | generates a parallax image from the picked-up image data of multiple viewpoints. It is a surface schematic diagram of another image pick-up element replaced with the image pick-up element shown in FIG. It is principal part explanatory drawing of the image pick-up element shown in FIG. It is a figure which shows the relationship between each viewpoint of FIG. 13, and a to-be-photographed object. It is a figure which shows the example of a to-be-photographed object seen from each viewpoint of FIG. It is a flowchart which shows the process sequence which produces | generates a parallax image from the captured image data by each viewpoint shown in FIG. It is a figure which shows the specific example of the parallax image produced | generated by the process of FIG. It is a flowchart which shows the process sequence which produces | generates the parallax image of a different viewpoint (diagonal direction) from the captured image data by each viewpoint shown in FIG. It is a figure which shows the specific example of the parallax image of the diagonal direction produced | generated by the process of FIG. It is a flowchart which shows the whole process sequence which produces | generates a parallax image from several viewpoint information. It is a figure which shows an example of a viewpoint position in case there are three viewpoints. It is a flowchart which shows the stereo image display procedure in case there are two or more viewers. It is a figure which shows the example of a display of the image information of each viewpoint by the display procedure of FIG. It is a flowchart which shows the stereo image display procedure of another embodiment in case there are two or more viewers. It is a figure which shows the example of the display timing by the control procedure of FIG. 24 when the image data of each viewpoint shown in FIG. 14 is displayed on a screen sequentially.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Prior to description of a stereoscopic image display device, a stereoscopic image imaging device will be described and what viewpoint stereoscopic image data will be acquired will be described. To do. Next, how to display stereoscopic images from various viewpoints based on these stereoscopic image data will be described.

  FIG. 1 is an external perspective view showing an example of a monocular stereoscopic image pickup apparatus. In this stereoscopic image capturing apparatus 10, a lens barrel 13 that accommodates a monocular photographing lens system 12 is attached to the front portion of a camera housing 11 so as to be retractable. A release button 14 is provided, and a liquid crystal display (not shown in FIG. 1) (display 22 in FIG. 2) is provided on the back of the camera housing 11.

  FIG. 2 is a functional block configuration diagram inside the stereoscopic image capturing apparatus 10. An imaging element 14 provided with a pupil division optical system 13 is disposed on the back of the photographing lens system 12. The imaging element 14 is formed on the light receiving surface of the semiconductor substrate as a set of four pixels (pixel A, pixel B, pixel C, and pixel D). The detection signal of the pixel A and the detection signal of the pixel B of each set , The detection signal of the pixel C and the detection signal of the pixel D are taken into the image input controller 16 through the analog-to-digital (A / D) converter 15 provided for each pixel, and for each pixel A, B, C, D A digital captured image signal is output from the image input controller 16 to the internal bus 17.

  The bus 17 includes a CPU 20 that performs overall control of the entire stereoscopic image capturing apparatus 10 and known image processing (for example, offset processing, gamma conversion processing, synchronization processing, RGB / YC conversion processing) for the captured image signal of the pixel A. Etc.) to generate first captured image data of the subject, and to perform well-known image processing on the captured image signal of the pixel B to generate second captured image data of the subject, and to capture the captured image signal of the pixel C Signal processing for generating the third captured image data of the subject by performing known image processing on the image, and generating the fourth captured image data of the subject by performing known image processing on the captured image signal of the pixel D The unit 21 is connected to a video encoder 23 that converts these captured image signals into display images and displays them on the display unit 22.

  The bus 17 is further connected with a ROM 25, a RAM 26, an AE / AWB detection circuit 27, a media recording control unit 29 for controlling writing / reading on the recording medium 28, a speaker 30 and a microphone 31, and voice guidance, etc. The audio input / output processing unit 32 that performs the above and the 3D image signal processing unit 33 that has a parallax correction function between the first, second, third, and fourth captured image data are connected.

  The CPU 20 includes a timing generator 35 that drives the image sensor 14, a motor driver 36 that motor-drives the telephoto position, the focus position, and the like of the photographing lens system 12, and input instructions from the user via user interface (UI) processing. An operation unit 37 to be performed and a power source 38 are connected.

  FIG. 3 is a schematic surface diagram illustrating an example of the image sensor illustrated in FIG. 2. In this example, a CCD image sensor is illustrated, but a CMOS image sensor may be used.

  The imaging element 14 shown in FIG. 3 has a plurality of pixels 41 arranged in a two-dimensional array form on the light-receiving surface of a semiconductor substrate, in the example shown in a square lattice form, and a vertical charge transfer path (VCCD) along each pixel column. 42 is formed. A line memory 43 is formed along the transfer direction end of each vertical charge transfer path 42, a horizontal charge transfer path (HCCD) 44 is formed in parallel with the line memory 43, and an output end of the horizontal charge transfer path 44 is An amplifier 45 that outputs a voltage value signal corresponding to the transferred signal charge amount as a captured image signal is provided.

  The line memory 43 includes a buffer 43a for temporarily storing signal charges for each vertical charge transfer path 42, and a signal sent from the vertical charge transfer path 42 as described in, for example, Japanese Patent Application Laid-Open No. 2006-157624. By holding the charge temporarily and controlling the timing when the signal charge is transferred to the horizontal charge transfer path 44, it has a function of performing pixel addition (mixing of signal charges) in the horizontal direction. In the present embodiment, the image sensor 14 equipped with the line memory 43 has been described. However, when the stereoscopic image data described below is taken, pixel addition (pixel mixing) using the line memory 43 is not performed.

  “R”, “G”, and “B” described on each pixel 41 in the drawing represent the color of the color filter (R = red, G = green, B = blue). In the image sensor 14 of the present embodiment, the nearest four pixels arranged vertically and horizontally and arranged at the four corner positions of each square are regarded as one set, and color filters of the same color are stacked on the four pixels in the set. The color filters are arranged so that the color arrangement for each set is a Bayer arrangement as a whole.

  And the micro lens 47 shown with a broken-line circle is provided in the upper part of 1 set of 4 pixels shown in FIG. These microlenses 47 constitute the pupil division optical system 13 shown in FIG. 2, and when light is incident on the respective lower pixels A to D through the microlens 47, they are directed to incident light to the respective pixels A to D. Thus, images having different parallaxes are captured by the pixels A to D.

  FIG. 4 is a diagram showing 4 × 4 = 16 pixels among the pixels 41 shown in FIG. 3, and shows four pixels as one set of four pixels and each micro lens 47 for pupil division. R, G, and B described in the center of each of the four pixels represent the color of the color filter. Among the four pixels, the upper left diagonal pixel A, the upper right diagonal pixel B, the lower left diagonal pixel C, and the lower right diagonal Is a pixel D. These pixels A to D are the pixels A to D described in the image sensor 14 of FIG.

  FIG. 5 shows a tree 49 as a subject. When the photographer's eyes are present on the front side of the sheet of FIG. 4 and the subject tree 49 is viewed through each group of four pixels, as shown in FIG. ), Image data obtained by viewing the tree 49 from the upper right (viewpoint B), and pixel C obtained image data by viewing the tree 49 from the lower left (viewpoint C). The pixel D obtains image data obtained by viewing the tree 49 from the lower right (viewpoint D).

  7A, 7B, 7C, and 7D are specific examples of image data viewed from the viewpoints A, B, C, and D described with reference to FIG. Captured image data (FIG. 7A) obtained by image processing of the captured image signal output from each set of pixels A by the signal processing unit 21 is stored in the recording medium 28 and output from each set of pixels B. The captured image data (FIG. 7B) obtained by subjecting the captured image signal to image processing by the signal processing unit 21 is stored in the recording medium 28, and the captured image signal output from each set of pixels C is stored in the signal processing unit. Captured image data (FIG. 7C) obtained by image processing by the image 21 is stored in the recording medium 28, and the captured image signal output from each set of pixels D is obtained by image processing by the signal processing unit 21. The captured image data (FIG. 7D) is stored in the recording medium 28.

  These four captured image data of the tree 49 in FIGS. 7A to 7D are associated with each other as a multi-picture format and stored in the recording medium 28. The captured image data to be recorded may be a still image or a moving image.

  In this way, when viewing captured image data taken from the viewpoints A, B, C, and D and stored in the recording medium 28 in a multi-picture format with a stereoscopic image display device capable of displaying a 3D image, the recording medium 28 is used. The viewer wears the stereoscopic image display device and the viewer views the stereoscopic image.

  FIG. 8 is a functional block configuration diagram of a stereoscopic image display apparatus according to an embodiment of the present invention. The stereoscopic image display device 50 includes a CPU 52 connected to the bus 51, an image selection unit 53, and a viewpoint detection unit 54 that detects the posture of the viewer (the direction in which the right eye viewpoint and the left eye viewpoint are aligned), The audio input / output to which the 3D image signal processing unit 55, the image signal processing unit 56, the ROM 57, the RAM 58, the speaker 60, and the microphone 61 having the same functions as the 3D image signal processing unit 33 described in FIG. Depending on the processing unit 61, the media read control unit 62 to which the recording medium 28 taken out from the stereoscopic image pickup device 10 is mounted, the video encoder 66 to which the display unit 65 for displaying 3D images is connected, and the 3D image display method. An auxiliary tool control unit 68 to which a viewer's viewing auxiliary tool 67 such as necessary polarized glasses is connected is connected. An operation unit 69 and a power source 70 are connected to the CPU 52. The display unit 65 uses a display unit as described in Patent Document 6 that can display a stereoscopic image in both portrait and landscape orientations.

  A display unit that can display stereoscopic images in both portrait and landscape orientations is usually placed in a rectangular frame like a television device and installed in the room so that the upper edge of the rectangle is horizontal. Is normal. As a result, the “horizontal” and the vertical / horizontal “vertical” are orthogonal to each other. In the following description, the direction along the upper edge of the frame will be referred to as the horizontal direction of the display unit, and this will be described as a reference. That is, when the viewer is standing upright with respect to the ground, the line connecting the left and right eyes is horizontal with respect to the ground, so the display unit is provided so that an image that can be viewed stereoscopically is provided to the viewer in that state. The horizontal direction when installed is referred to as the “horizontal direction of the display unit” (for example, when the display unit of Patent Document 6 is used, in the case of portrait orientation (vertical placement), the direction A in FIG. In the case of landscape orientation (horizontal orientation), the direction of FIG.

  The viewpoint detection unit 54 of the stereoscopic image display device 50 detects the relative inclination of the viewpoint with respect to the horizontal direction of the display unit. For example, a camera is provided at the upper edge of the frame body, the camera detects a face part such as the viewer's face or eyes, and the line connecting the viewer's left and right eyes is relative to the horizontal direction of the display unit. Detect tilt. Further, when the stereoscopic display method is a polarizing method, a field sequential method using a liquid crystal shutter, or the like, the tilt of the viewpoint with respect to the display unit may be detected by an attitude detection unit provided in the glasses.

  When the recording medium 28 is attached to the stereoscopic image display device 50 and an image to be viewed is selected from the operation unit 69, the image selection unit 53 displays the stereoscopic image data necessary for display from the recording medium 28. Read based on the detected viewpoint. In the embodiment, the stereoscopic image display device 50 is configured to read image data via the recording medium 28, but may be configured to read image data via a signal line such as a network.

  Since the stereoscopic image data is normally recorded in a compressed form such as JPEG format or MPEG format, the image signal processing unit 56 decompresses the compressed data and transfers it to the 3D image signal processing unit 55. Based on the viewer's viewpoint, the 3D image signal processing unit 55 passes the stereoscopic image data to be combined as described later to the video encoder 66 and displays the stereoscopic image on the 3D image display unit 65.

  FIG. 9 is an explanatory diagram of image data to be combined (combined) when displaying a stereoscopic image, and FIG. 10 is the same diagram as FIG. 9, but is a specific example of an image to be combined using an actual tree 49 image. An example is shown. FIG. 11 is a flowchart showing a processing procedure of image combination synthesis.

  On the recording medium 28, image data A, B, C, and D viewed from four viewpoints shown in FIGS. 7A, 7B, 7C, and 7D are recorded. The viewer's posture is determined (step S1), and the left and right eyes are relatively close to the horizontal direction with respect to the horizontal direction of the display unit (the angle with respect to the horizontal direction is smaller than a predetermined threshold (for example, 45 degrees)). ) (Determination result is Yes), the process proceeds to step S2 to select a horizontal parallax image. Specifically, an image of viewpoint A and an image of viewpoint C are added and synthesized to generate an A + C image, which is used as a left-eye display image. Further, the viewpoint B image and the viewpoint D image are added and synthesized to generate a B + D image, which is used as a right-eye display image, and these images are displayed on the display unit 65 in the next step S3.

  If the result of the determination in step S1 is negative (not in the horizontal direction, the left and right eyes are relatively close to the vertical direction relative to the horizontal direction of the display unit), the process proceeds to step S4, and the viewer's posture Thus, it is determined whether or not the detection position of the right eye is below the detection position of the left eye. If the determination result is affirmative (Yes), the process proceeds to step S5, and a parallax image with the right eye down is selected. That is, the image of viewpoint A and the image of viewpoint B are added and synthesized to generate an A + B image, which is used as an image for left eye display. Then, the image of viewpoint C and the image of viewpoint D are added and synthesized to generate a C + D image, which is used as a right-eye display image, and the process proceeds to step S3 where these images are displayed on the display unit 65.

  If the determination result in step S4 is negative, it is determined that the left eye detection position is below the right eye detection position, and the process proceeds to step S6 to select a parallax image with the left eye below. . That is, an image of viewpoint C and an image of viewpoint D are added and combined to generate a C + D image, which is used as an image for left eye display. Then, the image of the viewpoint A and the image of the viewpoint B are added and synthesized to generate an A + B image, which is used as an image for right eye display, and the process proceeds to step S3 to display these images on the display unit 65.

  Even if the viewer's posture is changed halfway by repeating the processes of steps S1, S2, S3, S4, S5, and S6 every predetermined time, for example, when the posture changes to a lying posture during viewing. The viewer can continuously view the stereoscopic image. Moreover, in the present embodiment, step S4 is provided, and it is determined whether the right eye is down or the left eye is down, so that the right eye image and the left eye image are erroneously displayed in reverse even with the same tilt. Disappears.

  In FIGS. 9 to 11, the description of the oblique 45-degree stereoscopic image is omitted, but as is clear from FIG. 6, the viewpoint A image is for the left eye and the viewpoint D image is for the right eye. For example, stereoscopic image data tilted 45 degrees to the right can be displayed. If the image of viewpoint C is for the left eye and the image of viewpoint B is for the right eye, stereoscopic image data tilted 45 degrees to the left can be displayed. .

  FIG. 12 is a schematic view of the surface of an image sensor 80 having a pixel arrangement different from that in FIG. In this example, a CCD image sensor is used, but a CMOS image sensor may be used as in the embodiment of FIG.

  In the image sensor 80 of the present embodiment, even-numbered pixel rows are arranged with a ½ pixel pitch shifted from odd-numbered pixel rows, and a vertical charge transfer path formed along each pixel column ( 3 is different from that of FIG. 3 in that the VCCD 42 is meandered so as to avoid each pixel 41, the same members as those in FIG.

  In the image sensor 80 of this embodiment, the same color filter is laminated as one set of four pixels. The pixel arrangement of the nearest four pixels constituting one set of the present embodiment is a configuration in which squares made up of four pixels are arranged in a rhombus shape as shown in FIG. One microlens 47 and the same color filter are laminated. The pixel A provided at the left corner of the rhombus-shaped array, the pixel B provided at the upper corner, the pixel C provided at the lower corner, and the pixel D provided at the right corner are provided below one microlens. Splitting is performed.

  FIG. 14 is a diagram illustrating viewpoint positions A to D when an image of the tree 49 is viewed through the pixels A to D. The viewpoint A (pixel A) sees the subject 49 slightly from the left side, the viewpoint B (pixel B) sees the subject 49 slightly from the upper side, and the viewpoint C (pixel C) views the subject 49 slightly from the lower side. The viewpoint D (pixel D) sees the subject 49 slightly from the right side.

  If the image sensor 14 in FIG. 3 and the image sensor 80 in FIG. 12 have the same number of pixels and the same chip size, the parallax in the horizontal direction (between pixel A and pixel D) of the image sensor 80 in FIG. This is wider than the parallax of the element 14 in the horizontal direction (between pixel A and pixel B), and is advantageous as data of a stereoscopic image.

  FIGS. 15A, 15B, 15C, and 15D are specific examples of image data viewed from the viewpoints A, B, C, and D described in FIG. The captured image data (FIG. 15A) of the tree 49 captured by each set of pixels A is stored in the multi-media format (hereinafter the same) on the recording medium 28 of FIG. The captured image data of the tree 49 (FIG. 15B) is stored in the recording medium 28, and the captured image data of the tree 49 (FIG. 15C) captured by each set of pixels C is stored in the recording medium 28. Then, the captured image data (FIG. 15D) of the tree 49 captured by each set of pixels D is stored in the recording medium 28.

  FIG. 16 is a flowchart showing a processing procedure in the stereoscopic image display device 50 that reads captured image data from the recording medium 28 storing the image data of FIGS. 15A to 15D and displays a stereoscopic image. FIG. 17 is a diagram showing image data selected according to the flowchart of FIG. 16.

  First, in step S11, the relative inclination of the viewer's viewpoint with respect to the horizontal direction of the display unit is detected, and the direction of the inclination is close to the horizontal (the angle with respect to the horizontal direction is smaller than a predetermined threshold (for example, 45 degrees)) ( When the determination result is Yes), the process proceeds to step S12, where the image data A captured by the pixel A is selected as the left-eye display image and the image data D captured by the pixel D is selected as the right-eye display image. In the next step S13, these image data A and D are displayed on the stereoscopic display screen.

  If the result of determination in step S11 is that the tilt direction is not nearly horizontal (when it is close to vertical and the determination result is No), the process proceeds to step S14, where the tilt is in the vertical direction and the detection position of the right eye is It is determined whether or not the detection position of the left eye is below. If the determination result is affirmative (Yes) and the right eye is on the lower side, the process proceeds to step S15, where the image data B is selected as the left-eye display image and the image data C is selected as the right-eye display image. In step S13, the image data B and C are displayed as a stereoscopic image.

  If the result of determination in step S14 is negative (No), that is, if the direction of tilt is close to the vertical direction and the left eye is on the lower side, the process proceeds to step S16, and the image data C is displayed as an image for left eye display. And the image data B is selected as the right-eye display image, and the image data C and B are displayed as a stereoscopic image in step S13.

  By repeatedly performing the processes of steps S11 to S16 every predetermined time, even when the viewer's posture is changed during the viewing, for example, when the posture changes to a lying-down posture during viewing, the viewer can continuously You can appreciate the images. Moreover, since step S14 is also provided in this embodiment and it is determined whether the right eye is down or the left eye is down, the right-eye image and the left-eye image can be erroneously displayed in reverse even with the same tilt. Disappear.

  In the process of FIG. 16, for the sake of simplicity, the case where the inclination direction is oblique is not considered, but FIG. 18 shows a processing procedure including the case where the inclination direction is oblique. FIG. 19 is a diagram showing image data selected in the process of FIG.

  In step S21 of FIG. 18, first, it is determined whether or not the relative inclination of the viewer's viewpoint with respect to the horizontal direction of the display unit is closer to the diagonal direction than the vertical and horizontal directions. If the result of this determination is negative (No), the process proceeds to step S22, image selection is performed in the processing steps of steps S11, S12, S14, S15, and S16 described in FIG. 18, and then the process proceeds to step S13. The image data selected with is displayed.

  If the determination result of step S21 is affirmative (Yes), the process proceeds to the next step S23, and it is determined whether or not the detection position of the right eye is below the detection position of the left eye. If the determination result is affirmative (Yes), the process proceeds to step S24 to generate an image A + B obtained by adding and synthesizing the image data A and the image data B as an image for left eye display, and image data C as an image for right eye display. And image data D are added and synthesized, and the process proceeds to the next step S13, where the images A + B and C + D are displayed as a stereoscopic image on the screen.

  If the result of determination in step S23 is negative (No), that is, if the detection position of the left eye is lower and the relative tilt of the viewer's viewpoint with respect to the horizontal direction of the display unit is oblique, the process proceeds to step S25 and left Next, an image A + C obtained by adding and synthesizing image data A and image data C is generated as an image for eye display, and an image B + D obtained by adding and combining image data B and image data D is generated as an image for right eye display. Proceeding to S13, the images A + C and B + D are displayed on the screen as stereoscopic images.

  In this way, if it is possible to deal with an oblique viewpoint, it is possible to adjust the tilt direction more finely. As a stereoscopic image display device that can easily cope with an oblique direction, it is desirable to use a polarization method by wearing glasses, a field sequential method, or the like that can easily separate the left and right without depending on the direction of the viewpoint.

  FIG. 20 is a flowchart showing the basics of the entire processing procedure in the stereoscopic image display apparatus. The stereoscopic image display device 50 detects the direction of the viewer's viewpoint. As described above, the stereoscopic image display device 50 shoots the viewer's face image with a camera or the like attached to the stereoscopic image display device, as described above. It is preferable to realize by detecting the straight line connecting the eyes and the inclination thereof with less burden on the viewer side. However, in the case of a method of wearing glasses (polarized light, field sequential), the glasses may have a function of transmitting tilt information detected by a gyro sensor or the like.

  First, in step S31, the stereoscopic image display device 50 detects the relative inclination and direction of the viewer's (viewer) viewpoint with respect to the horizontal direction of the display unit, and in the next step S32, in FIGS. The described image selection is performed, and the left and right parallax images are displayed on the screen in the next step S33 (= step S13 in FIG. 18).

  In the next step S34, it is determined whether or not the viewing (viewing) of the stereoscopic image has ended. If not, the process returns to step S31. As a result, even when the viewer changes the posture while viewing the stereoscopic image, for example, the posture changes to a posture to lie down, the parallax image according to the posture change is selected, synthesized, and displayed on the screen. Even if the viewer changes his / her posture, the viewer can continue to view the stereoscopic image without a sense of incongruity.

  In the above-described embodiment, the left-eye and right-eye images that can be viewed stereoscopically are selected or combined and displayed on the screen based on the captured image data of the four viewpoints captured by the monocular stereoscopic image capturing device. However, the stereoscopic image capturing apparatus does not need to be monocular, and may include four systems of photographing lens systems and image sensors corresponding to the pixels A, B, C, and D.

  Alternatively, a three-lens imaging device may be used, and it is possible to obtain image data of three viewpoints with a single eye and synthesize images of a plurality of viewpoint directions. For example, as shown in FIG. 21, if images of viewpoints A, B, and C that form an equilateral triangle with the subject as the center can be captured, a parallax image in the horizontal direction can be obtained between the images B and C, A parallax image in the vertical direction can be obtained from the image A and the image (B + C) / 2, and a parallax image inclined by 30 degrees can be obtained from the image B and the image (A + C) / 2. A parallax image tilted at an angle of 60 degrees with the image B can be obtained. Note that the viewpoint is not limited to that shown in FIG.

  In the case of three viewpoints, as described above, since the parallax images of vertical, horizontal, right 30 degrees, right 60 degrees, left 30 degrees, and left 60 degrees can be synthesized, The parallax images that are most stereoscopically viewable among these parallax images are synthesized even if they do not coincide with the tilt direction of the viewpoint relative to the horizontal direction. The same applies to the case of four viewpoints.

  In the embodiment described above, it is assumed that there is only one viewer. However, when there are a plurality of viewers, it is preferable that each person can view a stereoscopic image.

  FIG. 22 is a flowchart illustrating a processing procedure that can handle a plurality of viewers. First, display data is read from the recording medium 28 (step S41), and it is determined whether there is stereoscopic image display data (step S42). If there is no stereoscopic image display data in the recording medium 28 and there is only 2D image (planar image) data, the process proceeds to step S43, where the 2D image data is displayed and the process is terminated.

  If there is stereoscopic image display data as a result of the determination in step S42, it is next determined whether or not the stereoscopic image display data exists as a plurality of viewpoint information (step S44). That is, it is determined whether the data exists as the stereoscopic image data of the viewpoints A, B, C, and D in FIG. 14 (or exists as the stereoscopic image data of the start points A, B, and C in FIG. 21).

  When there is no stereoscopic image data with at least three or more viewpoints (when there is only two viewpoint information), the process proceeds from step S44 to step 45, where the stereoscopic image data with the two viewpoints is displayed and viewed. (Step S46), the process ends when viewing ends, and the process returns from step S46 to step 44 when viewing ends.

  As a result of the determination in step S44, if there is data of a stereoscopic image from at least three viewpoints, the process proceeds from step S44 to step S47, and it is determined whether there are a plurality of viewers. If there is a single viewer, the process proceeds to step S48, the process of FIG. 16 or 18 is performed, and the process proceeds to step S46.

  If the result of determination in step S47 is that there are a plurality of viewers, the process proceeds to step S50, and the posture of each viewer is calculated from the camera video or the like. In the next step S51, the average posture (average tilt direction) of a plurality of viewers is calculated, the average parallax image is synthesized and displayed on the screen (step S52), and the process proceeds to step S46.

  In this way, since the images in the average inclination direction of a plurality of viewers are combined and displayed, it is possible for each person to view the stereoscopic image in their respective postures.

  FIG. 23 is a diagram showing changes in the viewpoint information displayed by the processing procedure of FIG. The first and second users (viewers) initially look at the screen while maintaining a horizontal orientation, but the second user (viewers) lie down on the way and watch the screen (the tilt direction is (Vertical direction), then wake up again and watch the screen.

  At first, in order to obtain a horizontal parallax image, the image of viewpoint A shown in FIG. 14 is displayed for the left eye and the image of viewpoint D is displayed for the right eye. Since the posture has changed, based on the average value of the postures of the first and second viewers, the image of the viewpoint A that is 45 degrees obliquely is displayed for the left eye and the image of the viewpoint B is displayed for the right eye. Thereby, even if the second viewer changes the posture, the first and second viewers can continue to view the stereoscopic image.

  In the case of this embodiment, a viewer who cannot perform stereoscopic display may occur depending on the posture. At this time, a guide display or the like may be given to the viewer so as to notify the viewer of a posture capable of stereoscopic display.

  FIG. 24 is a flowchart illustrating a processing procedure corresponding to a plurality of viewers according to another embodiment. This embodiment is different from the embodiment of FIG. 22 only in steps S50, S51, and S52, and only steps S50a, S51a, and S52a that replace them will be described.

  In step S50a that proceeds when there are a plurality of viewers, each viewpoint data A, B, C, and D is frame-sequentially driven. In the next step S51a, the viewing posture of each viewer is detected. In the next step S52a, a timing signal is transmitted to the viewing device according to the viewing posture of each person, and the process proceeds to step S46.

  FIG. 25 is a diagram showing a change in information displayed by the processing procedure of FIG. By step S50a, each viewpoint data (image data) is cyclically switched and displayed on the screen of the stereoscopic image display device 50 in the order of viewpoint A → viewpoint B → viewpoint C → viewpoint D → viewpoint A →. 25 left edge). The first and second viewers viewing this screen are wearing viewing equipment such as active shutter glasses, and the stereoscopic image display device 50 transmits a control signal to the viewing equipment and each viewer's posture. The shutter opening / closing control of the glasses is performed separately for each viewer at a timing according to the above.

  As for the first viewer's posture, the user views the screen in such a manner that the tilt direction is the vertical direction at first, and wakes up halfway to make the tilt direction horizontal. At this time, when the tilt direction is vertical, the image of the viewpoint C is displayed for the left eye and the image of the viewpoint B is displayed for the right eye. Therefore, the glasses of the first viewer are displayed at the timing when the image of the viewpoint C is displayed. The left-eye shutter of the first viewer is set to “open”, and the right-eye shutter of the first viewer's glasses is set to “open” at the timing when the image of the viewpoint B is displayed on the screen.

  When the first viewer changes posture and the tilt direction becomes horizontal, the shutter of the left eye of the first viewer's glasses is set to “open” at the timing when the image of the viewpoint A is displayed, and the viewpoint D The shutter of the right eye of the first viewer's glasses is set to “open” at the timing when the image is displayed on the screen. Thereby, even if a 1st browsing person changes an attitude | position, he can always appreciate a stereo image.

  Similarly to the first viewer, the second viewer can change the posture in the middle by opening the left and right shutters of the second viewer at the timing when the corresponding viewpoint information is displayed on the screen. However, it is always possible to view stereoscopic images.

  In the case of this active shutter method, since an afterimage of the eye is used, for example, the timing at which the information on the viewpoint A is displayed and the timing at which the information on the viewpoint B is displayed when performing the above-described image composition (A + B). If the shutter is “open” in both cases, it is possible to perform image composition on the afterimage of the eye.

     In the case of this embodiment, a viewpoint data driving period that does not apply to any viewer's viewing posture may be assigned to other viewpoint data. In such a case, it is preferable to assign them evenly in order to prevent a luminance difference or a screen flicker due to a difference in display time for each viewpoint data. In addition, a viewpoint data driving period that does not apply to any viewer's viewing posture may be assigned to the display of black data to reduce crosstalk due to afterimages.

  According to the present embodiment, even when a plurality of viewers are viewing the same screen in different viewing postures, it is possible to view the correct stereoscopic video corresponding to each viewer's viewing posture, making it easier and easier 3D images can be viewed.

  As described above, the stereoscopic image display apparatus and method according to the embodiment store in the storage means image data obtained by imaging the subject from three or more different viewpoints forming a plane, and the image stored in the storage means Based on the data, the display unit displays the image for the left eye and the image for the right eye, and the viewpoint detection unit detects the relative inclination between the line connecting the left and right eyes of the viewer who browses the display unit and the display unit And a display control unit generates a left-eye image and a right-eye image from the image data based on the relative inclination.

  In the stereoscopic image display apparatus and method according to the embodiment, the stored image data is image data captured from four different viewpoints with respect to the subject, and the image data having the same orientation as the relative inclination is the image data. The determination unit determines whether or not the stored image data is present, and the display control unit determines the left-eye image and the right-eye image from the stored image data according to a result of the determination unit. It is characterized by generating.

  Further, the display control unit of the stereoscopic image display apparatus and method according to the embodiment selects a left-eye image and a right-eye image from the stored image data when the determination unit determines that there is image data. When the determination unit determines that there is no image data, the stored image data is combined to generate a left-eye image and a right-eye image that match the orientation of the viewpoint.

  Further, the determination means of the stereoscopic image display apparatus and method according to the embodiment determines whether or not there is image data in the same direction, whether or not there is image data whose angle with respect to the relative inclination is less than a predetermined threshold. It is characterized by judging whether there is no.

  Further, in the stereoscopic image display apparatus and method according to the embodiment, the display control unit may be configured for the left eye according to whether or not the viewer's right eye detected by the viewpoint detection unit is below the left eye. An image and the right-eye image are generated.

  Further, the display control unit of the stereoscopic image display apparatus and method according to the embodiment may be configured such that when the viewpoint detection unit detects a change in the relative inclination, the left-eye image according to the change in the relative inclination. And generating an image for the right eye.

  Further, in the stereoscopic image display apparatus and method according to the embodiment, when the viewpoint detection unit detects the relative inclinations of a plurality of persons as the viewer, the average value of the relative inclinations of the plurality of persons is averaged. Obtained by the value viewpoint detection means, the display control unit generates a left-eye image and a right-eye image in accordance with the average value.

  In addition, the display control unit of the stereoscopic image display apparatus and method according to the embodiment, when the viewpoint detection unit detects the relative inclination of the plurality of persons, the plurality of persons A left-eye image and a right-eye image corresponding to the relative inclination are generated, and the left-eye image and the right-eye image are displayed in the order of the relative inclinations of the plurality of persons.

  In addition, the stereoscopic image display apparatus and method according to the embodiment are a stereoscopic image display apparatus and method for a stereoscopic image display system including a plurality of browsing holders on which each of the plurality of persons is hung. The browsing holder controls the image of the display unit separately according to the relative inclination of each of the plurality of persons.

  Further, the stereoscopic image display apparatus and method according to the embodiment are characterized in that the four viewpoints are arranged in a square.

  In addition, the stereoscopic image display apparatus and method according to the embodiment are characterized in that the four viewpoints are arranged on the upper left, upper right, lower left, and lower right of the subject.

  In addition, the stereoscopic image display apparatus and method according to the embodiment are characterized in that the four viewpoints are arranged on the top, bottom, left, and right of the subject.

  According to the embodiment described above, even if the viewer changes the viewing posture, the stereoscopic video can be continuously viewed before and after the change, and the stereoscopic video corresponding to the various postures of the viewer using a small amount of stereoscopic video data. An image can be displayed, and an appropriate stereoscopic video can be displayed for each of a plurality of viewers.

  The stereoscopic image display method and apparatus according to the present invention allow a viewer (viewer) to continue to view a stereoscopic image even when the viewer (viewer) changes his / her posture while browsing (viewing). The present invention is useful when applied to a stereoscopic image display device such as a device or a television device.

DESCRIPTION OF SYMBOLS 10 Stereoscopic imaging device 12 Shooting lens 13 Pupil division | segmentation optical system 14 Image pick-up element 20 CPU
21 Image processing device 28 Recording medium 41 Pixel 42 Vertical charge transfer path (VCCD)
47 Microlens 50 Stereoscopic Image Display Device 52 CPU
53 Image selection unit 54 View point detection unit 55 3D image signal processing unit 65 3D image display unit

Claims (24)

Storage means for storing image data obtained by imaging a subject from three or more different viewpoints forming a plane;
A display unit for displaying a left-eye image and a right-eye image based on the image data stored in the storage unit;
A viewpoint detection unit that detects a relative inclination between a line connecting the left and right eyes of a viewer who browses the display unit and the display unit;
A stereoscopic image display device comprising: a display control unit that generates a left-eye image and a right-eye image from the image data based on the relative inclination. The stereoscopic image display device according to claim 1,
The stored image data is image data taken from four different viewpoints with respect to the subject, and it is determined whether or not there is image data in the same direction with respect to the relative inclination in the stored image data. Determining means for
The display control unit generates a left-eye image and a right-eye image from the stored image data according to the result of the determination unit. The stereoscopic image display device according to claim 1 or 2,
When the determination unit determines that there is image data, the display control unit selects a left-eye image and a right-eye image from the stored image data, and when the determination unit determines that there is no image data. A left-eye image and a right-eye image generated by synthesizing the stored image data to match the direction of the viewpoint are generated.   4. The stereoscopic image display device according to claim 2, wherein the determination unit determines whether or not there is image data in the same direction, and an angle with respect to the relative inclination is less than a predetermined threshold value. A stereoscopic image display device that determines whether image data exists or not.   5. The stereoscopic image display device according to claim 1, wherein the viewer's right eye detected by the viewpoint detector is lower than the left eye. The display control unit is a stereoscopic image display device that generates the left-eye image and the right-eye image. The stereoscopic image display device according to any one of claims 1 to 5,
The display control unit generates a left-eye image and a right-eye image according to the relative inclination change when the viewpoint detection unit detects the relative inclination change. Stereoscopic image display device. The stereoscopic image display device according to any one of claims 1 to 6,
When the viewpoint detection unit detects the relative inclination of a plurality of persons as the viewer, the viewpoint detection unit further includes an average viewpoint detection unit that calculates an average value of the relative inclinations of the plurality of persons.
The display control unit generates a left-eye image and a right-eye image in accordance with the average value. The stereoscopic image display device according to any one of claims 1 to 7,
The display control unit, when the viewpoint detection unit detects the relative inclination of the plurality of persons, from the stored image data, a left-eye image corresponding to the relative inclination of the plurality of persons and a right A stereoscopic image display device that generates an image for an eye and displays an image for a left eye and an image for a right eye in the order of the relative inclinations of the plurality of persons. A stereoscopic image display system comprising: the stereoscopic image display device according to claim 8; and a plurality of browsing holders on which each of the plurality of people is hung.
The plurality of browsing holders separately control the image on the display unit according to the relative inclination of each of the plurality of persons.   The stereoscopic image display device according to any one of claims 2 to 8, wherein the four viewpoints are arranged in a square shape.   The stereoscopic image display device according to claim 10, wherein the four viewpoints are arranged at an upper left corner, an upper right corner, a lower left corner, and a lower right corner of the subject.   The stereoscopic image display device according to claim 10, wherein the four viewpoints are arranged on the top, bottom, left, and right of the subject. Storing image data obtained by imaging a subject from three or more different viewpoints forming a plane in a storage means;
Based on the image data stored in the storage means, the display unit displays an image for the left eye and an image for the right eye,
The viewpoint detection unit detects a relative inclination between a line connecting the left and right eyes of a viewer who browses the display unit and the display unit,
A stereoscopic image display method in which a display control unit generates a left-eye image and a right-eye image from the image data based on the relative inclination. The stereoscopic image display method according to claim 13,
The stored image data is image data taken from four different viewpoints with respect to the subject, and it is determined whether or not the stored image data has image data in the same direction as the relative inclination. Judgment,
The display control unit generates a left-eye image and a right-eye image from the stored image data according to the result of the determination unit. The stereoscopic image display method according to claim 13 or 14,
When the determination unit determines that there is image data, the display control unit selects a left-eye image and a right-eye image from the stored image data, and when the determination unit determines that there is no image data. A method for displaying a three-dimensional image, comprising: synthesizing the stored image data to generate a left-eye image and a right-eye image in accordance with the orientation of the viewpoint.   16. The stereoscopic image display method according to claim 14, wherein the determination unit determines whether or not there is image data in the same direction, and an angle with respect to the relative inclination is less than a predetermined threshold value. A stereoscopic image display method for determining whether there is image data.   17. The stereoscopic image display method according to claim 13, wherein the viewer's right eye detected by the viewpoint detection unit is below the left eye. The display control unit is a stereoscopic image display method for generating the left-eye image and the right-eye image. A stereoscopic image display method according to any one of claims 13 to 17,
The display control unit generates a left-eye image and a right-eye image according to the relative inclination change when the viewpoint detection unit detects the relative inclination change. 3D image display method. The stereoscopic image display method according to any one of claims 13 to 18,
When the viewpoint detection unit detects the relative inclination of a plurality of persons as the viewer, an average value viewpoint detection means obtains an average value of the relative inclinations of the plurality of persons,
The display control unit generates a left-eye image and a right-eye image in accordance with the average value. The stereoscopic image display method according to any one of claims 13 to 19,
The display control unit, when the viewpoint detection unit detects the relative inclination of the plurality of persons, from the stored image data, a left-eye image corresponding to the relative inclination of the plurality of persons and a right A stereoscopic image display method comprising: generating an image for an eye and displaying an image for a left eye and an image for a right eye in the order of the relative inclinations of the plurality of persons. A stereoscopic image display method of a stereoscopic image display system comprising a plurality of browsing holders worn by each of the plurality of persons according to claim 20,
The three-dimensional image display method, wherein the plurality of browsing holders separately control images on the display unit according to the relative inclination of each of the plurality of persons.   21. The stereoscopic image display method according to any one of claims 14 to 20, wherein the four viewpoints are arranged in a square.   23. The stereoscopic image display method according to claim 22, wherein the four viewpoints are arranged at an upper left corner, upper right corner, lower left corner, and lower right corner of the subject.   23. The stereoscopic image display method according to claim 22, wherein the four viewpoints are arranged on the top, bottom, left, and right of the subject.

Images