JP6677385B2 - Stereoscopic display device and parallax image correction method - Google Patents

Description

  The present invention relates to a stereoscopic display device that generates a floating image in the air by forming a 3D (Dimension) image in the air, and a method of correcting a parallax image in the stereoscopic display device.

  As a technique for causing a viewer to recognize a stereoscopic image, a stereoscopic image display method based on binocular parallax using a positional difference between a left eye and a right eye is generally used. This method is based on the principle of a stereogram that allows the left eye and the right eye to visually recognize different two-dimensional images, and recognizes a three-dimensional stereoscopic image from the difference in how the brain looks. In addition, as a method for displaying a stereoscopic image, there is a method using glasses or a naked eye method without using glasses. The naked eye method includes a binocular method or a multi-eye method according to the number of viewpoints of an observer.

  In order to express a three-dimensional image by the naked eye method using a two-dimensional display such as a normal flat panel display, pixels for displaying the left-eye image and the right-eye image are provided on the two-dimensional display, and the two-dimensional display is observed. Optical means such as a lenticular lens in which a cylindrical lens is arranged or a parallax barrier in which a slit-shaped light-shielding pattern is arranged, and the left-eye image and the right-eye image on the screen are spatially separated from each other. A method is used in which the right and left eyes are separated and visually recognized.

  The above technique is to display an image three-dimensionally on the screen of a two-dimensional display, and furthermore, a technique to display an image as if an object is floating in the air has been developed. For example, using an imaging optical system such as a convex lens or a concave mirror, a two-dimensional display is arranged to be inclined with respect to the optical axis of the imaging optical system, and a two-dimensional image inclined with respect to the optical axis is moved by mirror scanning. A three-dimensional display method of a three-dimensional image by a volume scanning method has been proposed in which a three-dimensional image is formed by displaying a cross-sectional image of a display object on a two-dimensional display in synchronization with movement of the two-dimensional image.

  However, in the above method, since a convex lens or a concave mirror is used as the imaging optical system, distortion occurs in the image due to aberrations of the convex lens or the concave mirror, and the shape of the display object cannot be accurately reproduced. To solve this problem, a method has been proposed in which a real mirror image forming optical system such as an optical element having many dihedral corner reflectors composed of two mirror surfaces is used as the image forming optical system.

  FIG. 1 is a perspective view schematically showing a three-dimensional aerial image display device of Patent Document 1. FIG. 2 is a plan view schematically showing a dihedral corner reflector array as a real mirror image forming optical system of Patent Document 1 and a partially enlarged view of a portion A. For example, Patent Document 1 discloses a real mirror image forming optical system capable of forming a real image of a projection object at a plane symmetric position with respect to a certain geometric plane serving as a symmetry plane, as shown in FIGS. Are arranged side by side, and objects to be projected are arranged respectively corresponding to the respective imaging optical systems, and the right-eye image forming optical system, which is the real mirror image forming optical system arranged relatively on the left side, A real mirror image of the corresponding projection object formed by the optical system and a corresponding mirror image formed by the left-eye imaging optical system, which is the real mirror image forming optical system disposed relatively on the right side. There has been disclosed a three-dimensional aerial image display device for displaying a real mirror image of a projection object superimposed and displayed at the same position.

  FIG. 3 is a perspective view schematically showing a volume scanning type three-dimensional aerial image display device of Patent Document 2. FIG. 4 is a diagram schematically showing an image forming mode by a two-sided corner reflector array which is a real mirror image forming optical system of Patent Document 2. Patent Document 2 below discloses a real mirror image capable of forming a real image of a projection object as a mirror image at a plane symmetric position with respect to a certain geometric plane serving as a symmetry plane, as shown in FIGS. A display including an imaging optical system, a display surface disposed on the lower surface side of the plane of symmetry, and displaying an image as the projection target; and a motion including a component in a direction perpendicular to the display surface. Driving means for operating the display means, and changing the image displayed on the display surface in synchronization with the operation of the display by the driving means, thereby converting the image to a space on the upper surface side of the symmetry plane. Discloses a volume scanning type three-dimensional aerial image display device that forms an image as a three-dimensional image.

JP 2012-163702 A JP 2013-080227 A

  As described above, the two-dimensional display is combined with the real mirror image forming optical system 91 as shown in FIG. 2 or FIG. By moving the two-dimensional display as described above, a floating image in the air can be displayed. However, in the configuration of Patent Document 1, as shown in FIG. 1, a two-dimensional display for displaying the projection object 92 according to the number of viewpoints is required. In the configuration of Patent Document 2, FIG. As shown in the figure, a driving unit 95 for moving the two-dimensional display 94 is required, which causes a problem that the size of the apparatus is increased.

  To solve this problem, a two-dimensional display in which optical means such as a lenticular lens is arranged (a so-called naked-eye type 3D display) is combined with a spatial imaging element such as the above-described real-mirror video imaging optical system to form an aerial image. A method of displaying a floating image can be considered. However, in the case of this configuration, a new 3D normal viewing region (a region where the depth of the 3D object is correctly displayed) and a 3D reverse viewing region (a region where the depth of the 3D object is reversed) appear alternately in the airborne image. Problems arise.

  SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and a main object of the present invention is to provide a 3D normal viewing region and a 3D reverse viewing region in an airborne image in a configuration combining an autostereoscopic 3D display and a spatial imaging element. It is an object of the present invention to provide a stereoscopic display device and a parallax image correction method that can avoid the problem of the appearance of a parallax image.

One aspect of the present invention is an autostereoscopic device that projects different images to the left and right eyes of an observer arranged in a first direction based on a first input image and a second input image for two viewpoints. a display, light from an object, a plurality have a light reflecting element for reflecting the first and second reflective surfaces which are orthogonal to each other, the autostereoscopic displays the first input image projected by and the A flat spatial imaging element that forms a floating image corresponding to the second input image in the air, and the first input image and the second input generated in at least a part of the floating image that is formed in the air The corresponding area of the first input image corresponding to the reverse viewing area where the depth parallax and the pop-out parallax of the image are reversed, and the corresponding area of the second input image corresponding to the reverse viewing area are replaced. The first input image including a corresponding area of the second input image; And an image processing section for outputting said second input image containing the corresponding region of the the force image first input image to the autostereoscopic display, the spatial imaging element is projected from the autostereoscopic display In addition , a floating image corresponding to the first input image including the corresponding area of the second input image and the second input image including the corresponding area of the first input image is aerial on the viewer side . imaged on the side the autostereoscopic display is a main lobe and stereoscopic area outside of the first direction of the main lobe is a stereoscopic viewing zone in front of the normal neighboring passing through the center of the display surface characterized in that it closed the robe.

One aspect of the present invention is an autostereoscopic device that projects different images to the left and right eyes of an observer arranged in a first direction based on a first input image and a second input image for two viewpoints. a display, light from an object, a plurality have a light reflecting element for reflecting the first and second reflective surfaces which are orthogonal to each other, the autostereoscopic displays the first input image projected by and the and a plate-shaped space image forming element for forming an floating image corresponding to the second input image in the air, the autostereoscopic display is a stereoscopic viewing zone in front of the normal neighboring passing through the center of the display surface main lobe and the and a side lobe is outside of the stereoscopic area of the first direction of the main lobe, resulting in a part of the floating image formed in the air, the first input image and the second be reversed and the parallax jumping out the depth parallax input image of the Determining whether a reverse vision area exists, when said reverse vision area is present, the corresponding region of the first input image corresponding to the previous Kigyaku vision zone, the second corresponding to the inverse viewing area interchanging the corresponding region of the input image, and the first input image containing the corresponding region of the second input image and the second input image containing the corresponding region of the first input image the Output to an autostereoscopic display , wherein the spatial imaging element is projected by the autostereoscopic display, the first input image including a corresponding area of the second input image, and the correspondence of the first input image A process of forming a floating image corresponding to the second input image including a region in the air on the observer side is performed.

  According to the stereoscopic display device and the parallax image correction method of the present invention, in a configuration in which a 3D display and a spatial imaging element are combined, a problem that a 3D normal viewing region and a 3D reverse viewing region appear in an airborne image is avoided. be able to.

  The reason is that an image processing unit that processes the input image is provided, and the image processing unit analyzes the input image to extract a parallax image region in which reverse vision occurs, and extracts the left-eye image and the right-eye image of the extracted parallax image region. This is because control for inputting the image to the 3D display by exchanging the image for use is performed.

It is a perspective view which shows typically the three-dimensional aerial image display apparatus of patent document 1. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a plan view schematically showing a dihedral corner reflector array as a real mirror image forming optical system of Patent Document 1, and a partially enlarged view of a portion A. It is a perspective view which shows typically the volume scanning type three-dimensional aerial image display apparatus of patent document 2. FIG. 9 is a diagram schematically showing an image forming mode by a two-sided corner reflector array which is a real mirror image forming optical system of Patent Document 2. FIG. 11 is a perspective view illustrating a configuration of a conventional stereoscopic display device. FIG. 11 is a diagram illustrating a relationship between an input image and a viewer's view image in a conventional stereoscopic display device. FIG. 9 is a diagram for explaining reverse viewing in a conventional stereoscopic display device. FIG. 2 is a perspective view illustrating a configuration of a stereoscopic display device according to the present embodiment. FIG. 2 is a cross-sectional view illustrating a configuration of a 3D display included in the stereoscopic display device according to the embodiment. FIG. 3 is a top view illustrating a configuration of a 3D display included in the stereoscopic display device according to the embodiment. It is a perspective view showing an example of arrangement of a 3D display and photography means. FIG. 4 is an optical path diagram for explaining a stereoscopic viewing area formed when a lenticular lens is used as an optical unit of a 3D display. This is an example of a case where the photographing unit is arranged on the center line of the 3D display and the distance D between the photographing unit and the lenticular lens is changed. FIG. 14 is a diagram illustrating a correspondence relationship between a distance between a lenticular lens and a photographing unit and a photographed image in the configuration in FIG. 13. This is an example of a case where the photographing means is shifted to the right (to the right eye) with respect to the center line of the 3D display, and the distance D between the photographing means and the lenticular lens is changed. FIG. 16 is a diagram illustrating a correspondence relationship between a distance between a lenticular lens and a photographing unit and a photographed image in the configuration of FIG. 15. FIG. 16 is a diagram illustrating a configuration of a captured image in FIG. 15. This is an example of a case in which the photographing unit is shifted to the left (left eye side) with respect to the center line of the 3D display, and the distance D between the photographing unit and the lenticular lens is changed. FIG. 19 is a diagram illustrating a correspondence relationship between a distance between a lenticular lens and a photographing unit and a photographed image in the configuration in FIG. 18. FIG. 3 is a diagram illustrating an image visually recognized by a 3D display. It is a schematic diagram explaining pop-out parallax in a 3D display. FIG. 4 is a diagram for explaining pseudoscopy in a stereoscopic display device. FIG. 2 is a block diagram illustrating a configuration example of an image processing unit in the stereoscopic display device according to the first embodiment of the present invention. FIG. 3 is a flowchart illustrating a parallax image correction method in the stereoscopic display device according to the first embodiment of the present invention. FIG. 3 is a schematic diagram of an example illustrating a parallax image correction method in the stereoscopic display device according to the first embodiment of the present invention. FIG. 7 is a schematic diagram of another example illustrating a parallax image correction method in the stereoscopic display device according to the first embodiment of the present invention. FIG. 2 is a block diagram illustrating a configuration example of an image processing unit according to the embodiment. It is a flowchart figure which shows the processing method of the three-dimensional image of a present Example. FIG. 7 is a schematic diagram illustrating a parallax image correction method in a stereoscopic display device according to a second embodiment of the present invention. FIG. 7 is a schematic diagram illustrating a parallax image correction method in a stereoscopic display device according to a second embodiment of the present invention. FIG. 9 is a schematic diagram of a configuration of a 3D display according to a third embodiment of the present invention. FIG. 14 is an explanatory diagram of functions of a 3D display according to a third embodiment of the present invention. FIG. 14 is an explanatory diagram of functions of a 3D display according to a third embodiment of the present invention. 14 is a specific example showing a configuration of a sub-pixel according to a third embodiment of the present invention. FIG. 14 is an explanatory diagram of functions of a 3D display according to a third embodiment of the present invention. FIG. 32 is a diagram illustrating a correspondence relationship between a distance between a lenticular lens and a photographing unit and a photographed image in the configuration of FIG. 31. FIG. 4 is a diagram for explaining pseudoscopy in a stereoscopic display device. FIG. 9 is a schematic diagram illustrating a parallax image correction method in a stereoscopic display device according to a third embodiment of the present invention. FIG. 2 is a perspective view illustrating a configuration of a stereoscopic display device according to the present embodiment. It is a figure explaining the value of Px according to observer's movement. It is a figure which shows the visual recognition image observed according to a movement of an observer, and a pseudoscopic area. FIG. 14 is a block diagram illustrating a configuration example of an image processing unit and a viewpoint position detection unit in a stereoscopic display device according to a fourth embodiment of the present invention. FIG. 14 is a flowchart illustrating a parallax image correction method in a stereoscopic display device according to a fourth embodiment of the present invention. FIG. 14 is a flowchart illustrating a parallax image correction method in a stereoscopic display device according to a fourth embodiment of the present invention.

  As shown in the background art, a pixel for displaying the image for the left eye and the image for the right eye is provided on the two-dimensional display, and the left and right eyes of the observer are placed on the left and right eyes of the observer by optical means such as a lenticular lens or a parallax barrier. A 3D display 20 that displays a stereoscopic image by sorting the image and the image for the right eye has been developed. Furthermore, in order to display an image as if the object is floating in the air, a 3D aerial image display device that combines a 2D display and a spatial imaging device such as a real mirror image imaging optical system has also been developed. Have been.

  However, since the conventional three-dimensional aerial image display device forms a floating image in the air by arranging the two-dimensional display for viewpoints or moving the two-dimensional display, the two-dimensional display corresponding to the number of viewpoints is performed. There is a problem that a display is required, a driving means for moving the two-dimensional display is required, and the apparatus becomes large.

  To solve this problem, as shown in FIG. 5, a method of forming a floating image in the air by combining the 3D display 20 and a spatial imaging element such as the above-described real mirror image imaging optical system can be considered. As a result of the study by the present inventors, it has been found that this method causes a new problem that a 3D normal viewing region and a 3D reverse viewing region alternately appear in the airborne image.

  That is, in the case of this configuration, the distance between the 3D display 20 and the spatial imaging element needs to be extremely small so that the light emitted from the 3D display 20 enters the spatial imaging element. Are projected, and the visual recognition image projected to each of the left and right eyes with respect to the input image shown in FIG. 6 is a repetition of a left-eye image and a right-eye image. In addition, basically, if one reflection optical system is interposed, an inverted image whose depth direction is inverted is generated. As a result, as shown in FIG. 6, the image (main lobe image) projected on the central region is switched between left and right, and as shown in FIG. And appear alternately.

  Therefore, in an embodiment of the present invention, an image input to the 3D display 20 is subjected to image processing for exchanging a left-eye image and a right-eye image in a parallax image region where pseudoscopy occurs. Specifically, the angle between the line of sight of the observer and the normal direction of the spatial imaging element, the distance between the spatial imaging element and the observer, the distance between the 3D display 20 and the spatial imaging element, or the spatial imaging element Based on the child's optical characteristics and the like, a reverse viewing area in which the depth parallax and the pop-out parallax are reversed is extracted, and an image corresponding to the reverse viewing area in the left-eye image and the right-eye image is replaced.

  Thus, it is possible to avoid the problem that the 3D normal viewing region and the 3D reverse viewing region appear alternately when the three-dimensional display and the spatial imaging element are combined.

  In order to describe the above-described embodiment of the present invention in more detail, a stereoscopic display device and a parallax image correcting method according to a first embodiment of the present invention will be described with reference to FIGS. FIG. 8 is a perspective view illustrating the configuration of the stereoscopic display device according to the present embodiment. FIGS. 11 to 20 are diagrams for explaining images visually recognized by the 3D display 20. FIG.

  As shown in FIG. 8, the stereoscopic display device 10 according to the present embodiment includes a 3D display 20, a spatial imaging device 30, an image signal input unit 40, and an image processing unit 50. Although not shown, a unit (for example, a camera) for detecting the distance between the spatial imaging element 30 and the observer is arranged at a predetermined position on the stereoscopic display device 10 as necessary.

  The 3D display 20 is an autostereoscopic display that allows a stereoscopic image to be visually recognized with the naked eye, and includes a display panel such as a liquid crystal display device and optical means such as a lenticular lens or a parallax barrier (in this embodiment, a lenticular lens). It is composed of The display panel has a configuration in which left-eye pixels for displaying a left-eye image and right-eye pixels for displaying a right-eye image are alternately arranged in a first direction. In addition, the lenticular lens has a configuration in which a surface facing the display panel is a flat surface and the other surface is a semi-cylindrical cylindrical lens arranged in the first direction. The details of the 3D display 20 will be described later.

  The spatial imaging device 30 is a device that forms a three-dimensional image displayed on the 3D display 20 in space to form a floating image in the air. The spatial imaging element 30 is, for example, an optical element composed of two perpendicular mirror surfaces (reflecting on the first reflecting surface, and further arranged in a pair with the first reflecting surface, at a step difference; A plurality of light reflecting elements (light reflecting elements) that are reflected by a second reflecting surface intersecting with the surface and pass therethrough are arranged. As the spatial imaging element 30, for example, a real mirror image imaging optical system disclosed in Patent Documents 1 and 2 can be used. The structure having the two vertical mirror surfaces is, for example, a pillar having a square cross section having a height of about 100 μm is formed on a flat plate surface, and two orthogonal surfaces of the side surfaces are used as mirrors, It can be formed by digging a square hole in the plane of a flat plate having a thickness of 2 mm and making the two inner wall surfaces mirror surfaces.

  In FIG. 8, the spatial imaging element 30 is configured such that an optical element including two perpendicular mirror surfaces is arranged on a plane perpendicular to the two mirror surfaces (that is, the column or the hole is formed by a main surface of a flat plate). Is formed so as to be perpendicular to the surface of the 3D display 20 because the plane is formed at a predetermined angle with respect to the display surface of the 3D display 20. When formed so as to be inclined with respect to the normal line of the main surface, the spatial imaging element 30 can be arranged so that the plane thereof is parallel to the display surface of the 3D display 20.

  The image signal input unit 40 is a device that outputs a left-eye image and a right-eye image to be displayed on each pixel of the display panel to the image processing unit 50. The image for the left eye and the image for the right eye may be captured images captured by the camera from two viewpoints (images for two viewpoints), or a captured image captured by the camera from one viewpoint and each pixel of this image. A set of depth images representing depth information of the corresponding 3D object may be used. In that case, a virtual viewpoint image obtained by photographing from a virtual camera position virtually arranged in a three-dimensional space is generated based on the photographed image and the depth image, and the photographed image and the virtual viewpoint image are processed by the image processing unit. 50 may be output.

  The image processing unit 50 analyzes an image (a set of a captured image for two viewpoints or a set of a captured image for one viewpoint and a virtual viewpoint image) output from the image signal input unit 40, and generates an area (a parallax image area) where reverse vision occurs. ) Is extracted, the input image in the parallax image region is horizontally inverted (the left-eye image and the right-eye image are exchanged), and the image data after the image processing is output to the 3D display 20. The image processing unit 50 may be a device independent of the 3D display 20 or may be incorporated in the 3D display 20. The details of the image processing unit 50 will also be described later.

  First, the configuration and operation of the 3D display 20 will be described to facilitate understanding of the stereoscopic display device 10 according to the present embodiment. In this specification, an XYZ orthogonal coordinate system is set as follows for convenience. The X-axis direction is a direction in which a left-eye pixel 24L and a right-eye pixel 24R described later are repeatedly arranged. The + X direction is a direction from the right-eye pixel 24R to the left-eye pixel 24L. The Y-axis direction is the longitudinal direction of a cylindrical lens 29a described later. Further, the Z-axis direction is a direction orthogonal to both the X-axis direction and the Y-axis direction. The + Z direction is a direction toward the lenticular lens 29 (a direction toward the observer) from the surface on which the left-eye pixels 24L or the right-eye pixels 24R are arranged.

  FIG. 9 is a cross-sectional view illustrating the configuration of the 3D display 20 included in the stereoscopic display device according to the present embodiment. FIG. 10 is a top view illustrating the configuration of the 3D display 20 included in the stereoscopic display device according to the present embodiment. As shown in FIGS. 9 and 10, the 3D display 20 includes a display panel 21 using liquid crystal molecules as electro-optical elements, and a lenticular lens 29 as optical means.

  The display panel 21 is an active matrix type display panel having a thin film transistor (TFT: Thin Film Transistor). A TFT substrate 24 on which a TFT is formed and a counter substrate 26 face each other with a small gap therebetween. The liquid crystal layer 25 is arranged in the gap. On the TFT substrate 24, a pixel pair as a display unit including one left-eye pixel 24L and one right-eye pixel 24R is provided in a matrix. The liquid crystal layer 25 is configured to be, for example, a transmission type TN (Twisted Nematic) mode. The TFT substrate 24 is disposed on the −Z direction side of the display panel 21, and the counter substrate 26 is disposed on the + Z direction side. Further, a first optical film 23 is attached to the -Z direction side of the TFT substrate 24, and a second optical film 27 is attached to the + Z direction side of the counter substrate 26. A lenticular lens 29 is fixed to the + Z direction side of the second optical film 27 via an adhesive layer 28, and the backlight 22 is arranged on the −Z direction side of the first optical film 23.

  A plurality of gate lines G (G1 to G5 in the figure) extending in the row direction, that is, the X-axis direction are arranged on the inner surface of the TFT substrate 24, that is, the surface on the + Z direction side. Further, on the same surface of the TFT substrate 24, a plurality of data lines D (D1 to D7 in the figure) extending in the column direction, that is, the Y-axis direction are arranged. Then, a pixel (a left-eye pixel 24L or a right-eye pixel 24R) is formed in a region surrounded by the gate line G and the data line D, and the pixel is formed by a TFT arranged near the intersection of the gate line G and the data line D. Is driven. In FIG. 10, in order to clarify the connection relationship between each pixel and the gate line G and the data line D, for example, a pixel connected to the gate line G3 and the data line D2 is denoted by P32.

  The configuration of the TFT substrate is not limited to the arrangement in which the gate line G extends in the X-axis direction and the data line D extends in the Y-axis direction as shown in FIG. 10, but the data line D extends in the X-axis direction. The gate lines G may be arranged to extend in the Y-axis direction. Further, the display panel 21 can use various display elements other than the liquid crystal display element, for example, an organic EL (Electro Luminescence) element, a quantum dot element, or a field emission element. Further, the driving method of the display panel 21 is not limited to the active matrix method using a TFT or the like, but may be a passive matrix method.

  The lenticular lens 29 is a lens array in which a number of cylindrical lenses 29a are arranged one-dimensionally. The cylindrical lens 29a is a one-dimensional lens in which a convex portion having a semicylindrical shape extends in one direction. In the arrangement direction of the cylindrical lens 29a, a pixel pair including a left-eye pixel 24L and a right-eye pixel 24R is repeatedly arranged. The direction is set in the X axis direction, and one cylindrical lens 29a is configured to correspond to one pixel pair.

  The cylindrical lens 29a has a lens effect only in a direction orthogonal to the extending direction. The direction having the lens effect coincides with the direction in which the left-eye pixels 24L and the right-eye pixels 24R are repeatedly arranged. As a result, the cylindrical lens 29a functions as a light beam separating unit that can separate the light emitted from the left-eye pixel 24L and the light emitted from the right-eye pixel 24R in different directions. Thereby, the lenticular lens 29 can separate the image displayed by the left-eye pixel 24L of each display unit from the image displayed by the right-eye pixel 24R of each display unit in different directions. The focal length of the cylindrical lens 29a is set to the distance between the principal point of the cylindrical lens 29a, that is, the vertex of the lens, and the pixel surface, that is, the surface on which the left-eye pixel 24L or the right-eye pixel 24R is arranged. ing.

  The lenticular lens 29 may have a configuration in which a lens surface is disposed on a surface facing the display panel 21. The optical means is not limited to the lenticular lens 29, and various optical elements capable of separating light, such as a fly-eye lens, a parallax barrier, or a prism sheet, can be used. Further, as the optical means, for example, a GRIN (Gradient Index) lens using liquid crystal, a liquid crystal lens combining a concavo-convex substrate having a lens effect and liquid crystal molecules, a switching parallax barrier using liquid crystal, or the like can be used.

  Next, the state of the light emitted from the 3D display 20 having the above-described configuration and incident on the observer's eye will be described using an image captured by the image capturing means. FIG. 11 is a perspective view showing an example of the arrangement of the 3D display 20 and the photographing means 80. FIG. 11 also shows the position 60 of the left eye and the position 61 of the right eye of the observer. Here, for ease of explanation, it is assumed that each of the left-eye pixels 24L and the right-eye pixels 24R has a strip shape extending in the longitudinal direction of the cylindrical lens 29a and is arranged in the X-axis direction.

  As shown in FIG. 11, the photographing means 80 is arranged at a position where the display surface of the 3D display 20 is photographed. As the photographing unit 80, an image processing lens system, a general video camera, a digital camera, or the like is used. The imaging unit 80 is fixed at a position in the + Z direction with respect to the 3D display 20 that is focused on the vicinity of the display surface of the display panel 21. It is desirable that the photographing center 81 of the photographing means 80 coincides with the center 20a of the 3D display 20, and is ideally located between the left-eye pixel 24L and the right-eye pixel 24R near the center 20a.

  FIG. 12 is an optical path diagram for explaining a stereoscopic viewing area formed when the lenticular lens 29 is used as the optical unit of the 3D display 20.

  The display panel 21 has left-eye pixels 24L (L1 to L3), (C1 to C3) and (R1 to R3) and right-eye pixels 24R (L1 to L3) and (C1 to C2) in the X-axis direction. And (R1 to R3) are sequentially arranged. The cylindrical lens 29L corresponds to the left-eye pixel 24L (L1 to L3) and the right-eye pixel 24R (L1 to L3), and the cylindrical lens 29C corresponds to the left-eye pixel 24L (C1 to C3). The cylindrical lens 29R corresponds to the pixel 24R (R1 to R3) for the left eye and the pixel 24R (R1 to R3) for the right eye.

  1L1, 2L1 and 3L1 shown in FIG. 12 indicate optical paths of light emitted from the left-eye pixels 24L (L1 to L3) and refracted by the cylindrical lens 29L, and 1L2, 2L2 and 3L2 are right-eye pixels 24R. The optical path of light emitted from (L1 to L3) and refracted by the cylindrical lens 29L is shown. Further, 1C1, 2C1 and 3C1 show the optical paths of the light emitted from the left-eye pixels 24L (C1 to C3) and refracted by the cylindrical lens 29C, and 1C2 and 2C2 are the right-eye pixels 24R (C1 to C2). ) Shows the optical path of light emitted from the cylindrical lens 29C and refracted by the cylindrical lens 29C. Similarly, 1R1, 2R1 and 3R1 indicate the optical paths of light emitted from the left-eye pixels 24L (R1 to R3) and refracted by the cylindrical lens 29R, and 1R2, 2R2 and 3R2 are the right-eye pixels 24R ( R1 to R3) and the optical path of light refracted by the cylindrical lens 29R.

  Actual light traveling on the optical path has a width of a predetermined angle clockwise or counterclockwise with respect to these optical paths. A left-eye image area 62 is formed in an area including a point where 1L1, 1C1 and 1R1 intersect, and a right-eye image area 63 is formed in an area including a point where 1L2, 1C2 and 1R2 intersect. The left-eye image area 62 and the right-eye image area 63 form a stereoscopic viewing range in which stereoscopic viewing is possible. The left-eye image area 62 has a left-eye position 60, and the right-eye image area 63 has a right-eye position. When there is 61, the observer can see the stereoscopic image correctly.

  The distance between the position where the stereoscopic viewing range is maximum (that is, the distance in the X-axis direction between the left-eye image area 62 and the right-eye image area 63 is maximum) and the position of the lenticular lens 29 is determined as the optimal stereoscopic viewing distance ( Dop), and the distance between the Y-axis direction of the left and right eyes, the position where the left-eye image area 62 and the right-eye image area 63 intersect, and the position of the lenticular lens 29 are the maximum stereoscopic viewing distance (Dmax) and the minimum The stereoscopic viewing distance (Dmin) is used.

  Here, focusing on the cylindrical lens 29L, light contributing to the formation of the left-eye image region 62 and the right-eye image region 63 is emitted from the left-eye pixel 24L (L1) and the right-eye pixel 24R (L1). Light (1L1, 1L2). These are defined as primary light. The light is emitted from the left-eye pixel 24L (L2) or the right-eye pixel 24R (L2), which is a pixel adjacent to the left-eye pixel 24L (L1) or the right-eye pixel 24R (L1), and refracted by the cylindrical lens 29L. The light (2L1, 2L2) obtained is defined as secondary light. Similarly, the light is emitted from the left-eye pixel 24L (L3) or the right-eye pixel 24R (L3), which is the second adjacent pixel to the left-eye pixel 24L (L1) or the right-eye pixel 24R (L1), and is cylindrical. Light (3L1, 3L2) refracted by the lens 29L is defined as tertiary light. Similarly, with respect to the light related to the cylindrical lens 29C or 29R, the primary light also contributes to the formation of the left-eye image area 62 and the right-eye image area 63.

  As can be seen from the optical path diagram of FIG. 12, when the distance between the observer and the lenticular lens 29 is shorter than the minimum stereoscopic viewing distance Dmin, the secondary light or the tertiary light, such as secondary light or tertiary light, emitted from the left and right sides of the display panel 21 is reduced. It can be seen that the effects become apparent.

  Next, a photographed image obtained when the distance D between the photographing means 80 and the lenticular lens 29 is changed will be described. In FIGS. 13, 15, and 18, only the optical path of the primary light is shown for easy understanding of the optical path.

  FIG. 13 shows an example in which the photographing unit 80 is arranged on the center line of the 3D display 20 and the distance D between the photographing unit 80 and the lenticular lens 29 is changed. FIG. 14 is a diagram showing the correspondence between the distance between the lenticular lens and the photographing means and the photographed image in the configuration of FIG. In this case, if the interval D is near the optimum stereoscopic viewing distance Dop, as shown in FIGS. 14A and 14B, the image captured by the image capturing unit 80 is an input image of the left-eye pixel 24L on the left and a right-eye image on the right. The input image is the pixel 24R. On the other hand, when the interval D is gradually reduced (for example, when the interval is about 1/3 of the optimal stereoscopic viewing distance Dop), as shown in FIG. The input image of the right-eye pixel 24R appears, and the input image of the left-eye pixel 24L appears on the right side. Further, when the distance D is reduced (for example, when the distance is about 1/4 of the optimum stereoscopic viewing distance Dop), as shown in FIGS. 14D and 14E, the left-eye pixel is located on the left side of the captured image under the influence of the tertiary light. An input image of 24L appears, and an input image of the right-eye pixel 24R appears on the right side. That is, as the distance D between the photographing means 80 and the lenticular lens 29 becomes smaller than the optimal stereoscopic viewing distance Dop, the photographed image is affected by higher-order light such as secondary light or tertiary light, and the photographed image is input to the left-eye pixel 24L. This is a repetition of the image and the input image of the right-eye pixel 24R.

  FIG. 15 shows an example in which the photographing unit 80 is shifted to the right (to the right eye) with respect to the center line of the 3D display 20 and the distance D between the photographing unit 80 and the lenticular lens 29 is changed. FIG. 16 is a diagram showing the correspondence between the distance between the lenticular lens and the photographing means 80 and the photographed image in the configuration of FIG. In this case, if the interval D is in the vicinity of the optimal stereoscopic viewing distance Dop, as shown in FIG. 16A, the image captured by the image capturing means 80 is only the input image of the right-eye pixel 24R. Further, when the interval D becomes small and becomes about 1/2 of the optimal stereoscopic viewing distance Dop, as shown in FIG. 16B, the input image of the left-eye pixel 24L appears on both sides of the captured image. Further, when the interval D becomes smaller and becomes about 1/3 to 1/4 of the optimal stereoscopic viewing distance Dop, as shown in FIGS. 16C and D, the right eye is placed on both sides of the captured image under the influence of the secondary light. The input image of the use pixel 24R appears. Further, when the interval D becomes smaller and becomes 1/4 or less of the optimum stereoscopic viewing distance Dop, as shown in FIG. 16E, the input image of the left-eye pixel 24L is located on the left side of the captured image under the influence of the tertiary light. Appear.

  Such captured images include primary light constituting a main lobe, which is a front stereoscopic viewing area including a normal passing through the center of the display surface of the 3D display 20, and an outer side in the X-axis direction with respect to the main lobe. This is because high-order lights constituting other lobes (side lobes) which are the stereoscopic viewing area are mixed to form a captured image. FIG. 17 is a diagram illustrating the configuration of the captured image in FIG. For example, as shown in FIG. 17, regardless of the distance D between the photographing means 80 and the cylindrical lens 29, the input image of the right-eye pixel 24R is photographed by the primary light constituting the main lobe, but the distance D is small. As much as possible, many input images of the left-eye pixel 24L are captured under the influence of higher-order light constituting another lobe. As a result, when the interval D is the optimum stereoscopic viewing distance Dop (for example, 600 mm), the captured image is only the input image of the right-eye pixel 24R by the primary light constituting the main lobe, but the interval D is 0.5 × Dop. In the case of (for example, 300 mm), the captured image combines the input image of the right-eye pixel 24R by the primary light constituting the main lobe and the input image of the left-eye pixels 24L on both sides by the high-order light constituting the other lobe. The captured image has a configuration in which the input image of the right-eye pixel 24R is arranged in the center, and the input image of the left-eye pixel 24L is arranged on both sides thereof. Further, when the interval D is 0.33 × Dop (for example, 200 mm) or 0.28 × Dop (for example, 170 mm), the input image of the left-eye pixel 24 </ b> L due to the higher-order light constituting another lobe is shifted to the center. The image has a configuration in which the input image of the right-eye pixel 24R is arranged in the center, the input image of the left-eye pixel 24L is arranged on both outer sides thereof, and the input image of the right-eye pixel 24R is arranged on both outer sides thereof. Becomes When the interval D is 0.23 × Dop (for example, 140 mm), the input image of the left-eye pixel 24L due to the higher-order light constituting another lobe is further shifted toward the center, and further outwardly, the input image of the left-eye pixel 24L. Appears, the captured image has a configuration in which the input image of the left-eye pixel 24L and the input image of the right-eye pixel 24R are repeatedly arranged three times.

  FIG. 18 shows an example in which the photographing unit 80 is shifted to the left (to the left eye) with respect to the center line of the 3D display 20, and the distance D between the photographing unit 80 and the lenticular lens is changed. FIG. 19 is a diagram showing the correspondence between the distance between the lenticular lens and the photographing means 80 and the photographed image in the configuration of FIG. In this case, if the interval D is in the vicinity of the optimum stereoscopic viewing distance Dop, as shown in FIG. 19A, the image captured by the image capturing means 80 is only the input image of the left-eye pixel 24L. When the interval D becomes small and becomes about half of the optimal stereoscopic viewing distance Dop, as shown in FIG. 19B, the input image of the right-eye pixel 24R appears on both sides of the captured image. Further, when the interval D becomes smaller and becomes about 1/3 to 1/4 of the optimal stereoscopic viewing distance Dop, as shown in FIGS. 19C and D, the left eye is placed on both sides of the captured image under the influence of the secondary light. The input image of the use pixel 24L appears. Further, when the interval D is reduced and becomes equal to or less than 1/4 of the optimum stereoscopic viewing distance Dop, as shown in FIG. 19E, the input image of the right-eye pixel 24R is located on the left side of the captured image under the influence of the tertiary light. Appear.

  FIG. 20 is a diagram illustrating the configuration of the captured image in FIG. In this case, as shown in FIG. 20, the input image of the left-eye pixel 24L is photographed by the primary light constituting the main lobe regardless of the distance D between the photographing means 80 and the cylindrical lens 29. As the size becomes smaller, more input images of the right-eye pixel 24R are taken under the influence of higher-order light constituting another lobe. As a result, the captured image has a configuration in which the input image of the left-eye pixel 24L and the input image of the right-eye pixel 24R in FIG. 17 are interchanged.

  In other words, when the distance D between the photographing unit 80 and the cylindrical lens 29 is reduced, the photographed image is affected by the higher-order light, and the photographed image has the input image of the left-eye pixel 24L and the input image of the right-eye pixel 24R repeatedly arranged. When the position of the photographing unit 80 is shifted from the center line of the 3D display 20, an image formed by primary light constituting the main lobe and an image formed by higher-order light constituting another lobe change in accordance with the shift.

  Next, a case where the spatial imaging element 30 is combined with the 3D display 20 will be described. In the following, the spatial imaging element 30 has a configuration in which a pillar or a hole serving as a dihedral corner reflector is formed so as to be inclined with respect to the normal to the main surface of the flat plate. It is assumed that they are arranged so as to be parallel to the display surface of the 3D display 20.

  FIG. 21 is a schematic diagram illustrating pop-out parallax on the 3D display 20. Here, as shown on the left side of FIG. 21B, the input image in which the star-shaped object 71 is arranged slightly right in the center is displayed on the left-eye pixel 24L of the display panel 21, and the right-eye pixel 24R is displayed on the right-eye pixel 24R. As shown on the right side of FIG. 21B, when the input image in which the star-shaped object 71 is arranged on the slightly left side of the center is displayed, as shown in FIG. 21A, the star-shaped object 71 protrudes into the air. It becomes the pop-out parallax that is visually recognized. On the other hand, for example, as shown in FIG. 22A, when the spatial imaging element 30 is arranged at a position of D = 0.5 × Dop, a virtual camera virtually arranged at the position of the spatial imaging element 30 The captured image is as shown in FIG. 22B from FIG. 17 and FIG. Here, the spatial imaging element 30 is a reflection optical system, and the light emitted from the spatial imaging element 30 is in the same direction as the incident direction, so that the image viewed by the observer is switched left and right. As a result, the image viewed by the observer is as shown in FIG. 22C, the input image of the right-eye pixel 24R is arranged in the center of the image viewed by the left eye, and the left eye is positioned in the center of the image viewed by the right eye. The input image of the pixel for use 24L is arranged. That is, so-called reverse vision, in which the pop-out parallax becomes the depth parallax, occurs.

  As described above, when the 3D display 20 and the spatial imaging element 30 are combined, it has been found that pseudoscopy occurs when the 3D display 20 and the spatial imaging element 30 are combined. Therefore, in the stereoscopic display device 10 of the present embodiment, the image processing unit The left and right eyes of the region where reverse vision occurs (the depth disparity and the pop-out disparity are reversed) of the input image input from the image signal input unit 40 are provided. Image processing is performed, and the image data after the image processing is output to the 3D display 20.

  FIG. 23 is a block diagram illustrating a configuration example of the image processing unit 50 in the stereoscopic display device 10 according to the first embodiment of the present invention. As shown in FIG. 23, the image processing unit 50 includes a parallax image correcting unit 51 and a region parameter storage unit 56. The parallax image correcting unit 51 includes a parallax image region extracting unit 52, an image data replacing unit 53, It consists of.

  The parallax image area extraction unit 52 of the parallax image correction unit 51 acquires a set of a captured image for two viewpoints or a captured image for one viewpoint and a virtual viewpoint image from the image signal input unit 40, and configures the main lobe described above. Primary light angle (EVS (Eye Viewing Space) angle), the distance between the 3D display 20 and the spatial imaging element 30, the distance between the spatial imaging element 30 and the observer, or the aspect ratio of the spatial imaging element 30 (for example, Based on predetermined conditions, such as the ratio of the opening width and the opening height of the dihedral corner reflector, an area (parallax image area) where pseudoparallax occurs (depth parallax and pop-out parallax are reversed) is extracted.

  For example, when the EVS angle or the distance between the 3D display 20 and the spatial imaging element 30 or the distance between the spatial imaging element 30 and the observer decreases, the number of repetitive pixels of the left-eye image and the right-eye image increases. When the distance between the 3D display 20 and the spatial imaging element 30 or the distance between the spatial imaging element 30 and the observer is out of the predetermined range, the main lobe image does not appear at the center of the left and right eye visual recognition images. . In addition, even when the distance between the 3D display 20 and the spatial imaging element 30 and the distance between the spatial imaging element 30 and the observer are out of the predetermined range, the main lobe image does not appear at the center of the left and right eyes visually recognized images. Further, the position or the number of repetitive pixels of the left-eye image and the right-eye image changes depending on the aspect ratio of the spatial imaging element 30. Therefore, these conditions are stored in the area parameter storage unit 56, and when the input image is obtained from the image signal input unit 40, the conditions are read out from the area parameter storage unit 56, and at least one of the read out conditions is set as one of the read out conditions. Based on this, it is specified in which area the pseudoscopy occurs, and the specified area is extracted as a parallax image area. In other words, the position and size of the main lobe image are determined according to the above-described predetermined conditions, and it is possible to determine in which region the reverse sight occurs due to the position and size of the main lobe image. , The parallax image area is extracted.

  The EVS angle and the distance between the spatial imaging element 30 and the observer change according to the position of the observer. Therefore, a camera is installed on the three-dimensional display device 10, and the image processing unit 50 appropriately acquires an image of the observer from the camera, extracts feature points from the captured image, detects the positions of both eyes, The EVS angle and the distance between the spatial imaging element 30 and the observer may be specified and stored in the area parameter storage unit 56 on the basis of the position, the interval, and the like. Here, four examples of the predetermined conditions are the EVS angle, the distance between the 3D display 20 and the spatial imaging element 30, the distance between the spatial imaging element 30 and the observer, and the aspect ratio of the spatial imaging element 30. However, other conditions such as characteristics of the display panel 21 (for example, pixel size, pixel pitch or pixel array structure) or characteristics of the lenticular lens 29 (for example, shape, pitch, focal length, aberration or material of the cylindrical lens 29a) are set. May be used. Also, here, the aspect ratio of the spatial imaging element 30 has been exemplified as the predetermined condition regarding the characteristics of the spatial imaging element 30, but the mirror surface roughness of the two-sided corner reflector constituting the spatial imaging element 30, the mirror assembly Accuracy or reflectivity can also be used.

  The image data replacement unit 53 of the parallax image correction unit 51 specifies a left-eye image and a right-eye image corresponding to the parallax image region extracted by the parallax image region extraction unit 52 from the input image, and When the boundary positions of the image and the right-eye image substantially coincide with each other, image data in which the boundary positions are replaced is generated, and the generated image data is output to the 3D display 20.

  The area parameter storage unit 56 is a storage unit such as a memory, and stores the above-described predetermined condition that is referred to when the parallax image area extraction unit 52 extracts a parallax image area.

  Note that the parallax image area extraction unit 52 and the image data replacement unit 53 may be configured as hardware, or the image processing unit 50 may include a CPU (Central Processing Unit) and a ROM (Read Only Memory) or a RAM (Random Access Memory). A control unit including a memory such as a memory is provided, and the CPU expands the program stored in the ROM into the RAM and executes the program, so that the control unit functions as the parallax image area extraction unit 52 and the image data replacement unit 53. You may.

  FIG. 24 is a flowchart illustrating a parallax image correction method in the stereoscopic display device 10 according to the first embodiment of the present invention. FIG. 25 is a schematic view of an example illustrating a parallax image correction method in the stereoscopic display device 10 according to the first embodiment of the present invention. A parallax image correction method using the image processing unit 50 having the above configuration will be described with reference to a flowchart of FIG. 24 and a conceptual diagram of FIG.

  First, the parallax image area extraction unit 52 acquires an input image (a captured image for two viewpoints or a set of a captured image for one viewpoint and a virtual viewpoint image) from the image signal input unit 40 (Step S101). Here, as shown in FIG. 25A, it is assumed that a reverse sight has occurred in the main lobe image.

  Next, the parallax image region extraction unit 52 acquires a predetermined condition from the region parameter storage unit 56, and extracts a parallax image region based on the predetermined condition (Step S102). Here, the area of the main lobe image is extracted as the parallax image area.

Next, the image data exchange unit 53 exchanges the image for the left eye and the image for the right eye in the parallax image area (Step S103). For example, as shown in FIG. 25B, when the left eye image of the input image is composed of _L A, _{L B,} and _{L C,} the right-eye image is composed of _R A, _{R B} and _{R C,} left replacing the main lobe picture R _B of eye images, and a main lobe image L _B of the right-eye image.

  Then, the image data replacing unit 53 outputs the image data in which the image for the left eye and the image for the right eye in the parallax image area have been exchanged to the 3D display 20, and the 3D display 20 displays based on the image data. An image is displayed on panel 21 (step S104). Here, since the image for the left eye and the image for the right eye of the main lobe where pseudoscopy occurs are interchanged, as shown in FIG. 25C, the observer views the airborne image in which all the regions are 3D normal viewing regions. Can be visually recognized.

In the above description, the case where each of the left and right input images is composed of the three images of the left and right and the center is described. However, the same applies to the case where each of the left and right input images is composed of a plurality of regions. be able to. FIG. 26 is a schematic diagram of another example illustrating a parallax image correction method in the stereoscopic display device 10 according to the first embodiment of the present invention. For example, as shown in FIG. 26, when each of the left and right input images is composed of five images, as shown in FIG. 26A, the parallax image area extraction unit 52 determines the center image based on the above predetermined condition. The areas of the main lobe image and the side lobe images at both ends are extracted as parallax image areas, and as shown in FIG. 26B, the image data replacement unit 53 outputs the left image R _A , R _C and By exchanging R _E and L _A , L _C, and L _{E of the} image for the right eye and outputting them to the 3D display 20, as shown in FIG. The image can be viewed.

  As described above, the parallax image region in which reverse vision occurs is extracted from the input image, and the left-eye image and the right-eye image corresponding to the parallax image region are exchanged, so that the 3D normal viewing region and the 3D inverse The problem that the viewing region and the viewing region appear alternately can be avoided.

  Note that the stereoscopic display device 10 of the present embodiment can be applied to both cases of displaying a 3D object in monochrome and displaying in color. When displaying a 3D object in color, it can be realized by using a color filter (CF) substrate as the opposing substrate 26 forming the display panel 21. In this case, a structure in which a pair of pixels arranged in a direction (X-axis direction) having a lens effect of the cylindrical lens 29a have the same color, and a color is periodically changed in the longitudinal direction (Y-axis direction, that is, for each row) of the cylindrical lens 29a. Or a structure in which the X-axis direction and the Y-axis direction are reversed. Instead of providing the CF, for example, the backlight 22 is configured by a light source capable of emitting a single color of R (Red) / G (Green) / B (Blue), and the backlight 22 is set in accordance with the emission time of R / G / B. By displaying a desired pixel at a predetermined gradation (by so-called time-division driving), color display can also be realized.

  Next, a stereoscopic display device and a parallax image correction method according to a second embodiment of the present invention will be described with reference to FIGS. FIG. 27 is a block diagram illustrating a configuration example of an image processing unit according to the present embodiment. FIG. 28 is a flowchart illustrating a method of processing a stereoscopic image according to the present embodiment. FIG. 29 is a schematic diagram illustrating a parallax image correction method in the stereoscopic display device 10 according to the second embodiment of the present invention. FIG. 30 is a schematic diagram illustrating a parallax image correction method in the stereoscopic display device 10 according to the second embodiment of the present invention.

  In the first embodiment described above, the image for the left eye and the image for the right eye in the parallax image area are exchanged. However, when the depth (or pop-out) state differs between the image and the exchanged image, the exchange is performed. When the depth (or pop-out) changes unnaturally at the seam between the image and the image adjacent to the image, or when the depth (or pop-out) of the main lobe image is small, the stereoscopic effect cannot be sufficiently expressed. There is. Further, depending on the characteristics of the spatial imaging element, a region where the images are mixed (3D crosstalk region) near the boundary between the images becomes large, and when there is parallax, a double image may be visually recognized. Therefore, in this embodiment, the depth of the input image is used to adjust the amount of parallax of the input image (the amount of shift between each pixel of the left-eye image and the right-eye image) so that the airborne image is appropriately displayed. To be done.

  In this case, the configuration of the stereoscopic display device 10 is the same as that of FIGS. 8 to 10 of the first embodiment, but the parallax image correcting unit 51 of the image processing unit 50 outputs the parallax image area extraction as shown in FIG. In addition to the section 52 and the image data replacing section 53, a depth information extracting section 54 and a parallax amount adjusting section 55 are provided.

  When the image signal input unit 40 generates the virtual viewpoint image from the captured image and the depth image for one viewpoint, the depth information extracting unit 54 determines the captured image for one viewpoint and the virtual viewpoint image from the image signal input unit 40. In addition, a depth image is acquired, and depth information of each part of the 3D object is extracted based on the depth image. When acquiring captured images for two viewpoints from the image signal input unit 40, the depth information of each part of the 3D object is extracted by comparing the captured images for the two viewpoints. Note that the depth image represents a distance between an object corresponding to each pixel of a captured image captured from a certain viewpoint position in the 3D space and the viewpoint position.

  The parallax adjustment unit 55 adjusts the parallax of the input image (a captured image for one viewpoint and a virtual viewpoint image or a captured image for two viewpoints) based on the depth information extracted by the depth information extracting unit 54. At this time, predetermined conditions such as the EVS angle, the distance between the 3D display 20 and the spatial imaging element 30, the distance between the spatial imaging element 30 and the observer, and the aspect ratio of the spatial imaging element 30 are stored in the area parameter storage unit 56. Is read, and how to adjust the amount of parallax is determined with reference to the predetermined condition. For example, when the image for the left eye and the image for the right eye are exchanged, the depth or the projection at the right and left of the image may change unnaturally at the joint between the exchanged image and the image adjacent to the image. In such a case, the amount of parallax at both ends of the image (near the boundary between adjacent images) is made smaller than the amount of parallax other than near the boundary, and the connection of the images is smoothed. In addition, when the depth or the degree of protrusion of the main lobe image is small, the three-dimensional effect of the 3D object may not be sufficiently obtained. In such a case, the parallax amount of the main lobe image (the image of the portion corresponding to the reverse viewing area) is made larger than the parallax amounts other than the main lobe image to emphasize the stereoscopic effect of the 3D object. In addition, the spatial imaging element may increase the size of the 3D crosstalk area in the vicinity of the boundary of the image, so that a double image may be visually recognized. In such a case, similarly to the above, the amount of parallax at both ends of the image (near the boundary with the adjacent image) is reduced or set to zero so that the double image is hardly viewed.

  The parallax image region extraction unit 52 acquires an image (a captured image for one viewpoint and a virtual viewpoint image or a captured image for two viewpoints) in which the amount of parallax has been adjusted from the parallax amount adjustment unit 55, and , An EVS angle, a predetermined condition such as a distance between the 3D display 20 and the spatial imaging device 30, a distance between the spatial imaging device 30 and the observer, or an aspect ratio of the spatial imaging device 30, and the like. Based on at least one condition, an area (parallax image area) where pseudoscopy occurs is extracted.

  Note that, as in the first embodiment, since the EVS angle and the distance between the spatial imaging element 30 and the observer change with the position of the observer, a camera is installed on the stereoscopic display device 10 and image processing is performed. The unit 50 appropriately obtains an image of the observer from the camera, extracts feature points from the captured image, detects the positions of both eyes, and based on the positions and intervals of the eyes, the EVS angle and spatial connection. The distance between the image element 30 and the observer may be specified and stored in the area parameter storage unit 56. In addition, other conditions such as the characteristics of the display panel 21 or the characteristics of the lenticular lens 29 can be used as the predetermined conditions. In addition, as the predetermined condition regarding the characteristics of the spatial imaging element 30, the mirror surface roughness, mirror assembly accuracy, reflectance, and the like of the dihedral corner reflector constituting the spatial imaging element 30 can be used.

  The image data replacement unit 53 specifies a left-eye image and a right-eye image corresponding to the parallax image region extracted by the parallax image region extraction unit 52 from the input image, and outputs the left-eye image and the right-eye image. When the boundary positions substantially coincide with each other, image data in which the boundary positions are replaced is generated, and the generated image data is output to the 3D display 20.

  The depth information extracting unit 54, the parallax amount adjusting unit 55, the parallax image area extracting unit 52, and the image data replacing unit 53 may be configured as hardware, or the image processing unit 50 may include a CPU and a memory such as a ROM or a RAM. Is provided, and the CPU develops the program stored in the ROM into the RAM and executes the program. The control unit includes a depth information extracting unit 54, a parallax amount adjusting unit 55, a parallax image area extracting unit 52, and an image processing unit. You may make it function as the data replacement part 53. Further, similarly to the above-described first embodiment, the display panel 21 of the 3D display 20 of the present embodiment can use the opposite substrate 26 on which a color filter is formed or emit R / G / B monochromatic light. For example, a color display can be performed by using a suitable backlight 22.

  A parallax image correction method using the image processing unit 50 having the above configuration will be described with reference to the flowchart of FIG. 28 and the conceptual diagrams of FIGS.

  First, the depth information extraction unit 54 acquires an input image (a set of a captured image for one viewpoint, a virtual viewpoint image, and a depth image or a captured image for two viewpoints) from the image signal input unit 40 (Step S201). The depth information extracting unit 54 extracts the depth information of each part of the 3D object based on the result of processing the depth image or the captured image for two viewpoints (Step S202).

  Next, based on the depth information extracted by the depth information extraction unit 54 and the predetermined condition acquired from the area parameter storage unit 56, the parallax amount adjustment unit 55 generates an input image (a captured image signal for one viewpoint and a virtual viewpoint The parallax amount of the image signal or the captured image for two viewpoints) is adjusted (step S203). For example, when it is desired to improve the connection between the replaced image and an image adjacent to the replaced image, as shown in FIG. 29B, adjustment is performed so that the amount of parallax near the boundary of the image is smaller than the amount of parallax other than the boundary. When it is desired to sufficiently express the depth or pop-out of the 3D object, as shown in FIG. 29C, the main lobe image is adjusted so that the amount of parallax becomes larger than the amount of parallax other than the main lobe image. If it is desired to improve the problem that the double image is visually recognized, the adjustment is performed so that the amount of parallax near the boundary of the image is reduced. For example, when the 3D crosstalk is small, as shown in FIG. 30B, the adjustment is performed so that the amount of parallax near the boundary of the image is smaller than the amount of parallax other than the boundary. If the 3D crosstalk is significant, as shown in FIG. 30C, the adjustment is performed so that the parallax amount at the boundary portion of the image becomes zero.

  Next, the parallax image region extraction unit 52 acquires the captured image for one viewpoint and the virtual viewpoint image adjusted by the parallax amount adjustment unit 55. In addition, the parallax image area extraction unit 52 acquires a predetermined condition from the area parameter storage unit 56. Then, the parallax image region extracting unit 52 extracts a parallax image region based on the extracted data (step S204). Here, the area of the main lobe image is extracted as the parallax image area.

Next, the image data exchange unit 53 exchanges the left-eye image and the right-eye image in the parallax image area (Step S205). For example, as shown in FIG. 29A, FIG. 30A, left-eye image of the input image is composed of _L A, _{L B,} and _{L C,} the right-eye image is composed of _R A, _{R B} and _{R C} If, replacing the main lobe picture R _B of the left-eye image, and a main lobe image L _B of the right-eye image.

  Then, the image data replacing unit 53 outputs the image data in which the image for the left eye and the image for the right eye in the parallax image area have been exchanged to the 3D display 20, and the 3D display 20 displays based on the image data. An image is displayed on panel 21 (step S206).

  In the above description, the case where each of the left and right input images is composed of the three images of the left and right and the center is described. However, the same applies to the case where each of the left and right input images is composed of a plurality of regions. be able to.

  In this way, after adjusting the amount of parallax based on the depth image, by swapping the image for the left eye and the image for the right eye corresponding to the parallax image area, the airborne image can be displayed appropriately, The use value of the display device 10 can be increased.

  A stereoscopic display device and a parallax image correction method according to a third embodiment of the present invention will be described with reference to FIGS.

  In the first embodiment, as shown in FIG. 10, in the cylindrical lens 29a constituting the lenticular lens 29, the extending direction and the direction orthogonal to the extending direction and having the lens effect are Y in which the pixels are arranged. It is arranged so as to be parallel to the direction and the X direction, respectively. However, the present invention can also use the 3D display 20 in which the arrangement direction of the cylindrical lenses is rotated with respect to the arrangement of the pixels. The details will be described below.

  FIG. 31 is a schematic diagram of the configuration of the 3D display 20 according to the third embodiment of the present invention. FIG. 32 is an explanatory diagram of functions of the 3D display 20 according to the third embodiment of the present invention. FIG. 33 is an explanatory diagram of functions of the 3D display 20 according to the third embodiment of the present invention. FIGS. 31A and 31B show the 3D display 20 used in this embodiment. As shown in FIG. 31, there is an angle α between the extending direction of the arranged cylindrical lenses 29a and the Y-axis direction in which the pixels 124 of the display panel 21 are arranged.

  The display panel 21 is configured by arranging a plurality of pixels 124 in the X direction and the Y direction as shown in FIGS. 31A and 31B. The display panel 21 provides a viewer with a stereoscopic display by causing each of the pixels 124 to function as the left-eye pixel 24L and the right-eye pixel 24R according to the arrangement of the cylindrical lens 29a constituting the lenticular lens 29. Can be. For example, as shown in FIG. 32 in the 3D display 20 of FIG. 31A, and as shown in FIG. 33 in the 3D display 20 of FIG. 31B, the pixel 124 for the left eye is adjusted according to the light beam separation characteristic of the cylindrical lens 29a to be arranged. It functions as the pixel 24L and the right pixel 24R. Note that the pixel 125 shown in FIGS. 32 and 33 is a pixel in which light rays emitted from the pixel are divided by the cylindrical lens 29a in both directions on the right eye side and the left eye side of the observer. These pixels 125 may function as the left-eye pixel 24L and the right-eye pixel 24R so that the total number in the display panel is equal, or the adjacent left-eye pixel 24L and right-eye pixel 24R may be used. The pixel may be made to function as a pixel for displaying intermediate luminance, or may be made a pixel not to display (black display). Although a single pixel is illustrated in FIGS. 31 to 33, for convenience of description, a display panel in which the pixel 124 includes a plurality of sub-pixels can be used for color display. is there.

  FIG. 34 is a specific example showing the configuration of the sub-pixel according to the third embodiment of the present invention. 34A and 34B show a specific example in which the pixel 124 is composed of a plurality of sub-pixels. FIG. 34A is an example in which the pixel 124 is divided into three in the X direction, and sub-pixels 126, 127, and 128 are arranged. FIG. 34B is a diagram in which the pixel 124 is divided into three in the Y direction. This is an example in which sub-pixels 127 and 128 are arranged. In FIG. 34, a plurality of gate lines G (Gy, Gy + 1...) Extending in the X-axis direction and a plurality of data lines D (Dx, Dx + 1...) Extending in the Y-axis direction are arranged. A pixel is formed in a region surrounded by a data line D and a sub-pixel is driven by a TFT disposed near an intersection of the gate line G and the data line D. D may extend and the gate line G may extend in the Y-axis direction. Further, although the pixel 124 is constituted by three sub-pixels, the pixel 124 may be constituted by a plurality of sub-pixels.

  As described above, when the pixel 124 is composed of a plurality of sub-pixels, the pixel 125 shown in FIGS. 32 and 33 is provided in sub-pixel units in accordance with the light beam separation characteristics of the cylindrical lens 29a to be arranged. What is necessary is just to function as the pixel 24L for left eyes or the pixel 24R for right eyes. Hereinafter, optical characteristics when the cylindrical lens 29a is rotated with respect to the pixel array will be described. For convenience of description, FIG. 35 in which the pixels 125 are omitted from FIG. 32 will be used.

  By making the pixels 124 of the 3D display 20 in FIG. 31A function as the left-eye pixels 24L and the right-eye pixels 24R as shown in FIG. 35, stereoscopic display can be provided to the viewer. For example, since the XZ plane cross section of BB ′ in FIG. 35 can be described with the same cross-sectional view as FIG. 9, the light is emitted from the left-eye pixel 24L or the right-eye pixel 24R on BB ′ and refracted by the cylindrical lens. The optical path of the light going to the observer can be explained with reference to FIG. Further, a cross section of AA ′ or CC ′ that is different in position in the Y direction from BB ′ can also be described with reference to FIG. 9, but the left-eye pixel 24L or the right-eye pixel 24R is arranged according to the rotation angle α. Therefore, the position of the left-eye pixel 24L or the right-eye pixel 24R is shifted in the −X direction in AA ′ and in the + X direction in CC ′ as compared with BB ′. Therefore, the optical path shown in FIG. 5 is shifted according to the position in the Y direction. Therefore, the rotation angle α affects the visually recognized image.

  The effect of the rotation angle α on the visually recognized image will be described using an image captured using the image capturing unit as in the first embodiment.

  FIG. 36 is a diagram showing the correspondence between the distance between the lenticular lens 29 and the photographing means and the photographed image in the configuration of FIG. That is, when the 3D display 20 and the photographing unit 80 of the present embodiment are arranged in the same manner as in FIG. As shown in FIG. 36, as shown in FIG. 13 described in the first embodiment, the photographing unit 80 is arranged on the center line of the 3D display 20, and the distance D between the photographing unit 80 and the lenticular lens 29 is changed. This is an example, and is a captured image corresponding to FIG. 13 of the first embodiment. In this case, the captured image in which the interval D in the present embodiment is in the vicinity of the optimum stereoscopic viewing distance Dop is the input image of the left-eye pixel 24L on the left side and the right side on the right side, as in Embodiment 1, as shown in FIGS. Although the input image is the right-eye pixel 24R, the center boundary line 129 is a captured image inclined by β from the Y axis according to the rotation angle α. The inclination angle β of the boundary line with respect to the Y axis is ideally equal to the rotation angle α. However, if the position when the lenticular lens 29 is mounted deviates from the ideal, the deviation from the rotation angle α depends on the mounting position deviation. Occurs.

  When the interval D gradually decreases, as in the first embodiment, as shown in FIG. 36C, the input image of the right-eye pixel 24R appears on the left side of the captured image under the influence of the secondary light, and the left side on the right side. The input image of the eye pixel 24L appears. Further, when the interval D is reduced, as shown in FIGS. 36D and 36E, the input image of the left-eye pixel 24L appears on the left side of the captured image under the influence of the tertiary light, and the right-eye pixel 24R on the right side. The input image appears. That is, similarly to FIG. 14 of the first embodiment, as the distance D between the photographing unit 80 and the lenticular lens 29 is smaller than the optimal stereoscopic viewing distance Dop, the distance D is affected by higher-order light such as secondary light or tertiary light, The captured image is a repetition of the input image of the left-eye pixel 24L and the input image of the right-eye pixel 24R.

  In addition, when the photographing unit 80 described in the first embodiment is displaced to the right (to the right eye) with respect to the center line of the 3D display 20, and the distance D between the photographing unit 80 and the lenticular lens 29 is changed ( (FIG. 15), and the case where the photographing unit 80 is shifted to the left (left eye side) with respect to the center line of the 3D display 20, and the distance D between the photographing unit 80 and the lenticular lens 29 is changed (FIG. 18). ) Is the same as described above except that the boundary line 129 between the input image of the left-eye pixel 24L and the input image of the right-eye pixel 24R on the right has an inclination angle β from the Y axis. Description is omitted because there is.

  Next, a case where the 3D display 20 of the present embodiment illustrated in FIG. 31A and the spatial imaging element 30 are combined will be described. In the following, the spatial imaging element 30 has a configuration in which a pillar or a hole serving as a dihedral corner reflector is formed so as to be inclined with respect to the normal to the main surface of the flat plate. It is assumed that they are arranged so as to be parallel to the display surface of the 3D display 20.

  As in the first embodiment, as shown on the left side of FIG. 21B, an input image in which the star-shaped object 71 is arranged slightly to the right of the center is displayed on the left-eye pixel 24L, and the right-eye pixel 24R is displayed on the left-eye pixel 24R. As shown on the right side, when an input image in which the star-shaped object 71 is arranged slightly to the left in the center is displayed, the parallax is a pop-out parallax in which the star-shaped object 71 is visually recognized as popping out in the air.

  On the other hand, for example, as shown in FIG. 37A (similar to FIG. 22 of the first embodiment), when the spatial imaging element 30 is arranged at a position of D = 0.5 × Dop, the spatial imaging element 30 An image captured by a virtual camera virtually arranged at the position is as shown in FIG. 37B. Here, the spatial imaging element 30 is a reflection optical system, and the light emitted from the spatial imaging element 30 is in the same direction as the incident direction, so that the image viewed by the observer is switched left and right. As a result, the image viewed by the observer is as shown in FIG. 37C, and as described in the first embodiment, so-called reverse vision, in which the pop-out parallax becomes the depth parallax, occurs.

  Thus, even when the 3D display 20 of the present embodiment and the spatial imaging element 30 are combined, pseudoscopy occurs as in the first embodiment.

  The configuration of the stereoscopic display device 10 of the present embodiment is the same as that of the first embodiment except that the 3D display 20 is different. That is, the 3D display 20 shown in FIG. 31 is arranged in FIG. Therefore, the image processing unit 50 is provided in the same manner as in the first embodiment, the image processing is performed to horizontally reverse the area where the reverse vision of the input image input from the image signal input unit 40 occurs, and the image data after the image processing is displayed on the 3D display. 20.

  As the configuration of the image processing unit 50, the same configuration as that of the first embodiment can be applied, detailed description is omitted. Regarding the parallax image correcting method of the present embodiment, a schematic diagram for explaining the parallax image correcting method in the stereoscopic display device 10 according to the third embodiment of the present invention shown in FIGS. Will be explained.

  First, the parallax image area extraction unit 52 acquires an input image (a captured image for two viewpoints or a set of a captured image for one viewpoint and a virtual viewpoint image) from the image signal input unit 40 (Step S101). Here, as shown in FIG. 38A, it is assumed that the main lobe image has a pseudo-vision. At this time, the boundary line between the 3D normal viewing region and the 3D reverse viewing region is different from the first embodiment, and becomes the tilt angle γ according to the tilt angle β shown in FIG. The inclination angle γ is ideally equal to the inclination angle β. However, depending on the characteristics of the spatial imaging element (mirror surface roughness of the two-sided corner reflector, mirror assembly accuracy, reflectance, etc.), the display image of the stereoscopic display device may be changed. And the size of the floating image in the air, there is a case where a difference from the inclination angle β occurs.

  Next, the parallax image region extraction unit 52 acquires a predetermined condition from the region parameter storage unit 56, and extracts a parallax image region based on the predetermined condition (Step S102). Here, the area of the main lobe image is extracted as the parallax image area. The rotation angle α or the inclination angle β can be used as the parameter of the boundary line used in the region extraction. However, the 3D display 20 and the spatial imaging element 30 actually used in the stereoscopic display device are combined, and the It is preferable to apply the actually measured inclination angle γ.

Next, the image data exchange unit 53 exchanges the image for the left eye and the image for the right eye in the parallax image area (Step S103). For example, as shown in FIG. 38B, when the left eye image of the input image is composed of _L A, _{L B,} and _{L C,} the right-eye image is composed of _R A, _{R B} and _{R C,} left The main lobe image RB of the image for the eye and the main lobe image LB of the image for the right eye are exchanged.

  Then, the image data replacing unit 53 outputs the image data in which the image for the left eye and the image for the right eye in the parallax image area have been exchanged to the 3D display 20, and the 3D display 20 displays based on the image data. An image is displayed on panel 21 (step S104). In this case, since the image for the left eye and the image for the right eye of the main lobe where pseudoscopy occurs are exchanged, as shown in FIG. 38C, the observer views the airborne image in which all the regions are 3D normal viewing regions. Can be visually recognized.

  Note that, in the above description, the case where each of the left and right input images is composed of three images of the left and right and the center has been described. However, as described with reference to FIG. The same can be applied to a case where the region is constituted by regions.

  Further, the image processing unit of the second embodiment can be applied to the stereoscopic display device of the present embodiment, and the parallax image correction method described in the second embodiment can be applied.

  Further, similarly to the first and second embodiments, since the EVS angle and the distance between the spatial imaging element 30 and the observer change according to the position of the observer, a camera is installed on the stereoscopic display device 10. The image processing unit 50 appropriately obtains an image of the observer from the camera, extracts feature points from the captured image to detect the positions of both eyes, and calculates the EVS angle based on the positions and intervals of both eyes. The distance between the spatial imaging element 30 and the observer may be specified and stored in the area parameter storage unit 56. In addition, other conditions such as the characteristics of the display panel 21 or the characteristics of the lenticular lens 29 can be used as the predetermined conditions. In addition, as the predetermined condition regarding the characteristics of the spatial imaging element 30, the mirror surface roughness, mirror assembly accuracy, reflectance, or the like of the dihedral corner reflector constituting the spatial imaging element 30 can be used.

  As described above, the parallax image region in which reverse vision occurs is extracted from the input image, and the left-eye image and the right-eye image corresponding to the parallax image region are exchanged, so that the 3D normal viewing region and the 3D inverse The problem that the viewing region and the viewing region appear alternately can be avoided.

  The display panel 21 used in this embodiment can use various display elements such as a liquid crystal display element, an organic EL (Electro Luminescence) element, a quantum dot element, and a field emission element. Further, the driving method of the display panel 21 is not limited to the active matrix method using a TFT or the like, but may be a passive matrix method.

  A stereoscopic display apparatus and a parallax image correction method according to a fourth embodiment of the present invention will be described with reference to FIGS.

  FIG. 39 is a perspective view illustrating the configuration of the stereoscopic display device 130 according to the present embodiment. FIG. 40 is a diagram illustrating the value of Px according to the movement of the observer. FIG. 41 is a diagram illustrating a visual recognition image observed in accordance with the movement of the observer and a pseudoscopic region. FIG. 42 is a block diagram illustrating a configuration example of an image processing unit and a viewpoint position detection unit in a stereoscopic display device according to a fourth embodiment of the present invention.

  As shown in FIG. 39, the three-dimensional display device 130 of the present embodiment includes the 3D display 20, the spatial imaging device 30, the image signal input unit 140, the image processing unit 150, and the positions of the spatial imaging device 30 and the observer. It is composed of a viewpoint position detecting section 160 which is a means for detecting the relationship.

  As the 3D display 20, those described in the first and third embodiments can be used, and detailed description is omitted. Hereinafter, for convenience of explanation, an example in which the 3D display 20 of the first embodiment is used in the present embodiment will be described.

  FIG. 39 shows a normal line from the airborne image, and the distance on the normal line at which the observer can suitably view the airborne image in a stereoscopic manner is denoted by Lp. Further, the midpoint between the left eye position 60 and the right eye position 61 of the observer on an axis parallel to the X axis is Px. Next, the value of Px when the observer moves while maintaining the distance Lp from the airborne image to the airborne image shown in FIG. 39 and the observer from directly above will be described with reference to FIG. As shown in FIG. 40B, the value of Px when the observer is at the center position with respect to the airborne image is set to 0, and the value of Px when the observer moves to the left as shown in FIG. 40A is set to negative. 40C, the value of Px when the observer moves right is assumed to be positive. Using this Px, the observation position VPx is defined by the following equation.

  VPx = Px / Lp Expression (1)

  Next, a visual recognition image when the observer moves on an axis parallel to the X axis will be described. FIG. 41 is a view showing a visual image observed by the observer in accordance with the parallel movement with respect to the X axis shown in FIG. 39 when different input images are input to the right-eye pixel and the left-eye pixel of the 3D display 20, and the reverse vision It is a figure showing an area. Here, the value of VPx calculated by Expression (1) is used as the value of the observation position.

  For example, when VPx = 0, the image visually recognized by the left eye is the same as that described with reference to FIG. 25B in the first embodiment, and the right-eye image of the main lobe where pseudoscopy occurs in the center is visually recognized. The left-eye images of the side lobes are visually recognized at the left and right ends. Further, when VPx = 0, the right-eye image of the right eye is visually recognized in the center, the left-eye image of the main lobe in which pseudoscopy occurs, and the right-eye image of the side lobe is visually recognized on both left and right ends. You.

  When the observer moves rightward, the main lobe image in which pseudopsis occurs also moves rightward, and the side lobe images at both ends also move rightward. At this time, as shown in FIG. 17 and FIG. 20 of the first embodiment, the image of the main lobe (the image by the primary light) and the image of the side lobe (the image by the high-order light) appear repeatedly in the X direction. The main lobe image appears again to the left of the left side lobe image with respect to the image. Specifically, at VPx = + 0.022, the left-eye visual images are, in order from the left end, the right-eye image of the main lobe, the left-eye image of the side lobe, the right-eye image of the main lobe, and the side lobe image. The left-eye image and the right-eye visual image are, in order from the left end, the left image of the main lobe, the right-eye image of the side lobe, the left-eye image of the main lobe, and the right-eye image of the side lobe. . Therefore, when VPx = + 0.022, the pseudoscopic region appears at the left end in addition to the region where the pseudoscopic region of VPx = 0 has moved to the right. That is, when the observer changes the observation position, the visual recognition image changes as shown in FIG. 41, so that the pseudoscopic region also changes.

  As described above, since the position at which pseudoscopy occurs varies according to the position of the observer, the stereoscopic display device 130 according to the present embodiment further includes the viewpoint position detection unit 160, so that the three-dimensional display device 130 responds to the position of the observer. The left-right image and the right-eye image of the area where the reverse vision is generated are obtained by reversing the input image input from the image signal input unit 40. Is performed), and the image data after the image processing is output to the 3D display 20.

  As shown in FIG. 42, the viewpoint position detection unit 160 includes an observer imaging unit 161 and a binocular position detection unit 162. The image processing unit 150 includes a parallax image correction unit 51, an area parameter storage unit 164, and a relative position calculation unit 163, as shown in FIG.

  The observer photographing unit 161 is a unit for photographing an image of the observer in order to measure three-dimensional coordinates from the viewpoint position detecting unit 160 to both eyes of the observer, and includes a visible light camera, a visible light camera, and an infrared camera. Or multiple cameras can be used.

  The binocular position detector 162 calculates the three-dimensional coordinates of the observer's eyes based on the image data acquired by the observer imager 161. The calculation method differs depending on the observer imaging unit 161. For example, when only one visible light camera is used, the binocular position is calculated from the feature points of the face in the captured image, and the distance is calculated based on the size of the face. In the case of a combination of a visible light camera and an infrared camera, the positions of both eyes are calculated from the feature points of the face in the image captured by the visible light camera, and the distance is calculated from the captured image of the infrared light emitted with the pattern, or the light is projected. The distance is calculated by a method of measuring the time when the infrared rays return (TOF method: Time of Flight). When a plurality of cameras are used, the positions of both eyes are calculated from the feature points of the face in the captured image, and the distance is calculated by trigonometry.

  As described above, the measured values from the viewpoint position detecting unit 160 to the observer's both eyes are input to the relative position calculating unit 163. The relative position calculator 163 uses the acquired measurement values and design parameters such as the installation position of the viewpoint position detector 160 of the stereoscopic display device 130 to determine the positional relationship between the airborne image and the observer (the distance shown in FIG. 39). Lp and the binocular center position Px) are calculated, and the calculation result is output to the area parameter storage unit. The outputted positional relationship between the airborne image and the observer is stored in the area parameter storage unit 164.

  The parallax image correction unit 51 of the present embodiment is the same as that of the first embodiment, and includes a parallax image area extraction unit 52 and an image data replacement unit 53.

  The parallax image area extraction unit 52 of the parallax image correction unit 51 acquires a set of a captured image for two viewpoints or a captured image for one viewpoint and a virtual viewpoint image from the image signal input unit 40, and Predetermined conditions such as the positional relationship between the airborne image and the observer (the distance Lp and the binocular center position Px shown in FIG. 37), the distance between the 3D display 20 and the spatial imaging element 30, the aspect ratio of the spatial imaging element 30, and the like. Is read, and an area (parallax image area) in which pseudopsis occurs is extracted based on at least one of predetermined conditions. For example, as shown in FIG. 41, a pseudoscopic region corresponding to the observation position is extracted.

  The EVS angle that changes with the position of the observer and the distance between the spatial imaging element 30 and the observer are calculated based on the floating image in the air calculated by the viewpoint position detector 160 and the relative position calculator 163 and the position of the observer. It can be calculated by the relation. The rate of change for these observer positions may be stored in the area parameter storage unit 164. In addition, other conditions such as the characteristics of the display panel 21 or the characteristics of the lenticular lens 29 can be used as the predetermined conditions. In addition, as the predetermined condition regarding the characteristics of the spatial imaging element 30, the mirror surface roughness, mirror assembly accuracy, reflectance, or the like of the dihedral corner reflector constituting the spatial imaging element 30 can be used.

  The image data replacement unit 53 specifies a left-eye image and a right-eye image corresponding to the parallax image region extracted by the parallax image region extraction unit 52 from the input image, and generates image data by exchanging them. The output image data is output to the 3D display 20. For example, it identifies the pseudo-view area image of the input image for the left eye and the pseudo-view area image of the input image for the right eye shown in FIG. 41, generates image data in which these are interchanged, and outputs the image data to the 3D display 20.

  Except for the configuration of the present embodiment described above, the configuration is the same as that of the first embodiment, and a detailed description thereof will be omitted.

  FIG. 43 is a flowchart illustrating a parallax image correction method in the stereoscopic display device 10 according to the fourth embodiment of the present invention. A method for correcting a parallax image using the viewpoint position detection unit 160 and the image processing unit 150 shown in FIGS. 39 and 42 will be described below with reference to the flowchart in FIG.

  First, the image processing unit 150 acquires an input image (a captured image for two viewpoints or a set of a captured image for one viewpoint and a virtual viewpoint image) from the image signal processing unit 40 (step S401). Next, in the viewpoint position detecting section 160, the binocular position detecting section 162 detects the observer's binocular position from the image photographed by the observer photographing section 161 (step S402). In the image processing unit 150, the relative position calculation unit 163 calculates the positional relationship between the airborne image and the observer (distance Lp, binocular center position Px) based on the binocular position detected by the viewpoint position detection unit 160. Is output to the area parameter storage unit 164 (step S403). If the observer is not in the photographing range and cannot calculate the positional relationship between the airborne image and the observer, the previously calculated positional relationship or the ideal positional relationship is output to the area parameter storage unit 164. I just need.

  Next, the parallax image region extracting unit 52 acquires predetermined conditions including the positional relationship of the observer from the region parameter storage unit 164, and extracts a parallax image region based on these (Step S404). Here, as the parallax image area, a pseudoscopic area that changes depending on the position of the observer is extracted as illustrated in FIG. 41. Next, with respect to the input image acquired in step S401, the image for the left eye and the image for the right eye in the extracted pseudoscopic region are exchanged (step S405). For example, the reverse view area image of the left-eye input image and the reverse view area image of the right-eye input image shown in FIG. 41 are exchanged.

  Then, the image data replacing unit 53 outputs the image data in which the image for the left eye and the image for the right eye in the parallax image area have been exchanged to the 3D display 20, and the 3D display 20 displays based on the image data. An image is displayed on panel 21 (step S406). Here, since the image for the left eye and the image for the right eye in which the pseudopsis are generated are exchanged, the observer can visually recognize the floating image in the air in which all the regions are 3D normal viewing regions.

  In the above description, using the example of FIG. 41, the case where each input image on the left and right is composed of three images on the left and right and at the center when VPx = 0 is shown, but as described in the first embodiment, The same can be applied to a case where the input image is composed of a plurality of regions.

  In this manner, the position of the observer is detected, a parallax image region where reverse vision occurs is extracted from the input image according to the position of the observer, and a left-eye image and a right-eye image corresponding to the parallax image region are extracted. , The problem that the 3D normal viewing region and the 3D reverse viewing region alternately appear in the airborne image can be avoided even when the observer moves.

  Note that the 3D display 20 according to the first embodiment has been described as the stereoscopic display device according to the present embodiment, but the 3D display 20 according to the third embodiment may be used. When the 3D display 20 according to the third embodiment is used, the boundary formed by the pseudoscopic region may be set to have an inclination that matches the characteristics of the 3D display 20.

  Further, the processing of detecting the position of the observer described in the present embodiment and extracting the parallax image region where reverse vision occurs from the input image according to the position of the observer can be applied to the second embodiment. FIG. 44 shows a flowchart in this case.

  Note that the present invention is not limited to the above embodiments, and the configuration or control thereof can be appropriately changed without departing from the spirit of the present invention.

  INDUSTRIAL APPLICABILITY The present invention is applicable to a stereoscopic display device that generates a floating image in the air by forming a 3D image in the air, a parallax image correction method and a parallax image correction program in the stereoscopic display device, and a recording medium that stores the parallax image correction program. It is.

Reference Signs List 10 stereoscopic display device 20 3D display 20a center 21 display panel 22 backlight 23 first optical film 24 TFT substrate 24R right-eye pixel 24L left-eye pixel 25 liquid crystal layer 26 counter substrate 27 second optical film 28 adhesive layer 29 optical means (Lenticular lens)
29a, 29L, 29C, 29R Cylindrical lens 30 Spatial imaging element 40 Image signal input unit 50 Image processing unit 51 Parallax image correction unit 52 Parallax image area extraction unit 53 Image data replacement unit 54 Depth information extraction unit 55 Parallax amount adjustment unit 56 Area parameter storage unit 60 Left eye position 61 Right eye position 62 Left eye image area 63 Right eye image area 71 Object 80 Imaging means 81 Imaging center 90 3D aerial image display device 91 Real mirror image forming optical system ( Two-sided corner reflector array)
91a Two-sided corner reflector 92 Projected object 93 Volume scanning type three-dimensional aerial image display device 94 Two-dimensional display 95 Driving means 124, 125 pixels 126, 127, 128 Sub-pixel 129 Boundary line 130 Stereoscopic display device 140 Image signal input unit 150 Image processing section 160 View point position detecting section 161 Observer photographing section 162 Binocular position detecting section 163 Relative position calculating section 164 Area parameter storage section

Claims (18)

An autostereoscopic display that projects different images to the left and right eyes of the observer arranged in the first direction based on the first input image and the second input image for two viewpoints;
Light from an object, a plurality have a light reflecting element for reflecting the first and second reflective surfaces perpendicular to each other, the projected by the autostereoscopic display first input image and a second input A flat spatial imaging element that forms a floating image corresponding to the image in the air ,
The first input image corresponding to a reverse viewing area in which the depth parallax and the pop-out parallax of the first input image and the second input image are generated in at least a part of the floating image formed in the air. And the corresponding area of the second input image corresponding to the pseudoscopic area is replaced with the first input image and the first input image including the corresponding area of the second input image. And an image processing unit that outputs the second input image including the corresponding region to the autostereoscopic display ,
The space focusing elements, the autostereoscopic display is projected from the said first input image containing the corresponding region of the second input image, the second containing the corresponding region of the first input image A floating image corresponding to the input image is formed in the air on the observer side,
The autostereoscopic display is closed and the main lobe is a stereoscopic viewing zone in front of the normal neighboring passing through the center of the display surface, and side lobes outside the stereoscopic viewing area of the first direction of the main lobe A three-dimensional display device, characterized in that: A viewpoint position detection unit that detects a viewpoint position of the observer,
The image processing unit, when an image obtained by projecting one input image and an image obtained by projecting the other input image are mixed, according to the position of the observer detected by the viewpoint position detection unit, The stereoscopic display device according to claim 1, wherein a part corresponding to a viewing area is replaced and output to the autostereoscopic display. The viewpoint position detection unit,
An observer photographing unit for photographing the observer,
Including a binocular position detection unit that detects the binocular position from the captured image,
A relative position calculation unit that specifies a positional relationship between the spatial imaging element and the observer based on the positions of both eyes detected by the binocular position detection unit,
The image processing unit, in the case where an image obtained by projecting one input image and an image obtained by projecting the other input image are mixed, in accordance with the specified positional relationship, replaces a portion corresponding to the pseudoscopic region. The stereoscopic display device according to claim 2, wherein the stereoscopic display device outputs the image to the autostereoscopic display. The image processing unit,
A parallax image area extracting unit that extracts the pseudoscopic area based on a predetermined condition,
An image data replacement unit that replaces an image of a part corresponding to the pseudoscopic region,
The three-dimensional display device according to any one of claims 1 to 3, wherein: The stereoscopic display device according to claim 4, wherein a boundary line of the pseudoscopic region has an angle with respect to an arrangement direction of pixels included in the autostereoscopic display. The image processing unit,
A depth information extraction unit that extracts depth information of each part of the display target based on the input images for the two viewpoints;
The stereoscopic display device according to claim 4, further comprising: a parallax amount adjustment unit configured to adjust a parallax amount of the input images for the two viewpoints based on the predetermined condition. . The parallax amount adjustment unit, the parallax amount near the boundary between the image of the portion corresponding to the reverse viewing region and the image adjacent to the image is smaller than the parallax amount other than the vicinity of the boundary,
The three-dimensional display device according to claim 6, wherein: The parallax amount adjustment unit, the parallax amount of the image of the portion corresponding to the pseudoscopic region is larger than the parallax amount of the image other than the portion corresponding to the pseudoscopic region,
The three-dimensional display device according to claim 6, wherein: The predetermined condition includes an angle formed by a line of sight of the observer and a normal direction of the spatial imaging element, a distance between the autostereoscopic display and the spatial imaging element, the spatial imaging element and the observer. Including at least one of the distance or the optical properties of the spatial imaging element,
9. The three-dimensional display device according to claim 4, wherein: An autostereoscopic display that projects different images to the left and right eyes of the observer arranged in the first direction based on the first input image and the second input image for two viewpoints;
Light from an object, a plurality have a light reflecting element for reflecting the first and second reflective surfaces perpendicular to each other, the projected by the autostereoscopic display first input image and a second input and a plate-shaped space image forming element for forming an floating image corresponding to the image in the air,
The autostereoscopic display includes a main lobe is a three-dimensional viewing zone in front of the normal neighboring passing through the center of the display surface, and side lobes outside the stereoscopic viewing area of the first direction of the main lobe ,
It is determined whether or not there is a pseudoscopic region in which the depth parallax and the pop-out parallax of the first input image and the second input image, which are generated in a part of the floating image formed in the air, are reversed. If the region is present, by replacing the corresponding region of the first input image corresponding to the previous Kigyaku vision zone, and a corresponding region of the second input image corresponding to the inverse viewing region, the second Outputting the first input image including the corresponding region of the input image and the second input image including the corresponding region of the first input image to the autostereoscopic display ,
The first input image including a corresponding region of the second input image, the second image including a corresponding region of the first input image, wherein the spatial imaging element is projected by the autostereoscopic display; A parallax image correction method, comprising performing a process of forming a floating image corresponding to an input image in the air on the observer side . Detects the leading Symbol observer's viewpoint position,
In the case where an image obtained by projecting one input image is mixed with an image obtained by projecting the other input image, it is determined whether or not the reverse viewing region exists according to the detected viewpoint position of the observer. When the pseudoscopic region is present, the pseudoscopic region is specified in accordance with the detected viewpoint position of the observer,
The parallax image correction method according to claim 10, wherein a part corresponding to the specified pseudoscopic region is replaced and a process of outputting the part to the autostereoscopic display is performed. Photographed before Symbol observer,
Detects the position of both eyes from the captured image,
Identify the positional relationship between the spatial imaging element and the observer based on the detected positions of both eyes,
When an image obtained by projecting one input image is mixed with an image obtained by projecting the other input image, a portion corresponding to the pseudoscopic region is replaced according to the specified positional relationship, and the image is displayed on the naked-eye stereoscopic display. The method for correcting a parallax image according to claim 11, further comprising: outputting a parallax image. Based on Jo Tokoro conditions, it extracts the inverse viewing area,
The parallax image correction method according to any one of claims 10 to 12, wherein a process of exchanging an image of a portion corresponding to the pseudoscopic region in the input images for the two viewpoints is performed. . The parallax image correction method according to claim 13, wherein a boundary line of the pseudoscopic region has an angle with respect to an arrangement direction of pixels included in the autostereoscopic display. Before extracting pre Kigyaku vision area,
Based on the input images for the two viewpoints, depth information of each part of the display target is extracted,
The parallax image correction method according to claim 13, wherein a process of adjusting a parallax amount of the input images for the two viewpoints based on the predetermined condition is performed. 16. The processing according to claim 15, wherein a parallax amount near a boundary between an image of a portion corresponding to the pseudoscopic region and an image adjacent to the image is made smaller than a parallax amount other than the vicinity of the boundary. Parallax image correction method. The parallax image correction method according to claim 15, wherein a process of making a parallax amount of an image of a portion corresponding to the reverse viewing region larger than a parallax amount of an image other than the portion corresponding to the reverse viewing region is performed. . The predetermined condition includes an angle formed by a line of sight of the observer and a normal direction of the spatial imaging element, a distance between the autostereoscopic display and the spatial imaging element, the spatial imaging element and the observer. The parallax image correction method according to any one of claims 13 to 17, further comprising at least one of a distance from the image forming apparatus and an optical characteristic of the spatial imaging element.

Images