JP4825244B2 - Stereoscopic image display device and stereoscopic image display method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a device for displaying a stereoscopic vision image, etc. capable of automatically performing stereoscopic vision without depending on distance to an observation object on the basis of captured images from a pair of imaging parts whose convergence angle is fixed. <P>SOLUTION: The device includes: the imaging parts 12, 13 fixed at prescribed convergence angle at intervals 2d; a ranging part 16 which measures distance D (Pk) from a middle point S between the imaging parts to the observation object Pk; an image processing part which generates a pair of display images from a pair of obtained captured images; and a pair of display parts which display the pair of generated display images on each display picture, wherein the image processing part generates the pair of display images by performing keystone to the pair of captured images within a range corresponding to each display picture of the pair of display parts to be bilateral symmetric on the basis of the interval 2d, the distance D (Pk), the convergence angle or distance from an intersection P1 of light axes of the imaging parts 12, 13 to the middle point S. <P>COPYRIGHT: (C)2010,JPO&amp;INPIT

Description

  The present invention relates to a stereoscopic image display device including a pair of imaging units and a pair of display units with a fixed convergence angle, and a stereoscopic image display method applied to the stereoscopic image display device.

  An image obtained by imaging an observation object using a pair of imaging units (one is called a right-eye imaging unit and the other is a left-eye imaging unit) spaced apart by a predetermined interval, and is captured by the right-eye imaging unit There are various stereoscopic image display devices that display images on the right-eye display unit and images captured by the left-eye imaging unit on the left-eye display unit so that the images can be observed stereoscopically. As an example, one that is configured to have a shape similar to that of glasses (so-called stereoscopic glasses) can be given.

  Such a stereoscopic image display device is configured so that the imaging optical axis of the right-eye imaging unit and the imaging optical axis of the left-eye imaging unit intersect at a single point with a convergence angle, similar to when a human observes an observation object. However, in the case of a human eye, this convergence angle changes according to the distance to the observation object.

  This will be described with reference to FIGS. Here, FIG. 27 is a diagram showing a state of different convergence angles when observing a nearby observation object and a distant observation object, and FIG. 28 shows the degree of eyeball crossing when observing the observation object in the vicinity. FIG. 29 is a diagram showing a state in which the degree of crossing of the eyeball is reduced when observing a distant object.

  Consider a case where an observer observes from a nearby observation object to a distant observation object. When a human observes the observation object, the eyeball rotates so that the center of the pupil faces the gazing point of the observation object. Therefore, when the observation object is at a short distance, the degree of crossing is as shown in FIG. When the observation object is at a long distance, the degree of crossing is reduced as shown in FIG.

  This is a function called congestion adjustment that the human eye has. That is, the angle at which the line of sight intersects is called the convergence angle, but the human eye (shown as LEYE for the left eye and REYE for the right eye in FIGS. 27 to 29) is at a short distance, as shown in FIG. When observing an observation object (shown as NOBJ in FIG. 27), the convergence angle is increased, and when observing an observation object at a long distance (also shown as FOBJ), the convergence angle is increased. Make adjustments to reduce the.

  Therefore, when an image similar to that observed when a human is looking directly with his / her eyes is viewed, it is necessary to adjust the convergence angle according to the distance to the observation object, as shown in FIG. FIG. 30 is a diagram illustrating an example of a pair of left and right imaging units configured to adjust the convergence angle according to the distance to the observation object.

  The left-eye imaging unit 12 is provided to perform the function of the human left eye, and the right-eye imaging unit 13 is configured to perform the function of the human right eye. The left-eye imaging unit 12 and the right-eye imaging unit 13 are configured to be able to change the direction in which the optical axes LOA and ROA face in the horizontal plane, that is, can adjust the convergence angle. It has become.

  Then, a stereoscopic image display device capable of observing with the left eye the image obtained by photographing with the left-eye imaging unit 12, and observing with the right eye the image obtained by photographing with the right-eye imaging unit 13, For example, if the image is displayed on a stereoscopic image display device of the stereoscopic glasses type as shown in FIGS. 1 to 3 according to the embodiment of the present invention, stereoscopic viewing is possible, and the observer stereoscopically views the observation object. Can be observed.

  However, in order to provide a function for adjusting the convergence angle, for example, an imaging unit rotation drive mechanism for rotating the imaging unit for the right eye and the imaging unit for the left eye, or a mirror disposed on the front surface of the imaging unit. A configuration such as a mirror driving mechanism for rotating the mirror is required, which increases costs and increases the weight of the apparatus.

  On the other hand, a configuration in which the driving mechanism is not required by fixing to a predetermined convergence angle without adjusting the mechanical convergence angle is also conceivable.

  FIG. 4 which concerns on embodiment of this invention has shown the structural example which fixed such an imaging part. When the image pickup unit is fixed, the image pickup unit rotation drive mechanism and the like as described above are not necessary, so that there is an advantage that the cost is not required and the apparatus can be reduced in weight.

  However, in this case, there is a problem that when the observation object is at a distance other than the distance corresponding to the fixed convergence angle, the gazing point shifts from the center of the image. That is, it is assumed that the convergence angles of the pair of left and right imaging units are fixed in accordance with, for example, an observation object at a close position. In this case, when an observation object at this close position is imaged, an image in which the observation object is positioned at the center is captured in either of the left and right imaging units (see FIG. 5 according to the embodiment of the present invention). On the other hand, if an observation object at a distant position is imaged, the convergence angles do not match, and therefore, in the image captured by the left-eye imaging unit 12, the observation object is shifted to the left from the center, and the right-eye observation is performed. In the image picked up by the image pickup unit 13, the observation object is shifted in the right direction from the center (see FIG. 6 according to the embodiment of the present invention). As a result, depending on how large the deviation amount is, there are problems that stereoscopic viewing becomes difficult, eyes are tired, and stereoscopic viewing is impossible.

  As a technique corresponding to this problem, Japanese Patent Application Laid-Open Nos. 7-95623 and 8-336165 disclose a technique (a convergence angle is set so that the gazing point is positioned at the center of an image by performing image processing). (Technology for matching image processing). That is, in the techniques described in these publications, the left and right images obtained by imaging are temporarily captured in a frame memory or the like, and the gazing point (for example, the center position of the observation object) is the center of the image in each of the left and right images. Thus, the convergence angle is adjusted by cutting out the image. The former Japanese Patent Application Laid-Open No. 7-95623, which is one of the above two publications, further describes a technique for determining the convergence angle by this image processing based on the measurement result of the focus distance.

In addition, a technique for performing so-called electronic zoom by adjusting the size of an image clipping range based on a set zoom magnification is also known.
JP-A-7-95623 JP-A-8-336165

  However, the difference between the case where the mechanical convergence angle is adjusted and the case where the convergence angle is fixed is not only that the gazing point deviates from the center of the image. That is, since the imaging plane of the image is perpendicular to the imaging optical axis, the imaging plane of the image obtained at a fixed convergence angle and the imaging plane of the image when the convergence angle is actually adjusted mechanically , Generally not parallel. Therefore, if an object is assumed to be equidistant from a pair of left and right imaging units, if an image is cut out centering on an object at a distance other than the distance corresponding to the fixed convergence angle, a trapezoidal distortion is generated in the image. Will occur.

  For example, when the convergence angle of the imaging unit is fixed in accordance with an object at a long distance as described in JP-A-7-95623, the image of the object at a short distance is displayed sideways. Trapezoidal distortion (that is, trapezoidal distortion in which two parallel lines constituting the trapezoid are arranged in the vertical direction). Moreover, since the trapezoidal distortion that occurs in the image of the observation object is left-right mirror-symmetrical between the left-eye imaging unit and the right-eye imaging unit, the left-right image is distorted from the original image as it goes to the periphery of the image. As a result, although normal stereoscopic vision cannot be performed, the technique described in the publication has not been considered for such problems.

  This trapezoidal distortion will be specifically described with reference to FIGS. 8 to 10 according to the embodiment of the present invention.

  First, FIG. 8 is centered on the observation object in the captured image as shown in FIG. 6B obtained by imaging the long-distance observation object FOBJ by the right-eye imaging unit 13 arranged as shown in FIG. The shape of the trapezoid distortion generated in the image portion is shown.

  On the other hand, FIG. 9 is centered on the observation object in the captured image as shown in FIG. 6A obtained by imaging the long-distance observation object FOBJ by the left-eye imaging unit 12 arranged as shown in FIG. The shape of the trapezoid distortion generated in the image portion is shown.

  When the image obtained by the right-eye imaging unit 13 as shown in FIG. 8 is observed with the right eye and the image obtained by the left-eye imaging unit 12 as shown in FIG. It is synthesized and observed as shown in FIG. That is, the left and right images are observed in the vicinity of the center of the image, whereas the left and right images are gradually shifted as they approach the periphery of the image. Thus, as a result of the difference between the image observed with the right eye and the image observed with the left eye, it becomes difficult to perform normal stereoscopic vision.

  The conventional technology as described above has not been considered in this regard.

  The present invention has been made in view of the above circumstances, and based on a pair of photographed images obtained from a pair of imaging units arranged with a fixed convergence angle, it is more natural without depending on the distance to the observation object. It is an object of the present invention to provide a stereoscopic image display device and a stereoscopic image display method capable of performing stereoscopic viewing.

  In order to achieve the above object, a stereoscopic image display device according to an aspect of the present invention includes a pair of imaging units fixed in left-right mirror symmetry so as to have a predetermined convergence angle at a predetermined interval, and the pair of imaging units described above. A distance measuring unit that substantially measures the distance from the midpoint of the image capturing unit to the observation object, and a pair of captured images obtained from the pair of image capturing units, respectively, and performs image processing on the pair of image capturing units. An image processing unit that generates a corresponding pair of display images; and a pair of display units that display the pair of display images generated by the image processing unit on each display screen. , The distance between the pair of imaging units, the distance from the midpoint of the pair of imaging units substantially measured by the ranging unit to the observation object, the convergence angle of the pair of imaging units, or the pair of imaging The midpoint of the pair of imaging units from the intersection of the optical axes Based on the distance, the pair of captured images in a range corresponding to each display screen of the pair of display units, by correcting the trapezoid so as to be symmetric with respect to the mirror surface of the left-right mirror surface, The pair of display images is generated.

  In addition, a stereoscopic image display method according to another aspect of the present invention includes a pair of imaging units fixed in a left-right mirror symmetry so as to have a predetermined convergence angle at a predetermined interval, and a midpoint of the pair of imaging units. A distance measuring unit that substantially measures the distance to the observation object, and a pair of display images corresponding to the pair of imaging units generated based on the pair of captured images obtained from the pair of imaging units. A stereoscopic image display method applied to a stereoscopic image display device including a pair of display units displayed on each display screen, wherein the distance between the pair of imaging units and the distance measuring unit substantially The measured distance from the midpoint of the pair of imaging units to the object to be observed and the angle of convergence of the pair of imaging units or the optical axis of the pair of imaging units to the midpoint of the pair of imaging units And corresponding to each display screen of the pair of display units based on A pair of photographed images of the circumference, by keystone correction so as to be symmetrical with respect to the mirror surface of the right and left mirror symmetry, a method of generating the pair of the display image.

  According to the stereoscopic image display device and the stereoscopic image display method of the present invention, the distance to the observation object is determined based on a pair of captured images obtained from a pair of imaging units arranged with a fixed convergence angle. Thus, stereoscopic viewing can be performed more naturally.

  Embodiments of the present invention will be described below with reference to the drawings.

[Embodiment 1]
1 to 26 show Embodiment 1 of the present invention, FIG. 1 is a diagram showing a stereoscopic image display device from the front side, and FIG. 2 is a right side (left eye side) of the stereoscopic image display device. FIG. 3 is a diagram showing the stereoscopic image display device from the back side.

  The stereoscopic image display device 1 of the present embodiment is configured as a stereoscopic glasses type display device that is used by being mounted on the head in substantially the same manner as glasses.

  That is, the main body 2 is disposed on the front side of the stereoscopic image display device 1. Further, of the end portions of the main body portion 2, the left temple portion 3 for hanging on the left ear is attached to the end portion shown on the left side of FIG. 3, and the right ear is attached to the end portion shown on the right side of FIG. The right temple portions 4 for hanging on are arranged respectively.

  A frame 5 is provided on the upper portion of the main body 2 described above, and a control unit 11 is built in the frame 5.

  A left-eye portion 6 and a right-eye portion 7 are disposed below the frame 5, and a left-eye imaging unit 12 is disposed on the front side (observed object side) of the left-eye portion 6. On the side (observation object side), the right-eye imaging unit 13 is arranged. The right-eye imaging unit 13 and the left-eye imaging unit 12 are spaced at a standard eye width of, for example, 63 mm (hereinafter referred to as a camera interval) and fixed at a predetermined convergence angle as described later. It is a small camera arranged. Although not shown, each of the imaging units 12 and 13 is an imaging element composed of a CMOS, a CCD, or the like, and a left eye and a right eye for forming an optical image on the imaging surface of these imaging elements. The image pickup optical system includes a focus mechanism for adjusting the focus of the image pickup optical system, a signal processing circuit for performing signal processing on an image obtained from each image pickup device, and the like.

  The image signals obtained from the right-eye imaging unit 13 and the left-eye imaging unit 12 are output to the control unit 11 described above. As will be described later, the control unit 11 cuts out at least a part of the image obtained from the right-eye image pickup unit 13, cuts out at least a part of the image picked up by the left-eye image pickup unit 12, and forms each cut-out image. Then, a process for correcting the inclination of the imaging plane based on the difference from the fixed convergence angle is performed to generate a right-eye display image and a left-eye display image.

  Further, as shown in FIG. 3, a left-eye display unit 14 for displaying a left-eye image generated by the control unit 11 is provided on the back side (head side) of the left-eye unit 6. On the back side (head side), a right-eye display unit 15 for displaying a right-eye image generated by the control unit 11 is arranged. The left-eye display unit 14 and the right-eye display unit 15 are small display devices configured using, for example, a transmissive LCD, a reflective LCD, or an OLED. The distance between the center position of the left-eye display unit 14 and the center position of the right-eye display unit 15 can be adjusted according to the actual eye width of the user with a standard eye width of 63 mm as a reference. It is comprised so that it may become.

  Furthermore, on the front side (observed object side) of the bridge portion 8 connecting the left eye portion 6 and the right eye portion 7, an observation is made at a position near the midpoint between the right eye imaging portion 13 and the left eye imaging portion 12. A distance measuring unit 16 for measuring the distance to the object is provided. The distance measuring unit 16 measures the distance from the stereoscopic image display device 1 to the observation object based on, for example, a triangulation method. That is, the distance measuring unit 16 includes a reflection type distance measuring device that includes an infrared light emitting unit for emitting infrared rays toward the observation object and a light receiving unit that receives infrared rays reflected by the observation object. It is a sensor.

  The distance measurement unit 16 is not limited to the distance measurement based on the triangulation method, and may be a distance measurement by a laser measurement, for example, or a distance measurement using a method based on image processing. It may be a thing, and what measures distance using other techniques may be used.

  Further, here, in order to directly measure the distance from the midpoint between the right-eye image pickup unit 13 and the left-eye image pickup unit 12 to the observation object, the distance measuring unit 16 is provided at the position (near the center) of this midpoint. However, it is not limited to this. If the position of the distance measuring unit 16 is fixed with respect to the positions of the right-eye imaging unit 13 and the left-eye imaging unit 12, the right-eye imaging unit 13 and the left-eye imaging unit 13 and the left-eye imaging unit 16 are based on the arrangement determined at the time of design. It is possible to calculate the distance from the midpoint of the imaging unit 12 to the observation object (for example, using the three-square theorem or the cosine theorem). As described above, the distance measuring unit 16 only needs to be able to substantially measure the distance from the midpoint between the right-eye imaging unit 13 and the left-eye imaging unit 12 to the observation object, and does not need to be able to directly measure.

  Next, FIG. 11 is a block diagram illustrating a circuit configuration of the stereoscopic image display device 1.

  As described above, the stereoscopic image display device 1 includes the control unit 11, the left-eye imaging unit 12, the right-eye imaging unit 13, the left-eye display unit 14, the right-eye display unit 15, and the distance measuring unit. 16.

  Among these, the control unit 11 includes a processor 21, a nonvolatile memory 22, a left-eye image processing unit 23, and a right-eye image processing unit 24.

  The nonvolatile memory 22 is a memory that can hold various data and parameters without receiving power supply, and is configured using an EEPROM, a flash memory, or the like. Here, various parameters stored in the nonvolatile memory 22 include a convergence angle ψ (see FIG. 26 and the like) formed by the left-eye imaging unit 12 and the right-eye imaging unit 13 (or the light of the left-eye imaging unit 12). The distance from the intersection P1 (see FIG. 26, etc.) between the axis and the optical axis of the right-eye imaging unit 13 to the midpoint S (see FIG. 26, etc.) between the left-eye imaging unit 12 and the right-eye imaging unit 13), The camera interval 2d between the left-eye imaging unit 12 and the right-eye imaging unit 13 (see FIG. 26 and the like), the left-eye imaging unit 12, the right-eye imaging unit 13, the left-eye image processing unit 23, the right-eye image processing unit 24, and the left eye Examples of the register setting values of the display unit 14 for use, the display unit 15 for the right eye, etc. are given as some examples. The nonvolatile memory 22 also stores a control program for the stereoscopic image display device 1 that is executed by the processor 21. The distance D (Pk) to the observation object (see FIG. 13 and the like) is input to the control unit 11 when the distance measurement unit 16 performs the measurement.

  The left-eye image processing unit 23 processes the image obtained from the left-eye imaging unit 12 to generate a left-eye display image, and outputs the generated left-eye display image to the left-eye display unit 14. As a result, the left-eye display unit 14 displays the left-eye display image.

  The right-eye image processing unit 24 processes the image obtained from the right-eye imaging unit 13 to generate a right-eye display image, and outputs the generated right-eye display image to the right-eye display unit 15. As a result, the right-eye display unit 15 displays a right-eye display image.

  The processor 21 includes a left-eye imaging unit 12, a right-eye imaging unit 13, a left-eye display unit 14, a right-eye display unit 15, a ranging unit 16, a nonvolatile memory 22, a left-eye image processing unit 23, and a right-eye image process. Each unit 24 is bidirectionally connected, and based on the control program described above, the entire stereoscopic image display apparatus 1 including these is controlled. In particular, the processor 21 determines the distance D (Pk) to the observation object obtained from the distance measuring unit 16 as described above and the camera convergence angle ψ read from the nonvolatile memory 22 (or the intersection point P1 to the midpoint S). Based on the distance) and the camera interval 2d, the left eye image processing unit 23 performs projection conversion as described later to create an appropriate left eye display image, and similarly, the right eye image processing unit 24 projects the projection. Control is performed so that an appropriate right-eye display image is generated by performing conversion.

  First, in this embodiment, the situation as shown in FIG. 4 is assumed. FIG. 4 is a diagram illustrating a configuration example in which two imaging units are fixed at a predetermined convergence angle.

  That is, the left-eye imaging unit 12 and the right-eye imaging unit 13 are arranged with a predetermined camera interval so as to be left-right mirror-symmetric with respect to the left-right center line LOC. Furthermore, the left-eye imaging unit 12 and the right-eye imaging unit 13 are fixed so that the imaging optical axes LOA and ROA intersect each other with a predetermined convergence angle on the left-right center line LOC. In particular, in the present embodiment, it is assumed that the object to be observed is, for example, a tooth in the dental field, and the position where the imaging optical axis LOA and the imaging optical axis ROA intersect is the closest distance assumed at the time of tooth observation. Designed to be Therefore, the position at which the imaging optical axes intersect is referred to as the closest gazing point (the observation object located at this closest gazing point is indicated as NOBJ). However, the reason for fixing the position where the imaging optical axes intersect to the closest point of sight is to make the explanation easy to understand, so of course it is not limited to fixing so as to intersect at this position, You may fix so that it may cross at a position.

  Therefore, when the near-observed object NOBJ located at the closest gazing point is imaged, as shown in FIG. 5, the gazing point imaged by the left-eye imaging unit 12 is positioned at the center of the image (see FIG. 5A). The gazing point imaged by the right-eye imaging unit 13 is also located at the center of the image (see FIG. 5B). FIG. 5 is a diagram showing a state of an image obtained by fixing the convergence angle of the image pickup unit according to the close observation object and picking up the close observation object by the pair of left and right image pickup units. In FIG. 4, the horizontal imaging range (horizontal angle of view) of the right-eye imaging unit 13 (and the left-eye imaging unit 12 (not shown)) is 2θx.

  Then, the gazing point of the observation object at an arbitrary distance is the left-eye imaging unit 12 on a plane having the left-eye imaging unit 12, the right-eye imaging unit 13, and the closest gazing point (the closest gazing point of the closest observation object NOBJ). Suppose that it is located on the straight line (namely, the above-mentioned left-right center line LOC) orthogonal to the midpoint of the line segment which connects image pick-up part 13 for right eyes. In other words, the gazing point at an arbitrary distance is assumed to be located at an equal distance from the left-eye imaging unit 12 and the right-eye imaging unit 13 on a plane with the left and right imaging optical axes.

  In such an arrangement, it is assumed that a long-distance observation object FOBJ located at a position far from the closest point of sight is imaged. Then, as can be seen with reference to FIGS. 4 and 7, since the long-distance observation object FOBJ is located on the right side of the imaging optical axis ROA of the right-eye imaging unit 13, the image is captured by the right-eye imaging unit 13. The image obtained in this way is as shown in FIG. 6B, and the image of the long-distance object FOBJ is closer to the right side than the center of the image. Similarly, since the long-distance observation object FOBJ is located on the left side of the imaging optical axis LOA of the left-eye imaging unit 12, an image obtained by imaging with the left-eye imaging unit 12 is shown in FIG. Thus, the image of the long-distance object FOBJ is closer to the left side than the center of the image. FIG. 6 is a diagram showing an image obtained by fixing the convergence angle of the imaging unit according to the close-up observation object and imaging the long-distance observation object with the pair of left and right imaging units, and FIG. 7 is the imaging for the right eye. It is a perspective view which shows the imaging | photography range at the time of image | photographing the near observation object NOBJ and the long distance observation object FOBJ by the part 13, respectively with a rectangle. In FIG. 7, the photographing range when photographing the close observation object NOBJ is indicated by reference numeral S1, and the photographing range when photographing the long distance observation object FOBJ is indicated by reference numeral S2.

  And the trapezoid distortion which arises in the vicinity of the long-distance observation object FOBJ is as shown in FIGS.

  First, FIG. 8 is centered on the observation object in the captured image as shown in FIG. 6B obtained by imaging the long-distance observation object FOBJ by the right-eye imaging unit 13 arranged as shown in FIG. It is a figure which shows the shape of the trapezoid distortion which has arisen in the image part made into. The trapezoidal distortion shown in FIG. 8 is a plane that is perpendicular to the straight line from the right-eye imaging unit 13 to the long-distance observation object FOBJ in the arrangement shown in FIG. 4, and is horizontally long on the plane that passes through the long-distance observation object FOBJ. When the rectangle is placed, the shape of the rectangle imaged by the right-eye imaging unit 13 is shown.

  On the other hand, FIG. 9 is centered on the observation object in the captured image as shown in FIG. 6A obtained by imaging the long-distance observation object FOBJ by the left-eye imaging unit 12 arranged as shown in FIG. It is a figure which shows the shape of the trapezoid distortion which has arisen in the image part made into. The trapezoidal distortion shown in FIG. 9 is the plane perpendicular to the straight line from the left-eye imaging unit 12 to the long-distance observation object FOBJ in the arrangement shown in FIG. When the same horizontally long rectangle is placed, the shape of the rectangle imaged by the left-eye imaging unit 12 is shown.

  First, the trapezoidal distortion shown in FIG. 8 has a shape in which the left side is relatively reduced rather than the right side of the image. On the other hand, the trapezoidal distortion shown in FIG. 9 has a shape in which the right side is relatively reduced rather than the left side of the image. The trapezoidal distortion shown in FIG. 8 and the trapezoidal distortion shown in FIG. 9 are left-right mirror symmetrical.

  FIG. 10 is a diagram illustrating a situation when the trapezoidal distortion illustrated in FIG. 8 and the trapezoidal distortion illustrated in FIG. 9 are observed with both eyes.

  When the image obtained by the right-eye imaging unit 13 as shown in FIG. 8 is observed with the right eye and the image obtained by the left-eye imaging unit 12 as shown in FIG. 9 is observed with the left eye, these are synthesized. As shown in FIG.

  That is, the left and right images are observed in the vicinity of the center of the image, whereas the left and right images are gradually shifted as they approach the periphery of the image (for example, the right eye image is observed due to trapezoidal distortion on the right side). If the image is on the enlarged side, the right side of the left-eye image is the reduced side of the image due to trapezoidal distortion, so the image observed with both the left and right eyes will be significantly different from the surrounding area).

  Therefore, image processing based on projective transformation for correcting such trapezoidal distortion is performed in the above-described left-eye image processing unit 23 and right-eye image processing unit 24 as described below. Since the left-eye imaging unit 12 and the right-eye imaging unit 13 are arranged so as to be left-right mirror-symmetric as described above, the following description mainly relates to the right-eye imaging unit 13 (that is, It can be considered that the left-eye imaging unit 12 is mirror-symmetrical with the right-eye imaging unit 13).

  In the present embodiment, it is assumed that the distance range to the observation object is from the closest gazing point P1 to the farthest gazing point P2. At this time, FIG. 12 is a diagram showing an arrangement when the right-eye imaging unit 13 and the closest gazing point P1 and the farthest gazing point P2 are viewed from above, and FIG. It is a figure which shows arrangement | positioning when the arbitrary gaze points Pk between the gaze point P1 and the farthest gaze point P2 are seen from upper direction.

  First, each reference numeral shown in FIGS. 12 and 13 will be described.

  The point O indicates the position of the right-eye imaging unit 13 (this position corresponds to the position of the human right eye).

  The point S indicates the position of the midpoint between the right-eye imaging unit 13 and the left-eye imaging unit 12 (this position corresponds to the center point of the human eye width). It is assumed that the distance measuring unit 16 described above is disposed at the position of this point S. The length from point O to point S is d. Therefore, this length d represents half the camera interval between the left-eye imaging unit 12 and the right-eye imaging unit 13 (in other words, the camera interval is 2d).

  The left-right center line LOC is a left-right symmetric axis between the right-eye image pickup unit 13 and the left-eye image pickup unit 12 on a plane having the points O, S, and P1. The left-right center line LOC passes through the point S and is orthogonal to the straight line connecting the left-eye imaging unit 12 and the right-eye imaging unit 13 at this point S.

  As described above, the point P1 is a close focus point. The distance from the point O to the closest point of sight P1 is L1. In addition, the distance from the point S to the closest gazing point P1 is D (P1). In the present embodiment, as described above, the optical axis of the right-eye imaging unit 13 is set so as to coincide with the straight line OP1.

  Point P2 is the farthest gazing point as described above. The distance from the point O to the farthest gazing point P2 is L2. Further, the distance from the point S to the farthest gazing point P2 is D (P2).

  The point Pk is an arbitrary gazing point as described above, and indicates a gazing point at an arbitrary position between the points P1 and P2 (but including both end points). The distance from the point O to the arbitrary gazing point Pk is Lk (according to the definition of the arbitrary gazing point Pk described here, L1 ≦ Lk ≦ L2 is satisfied). Further, the distance from the point S to the arbitrary gazing point Pk is defined as D (Pk).

  The plane β is an imaging plane that virtually shows the imaging range (assumed to be a rectangular area as described above) by the right-eye imaging unit 13 at an appropriate distance position. This imaging plane β is defined based on angles θx and θy (described later) indicating the angle of view when viewed from the point O, and is not defined by the distance from the point O. FIG. 17 is a perspective view showing the positional relationship between the point O and the imaging plane β and the observation plane αk (described later), and FIG. 18 is a front view showing the positional relationship between the point O and the imaging plane β. (B) Plan view, (C) Right side view, FIG. 19 shows (A) Front view, (B) Plan view, (C) Right side view, FIG. 21 shows the positional relationship between the point O and the observation plane αk. FIG. 6 is a perspective view showing a positional relationship between a point O, a point A on an observation plane αk, and a point B on an imaging plane β. Hereinafter, description will be made with reference to FIGS. 17 to 19 and 21 as appropriate.

  A point Q indicates a center point on the imaging plane β, and is an intersection of the straight line OP1 and the imaging plane β.

  The line segment Lw is a line segment connecting the point O and the point Q, and is a perpendicular line of the imaging plane β. If the length (distance) of the line segment Lw is similarly expressed by Lw, the distance Lw indicates the distance from the point O to the point Q (that is, the distance from the point O to the imaging plane β). As described above, since the imaging plane β is a virtual plane, this distance Lw can be arbitrarily set.

  As shown in FIG. 18B, the angle θx is the left or right angle of view around the point Q of the imaging plane β (that is, the horizontal angle of view of the entire imaging plane β is 2θx). . Further, as shown in FIG. 18C, the angle θy is an angle of view on the upper or lower side of the imaging plane β with respect to the point Q (that is, the angle of view in the vertical direction of the entire imaging plane β is 2θy). Becomes).

  The plane α1 is an observation plane (which is a rectangular area) when observing the closest gazing point P1, and is set within the range of the angle of view formed by the imaging plane β viewed from the point O (this is imaging This is because the image portion corresponding to the observation plane α1 is cut out from the image obtained in this manner). The observation plane α1 is perpendicular to the line segment OP1 (the line segment with the distance L1 described above). The left or right field angle of the observation plane α1 centered on the point P1 is ωx (1) (that is, the horizontal field angle of the entire observation plane α1 is 2ωx (1)).

  The plane α2 is an observation plane (assumed to be a rectangular area) when observing the farthest gazing point P2, and is set within the range of the angle of view formed by the imaging plane β viewed from the point O (same as described above). This is because the image portion corresponding to the observation plane α2 is cut out from the image obtained by imaging. This observation plane α2 is perpendicular to the line segment OP2 (the line segment having the distance L2 described above). The left or right field angle of the observation plane α2 around the point P2 is ωx (2) (that is, the horizontal field angle of the entire observation plane α2 is 2ωx (2)).

  The plane αk is an observation plane (assuming a rectangular area) when observing an arbitrary gazing point Pk between the closest gazing point P1 and the farthest gazing point P2, and the imaging plane β viewed from the point O is It is set within the range of the angle of view to be made (as described above, because the image portion corresponding to this observation plane αk is cut out from the image obtained by imaging). This observation plane αk is perpendicular to the line segment OPk (the line segment having the distance Lk described above). The viewing angle on the left or right side of the observation plane αk with respect to the point Pk is ωx (k) as shown in FIG. 19B (that is, the horizontal viewing angle of the entire observation plane αk is 2ωx ( k)). Similarly, the upper or lower field angle of the observation plane αk with respect to the point Pk is set to ωy (k) as shown in FIG. 19C (that is, the vertical field angle of the entire observation plane αk). Becomes 2ωy (k)).

  These ωx (k) and ωy (k) are angles of view indicating the display range. Further, by making these ωx (k) and ωy (k) variable within the angles θx and θy indicating the imaging range of the imaging unit, the display range can be enlarged / reduced (electronic zoom is performed). it can. At this time, these ωx (k) and ωy (k) are the enlargement / reduction parameters.

  The angle δ2 is an angle formed by the closest gazing point P1 and the farthest gazing point P2 viewed from the point O (or an angle formed by the point Q and the maximal gazing point P2 viewed from the point O). Since the observation plane α1 and the imaging plane β are parallel, the angle δ2 is also an angle formed by the observation plane α2 and the imaging plane β.

  The angle δk is an angle formed by the closest gazing point P1 and the gazing point Pk viewed from the point O (or an angle formed by the point Q and the gazing point Pk viewed from the point O). As described above, since the observation plane α1 and the imaging plane β are parallel, the angle δk is also an angle formed by the observation plane αk and the imaging plane β. Based on the definition of the observation plane αk described above, the angle δk takes a value in the range of 0 ≦ δk ≦ δ2.

  Point A indicates a certain display pixel position in the observation plane αk.

  Point B is a point where the straight line OA intersects the imaging plane β (see also FIG. 21). The pixel value of the coordinates of the point B is set as the pixel value of the point A. However, it cannot be expected that the pixel of the captured image is exactly located at the coordinates of the point B, and is generally an intermediate position between pixels obtained by imaging. Therefore, by performing an interpolation process based on the imaging pixel located in the vicinity of the point B coordinate, a pixel value at the point B coordinate is obtained.

  Further, when the point O is taken as the origin and the close focus point P1 is watched, the vector OP1 (here, the vector should be represented by, for example, a bold character, but for the reason of notation, a normal character is appropriately substituted. The same applies hereinafter.) When the farthest gazing point P2 is watched, the vector OP2 is used. When the gazing point Pk is watched, the direction of the vector OPk is the z-axis direction. Further, on the plane passing through the point O, the point S, the point P1, the point P2, or the point Pk, the direction orthogonal to the z-axis direction is the x-axis direction, and the direction orthogonal to the x-axis and z-axis is the y-axis direction. Let's set the coordinate system.

  First, it is assumed that the observation object is a planar chart as shown in FIG. FIG. 20 is a diagram showing an example of a plane chart as an observation object. This planar chart has a rectangular shape that matches the aspect ratio of the display screen (for example, horizontal 4: longitudinal 3), and horizontal lines and vertical lines that intersect at the center point are drawn.

  The plane chart matches the observation plane α1 when it is at the closest position, matches the observation plane α2 when it is at the farthest position, and matches the observation plane αk when it is at a position passing through the point Pk. (That is, the size of the planar chart is appropriately scaled according to the range to be observed (according to the above-described scaling parameters ωx (k), ωy (k)). And). As a result, an image of the entire area of the planar chart is displayed on the entire display screen.

The outline of the processing procedure to be performed by the control unit 11 under such settings and definitions is as follows.
(1) First, the coordinates of a certain pixel position (point A) in the observation plane αk are obtained.
(2) Next, the coordinates of the point B on the imaging plane β corresponding to the point A are obtained.
(3) Subsequently, for example, four pixel positions located in the vicinity of the point B are obtained. Then, the pixel value at the point B is obtained based on the obtained pixel values at the four pixel positions, and the obtained pixel value is set as the pixel value at the point A.
By performing the processing as described above for all pixel positions on the observation plane αk, an image of the observation plane αk can be displayed on the display unit.

  Details of such processing will be described below.

  First, the processing procedure (1) will be described with reference to FIG. FIG. 22 is a diagram showing a pixel configuration on the observation plane αk. In FIG. 22, lattice points indicate pixel positions.

  In FIG. 22 (and FIG. 24 to be described later), for the sake of simplicity, the display unit is described as being modeled assuming that the pixel configuration is 9 × 7 pixels. In FIGS. 23 and 24, which will be described later, for the sake of simplicity, the pixel configuration of the image pickup unit is described as being modeled as having horizontal 11 × vertical 9 pixels. However, in an actual product or the like, the pixel configuration of the imaging unit takes a large value, for example, UXGA (horizontal 1600 × vertical 1200), and the pixel configuration of the display unit, for example, SVGA (horizontal 800 × vertical 600).

When the xyz coordinate system is set with the point O as the origin as described above, the coordinates of the center of the observation plane αk are (0, 0, Lk) with reference to FIG. At this time, the coordinates of the four corner points of the observation plane αk are as shown in FIG. That is, the coordinates of the upper right corner point are
(Lk · tan {ωx (k)}, Lk · tan {ωy (k)}, Lk)
The coordinates of the upper left corner point are
(−Lk · tan {ωx (k)}, Lk · tan {ωy (k)}, Lk)
The coordinates of the lower left corner point are
(−Lk · tan {ωx (k)}, −Lk · tan {ωy (k)}, Lk)
The coordinates of the lower right corner point are
(Lk · tan {ωx (k)}, −Lk · tan {ωy (k)}, Lk)
It becomes.

An arbitrary pixel Aij on the observation plane αk (where i is an integer indicating the pixel arrangement in the x direction, j is an integer indicating the pixel arrangement in the y direction, and in the example shown in FIG. The coordinates of (≦ i ≦ 4, −3 ≦ j ≦ 3) (the coordinates of the point A) are expressed by the following mathematical formula 1.
[Equation 1]

  Next, the above-described processing procedure (2) is as follows.

First, the following formula is established from the geometrical arrangement of each point. That is, when t is a scalar real parameter, the point B is located on the extension of the line segment OA (see FIG. 13), so the following Equation 2 is established between the vector OA and the vector OB.
[Equation 2]
OB = t · OA

Further, since the line segment OQ is orthogonal to the imaging plane β, the following Equation 3 is established between the vector OQ and the vector QB.
[Equation 3]
(OQ, QB) = 0
Here, the symbol (x, y) represents the inner product of the vector x and the vector y.

The vector QB appearing in the expression 3 is expressed as the following expression 4 using the vector OB and the vector OQ.
[Equation 4]
QB = OB-OQ

Substituting Equation 2 into the vector OB portion on the right side of Equation 4 and then substituting Equation 4 into Equation 3 yields the following Equation 5.
[Equation 5]
(OQ, t · OA−OQ) = 0

By modifying Equation 5, the parameter t is obtained as shown in Equation 6 below.
[Equation 6]

Therefore, the coordinates of the vector OB are obtained as shown in the following Expression 7 based on the vector OQ and the vector OA by substituting Expression 6 into Expression 2.
[Equation 7]

It can be seen that the vector OQ appearing on the right side of Equation 7 has a component as shown in Equation 8 below with reference to FIG.
[Equation 8]

  By the way, as described above, the imaging plane β is a virtual plane, and the distance Lw can be arbitrarily set. The meaning that the distance Lw can be arbitrarily set will be described with reference to FIGS. 15 and 16. 15 is a plan view showing the positional relationship between the imaging plane β whose distance from the point O is Lw and the imaging plane β ″ which is Lw ″, and FIG. 16 is an imaging plane β and Lw whose distance from the point O is Lw. FIG. 6 is a perspective view showing a positional relationship with “imaging plane β”. FIG. 16 shows an example in which the point B is on the line segment QC and the point B ″ is on the line segment Q ″ C ″. However, in general, the point B is an arbitrary point on the imaging plane β, The point B ″ may be an arbitrary point on the imaging plane β ″.

  The fact that the distance Lw can be arbitrarily set means that a plane at an arbitrary distance from the point O can be set as a new shooting plane if the plane is parallel to the shooting plane β. . However, since the photographing plane is defined by the angle θx indicating the horizontal field angle and the angle θy indicating the vertical field angle as described above, it is reduced in proportion to the distance on the short distance side, If it is on the distance side, it is enlarged in proportion to the distance.

  By the way, an arbitrary shooting pixel position in the shooting plane β is a relative position in the shooting plane β, and can be specified by setting a relative pixel coordinate in the shooting plane β.

  As a specific example, it is assumed that the imaging device has a substantially UXGA configuration with horizontal 1601 × vertical 1201 pixels. The relative pixel coordinates on the imaging plane β are set so that the center of the imaging plane is the origin Q (0, 0) and the relative pixel coordinates of the upper right corner point C are (800, 600). To do. If this coordinate is taken without depending on the distance Lw to the imaging plane β, the relative pixel coordinate of the origin Q ″ is (0, 0) and the upper right corner point C ″ is on the imaging plane β ″. The relative pixel coordinates are (800, 600), and at this time, the point B on the imaging plane β and the point B ″ on the imaging plane β ″ represented by the same relative pixel coordinates pass through the point O. This is because the shooting plane is defined by the angles θx and θy, and the size of the shooting plane is reduced / enlarged in proportion to the distance from the point O (that is, a triangle). The OQC and the triangle OQ “C” are similar, and the triangle OQB and the triangle OQ “B” are also similar).

  Accordingly, the relative pixel coordinates on the imaging plane β at the point B where the straight line passing through the point A and the point O on the observation plane αk and the imaging plane β intersect are the same regardless of the distance Lw. The neighboring pixel value of the point B on the imaging plane β to be referred to as the pixel value of A is the same without depending on the distance Lw.

For this reason, it is understood that the distance Lw may be arbitrary when referring to the pixel value in the vicinity of the position of the point B on the imaging plane β in order to calculate the pixel value of the point A on the observation plane αk. In the following, it is assumed that Lw = 1. Then, Formula 8 is rewritten to the following Formula 9.
[Equation 9]

  Therefore, based on the coordinates of the point B obtained by the above-described Equations 1, 7, and 9, the pixel value at the point B can be interpolated from the pixel value of the pixel located near the point B. However, when the calculation is performed using a four-point interpolation formula described later, if the z coordinate of the point B is different from the z coordinate of the pixel used for the interpolation, the calculation becomes complicated. Then, necessary information regarding the imaging plane β ′ obtained by rotating the imaging plane β around the y-axis by an angle δk (as shown in FIG. 14, this imaging plane β ′ is parallel to the observation plane αk). I will ask for it. FIG. 14 is a diagram showing an arrangement when the imaging plane β ′ and the observation plane αk obtained by rotating the imaging plane β about the y axis by an angle δk when viewed from above. FIG. 24 is a diagram showing a state of the imaging plane β ′ obtained by further rotating the imaging plane β on which the observation plane αk is projected by projection around the point O by an angle δk around the y axis. . In FIG. 24, to help understanding, the coordinates of the point Q ′ and the four corner points on the imaging plane β ′ are shown (note that Lw is clearly shown in FIG. 24, As described above, Lw = 1 may be set in actual calculation).

  The necessary information regarding the rotated imaging plane β ′ includes information on the coordinates of the point Q ′ obtained by rotating the point Q about the y axis by an angle δk, and rotating the point B about the y axis by an angle δk. Information of the coordinates of the point B ′ obtained by the above. Since the arbitrary point on the imaging plane β ′ after rotation has the same z coordinate, the arbitrary position on the imaging plane β ′ can be expressed only by a combination of x and y coordinates.

  First, the point Q ′ is necessary for calculating the coordinates of each imaging pixel on the rotated imaging plane β ′. That is, the imaging pixels are discretely arranged at predetermined pixel intervals in the x-axis direction and the y-axis direction on the imaging plane β ′. The vertical and horizontal sizes of the imaging plane β ′ are the same as the vertical and horizontal sizes of the imaging plane β because they do not change with rotation. The size of the imaging plane β in the x-axis direction is 2 · Lw · tan θx as shown in FIGS. 12 to 14, and the size in the y-axis direction is also 2 · Lw · tan θy. is there. Therefore, if the coordinates of the point Q ′ are known, the coordinates of an arbitrary imaging pixel are calculated based on the size in the x-axis direction and the y-axis direction described above and the pixel arrangement in the x-axis direction and the y-axis direction of the imaging element. Is possible.

Thus, if a rotation matrix that rotates about the y-axis by δk is V (δk), a vector OQ ′ obtained by rotating the vector OQ about the y-axis by δk is calculated as shown in Equation 10 below. .
[Equation 10]
OQ '= V (δk) OQ

Similarly, a vector OB ′ obtained by rotating the vector OB about the y axis by δk is calculated as shown in the following Expression 11.
[Equation 11]
OB '= V (δk) OB

  Since the point Q ′ and the point B ′ are points on the imaging plane β ′ after rotation, the z component is the same as described above.

Specifically, the rotation matrix V (δk) of the angle δk around the y-axis is expressed as a 3 × 3 matrix as shown in the following Expression 12. Here, the positive direction of rotation is a clockwise direction in accordance with FIG.
[Equation 12]

  Therefore, the coordinates of the point B ′ and the coordinates of the point Q ′ can be obtained by calculating the equation 12 into the vector OB obtained by the equation 7 and the vector OQ obtained by the equation 9.

That is, if the formula 7 is substituted into the formula 11, the following formula 13 is obtained.
[Equation 13]

  Therefore, the vector OB ′ is obtained by substituting the vector OA sequentially calculated for each pixel by the above-described formula 1 into the formula 13. Further, the vector OQ 'is similarly obtained.

  Next, the process procedure (3) described above will be described with reference to FIGS. In the above description, the processing procedure (3) has been described as obtaining the pixel positions of, for example, four points located in the vicinity of the point B. However, in order to simplify the calculation, the processing procedure (3) is replaced with the point B. Since the point B ′ is obtained, here, for example, four pixel positions located in the vicinity of the point B ′ are obtained. Accordingly, the pixel value at the point B ′ is obtained based on the obtained pixel values at the four pixel positions, and the obtained pixel value is set as the pixel value at the point A.

  Here, FIG. 22 shows the pixel configuration of the observation plane αk as described above, and FIG. 24 shows the imaging plane β on which the observation plane αk is projected by projection around the point O as described above, and further around the y axis. Shows the state of the imaging plane β ′ obtained by rotating the angle δk. FIG. 23 is a diagram illustrating a pixel configuration on the imaging plane β, and FIG. 25 is a diagram illustrating a point B ′ and four pixel positions located in the vicinity thereof. FIG. 25 is an enlarged view of a circled portion in FIG.

  The grid lines shown in FIGS. 22 to 25 mean that there are imaging pixels of the imaging unit or display pixels of the display unit at intersections (lattice points) of the grid lines. Then, in the circled part of FIG. 24 and FIG. 25, circle points are added to the lattice points to clearly show that the pixel is located at the lattice point.

  As described above, the point A is a point on the observation plane αk and corresponds to one of the pixels of the display unit. However, the point B ′ on the imaging plane β ′ corresponding to the point A is a coincidence. Except for the image pickup pixel of the image pickup unit, it is an intermediate position between the pixels of the image pickup element. Therefore, in the present embodiment, the pixel value of the point B ′ is calculated using a four-point interpolation method. For this purpose, first, four photographing pixel positions located in the vicinity of the point B ′ are obtained.

  Specifically, the pixel value is calculated as follows.

  First, coordinate scaling is selected so that the interval between pixels adjacent in the x-axis direction and pixels adjacent in the y-axis direction is 1.

  Then, the coordinates of the point B ′ on the imaging plane β ′ are obtained by rotating (x + dx, y + dy) (where 0 ≦ dx <1 and 0 ≦ dy <1) (rotated by the angle δk as described above. It is assumed that the z plane coordinate is fixed (Lw) and thus the z plane coordinate is omitted).

  Then, the coordinates of the four imaging pixels located in the vicinity of the point B ′ are (x, y), (x + 1, y), (x, y + 1), and (x + 1, y + 1). If the pixel values of these four image pickup pixels are expressed by adding a character β in front of the coordinates, β (x, y), β (x + 1, y), β (x, y + 1), β (x + 1), respectively. , Y + 1).

Then, the pixel value β (x + dx, y + dy) at the point B ′ can be calculated based on the four-point interpolation method as shown in the following Expression 14, using the pixel values of these four imaging pixels.
[Formula 14]
β (x + dx, y + dy) = β (x, y) × (1-dx) (1-dy)
+ Β (x + 1, y) × dx (1-dy)
+ Β (x, y + 1) × (1-dx) dy
+ Β (x + 1, y + 1) × dxdy

  Then, the pixel value obtained by Equation 14 is the pixel value of the display pixel at the point A.

  By performing such processing for all display pixel positions on the observation plane αk (that is, for all pixels of the display device), it is possible to perform observable display.

  Note that the calculation of the pixel value performed by the processing as described above may be performed by calculation each time, or by creating a lookup table and referring to this lookup table. It doesn't matter if you do.

  In addition, in Formula 14, an example in which the pixel value of the point B ′ is calculated by the four-point interpolation method is shown, but the present invention is not limited to this. As an example of another method, it is conceivable to use a technique of selecting a pixel value of one point located closest to the point B ′. By using this technique, there is an advantage that processing can be speeded up by reducing the addition of operations. As another example of the method, it is conceivable to use a technique of detecting the edge direction and averaging (or weighted average) pixel values of a plurality of points located in the vicinity in the detected edge direction. By using this technique, horizontal edges, vertical edges, diagonal edges, and the like can be stored, and there is an advantage that image quality deterioration when an observation image is created from a captured image can be reduced. The pixel values are not limited to the examples given above, but may be calculated by other methods.

  Next, components for calculating the coordinates of the point B ′ will be described. Note that the component is an element necessary for calculating coordinates, and means, for example, a shooting angle, a shooting distance, a display angle of view, and the like.

  The point B ′ is calculated by Expression 13, which is composed of a vector OQ, a vector OA, and a rotation matrix V (δk).

  Here, the vector OQ is expressed by Equation 9, and its constituent elements are sin δk and cos δk.

  The vector OA is expressed by Equation 1, and its components are Lk, ωx (k), and ωy (k).

  Further, the rotation matrix V (δk) is expressed by Equation 12, and its constituent elements are sin δk and cos δk.

Therefore, the component for calculating the coordinates of the point A is
Lk, ωx (k), ωy (k)
It will be. Further, the component for calculating the coordinates of the point B ′ is
Lk, sin δk, cos δk, ωx (k), ωy (k)
It will be.

Of these, cos δk is obtained by using the cosine theorem as shown in the following Expression 15.
[Equation 15]

  Further, sin δk is obtained as shown in the following equation 16 by substituting the equation 15 into the basic formula of the trigonometric function.

[Equation 16]

Therefore, the component for calculating the coordinates of the point B ′ is
d, L1, Lk, ωx (k), ωy (k)
In other words.

［数１８］

Furthermore, if the distance between the point S and the point Pk measured by the distance measuring unit 16 is D (Pk), and the distance between the point S and the point P1 predetermined by design is D (P1), The distance Lk is obtained by the following equation 17, and the distance L1 is obtained by the following equation 18.
[Equation 17]

[Equation 18]

Therefore, the component for calculating the coordinates of the point A is
d, D (Pk), ωx (k), ωy (k)
In other words. Further, the component for calculating the coordinates of the point B ′ is
d, D (P1), D (Pk), ωx (k), ωy (k)
It can be further rephrased as

  D of these is half of the camera interval between the left-eye imaging unit 12 and the right-eye imaging unit 13 (that is, the camera interval is 2d), and is known because it is a fixed value determined by design. It is.

  Further, from the distance measuring unit 16 disposed at the midpoint S between the left-eye imaging unit 12 and the right-eye imaging unit 13, the point P1 where the optical axis of the left-eye imaging unit 12 and the optical axis of the right-eye imaging unit 13 intersect. The distance D (P1) is also known because it is a fixed value determined by design such as the attachment position and angle of the left-eye imaging unit 12 and the right-eye imaging unit 13.

  Further, the distance D (Pk) from the distance measuring unit 16 to the gazing point Pk of the observation object is obtained by measuring by the distance measuring unit 16.

  In addition, the enlargement / reduction parameters ωx (k) and ωy (k) for determining the observation plane that is the display range of the display unit within the imaging region of the imaging unit are determined by manual setting or automatic setting of the electronic zoom. Is done.

  Thus, since all the components for calculating the coordinates of the point B ′ are given, it can be seen that the position of the point B ′ corresponding to the point A of the display pixel can be obtained.

When the electronic zoom is not performed, the enlargement / reduction parameters ωx (k) and ωy (k) are fixed values, and thus represent an angle of view indicating a certain display range. In this case, the remaining components for calculating the coordinates of an arbitrary pixel Aij on the observation plane αk are:
d, D (Pk)
It will be. Also, from this, the remaining components for calculating the coordinates of the point B ′ are
d, D (P1), D (Pk)
It will be. That is, when the display range (observation plane) of the display unit is fixed, if d and D (Pk) are given, the coordinates of the point A can be calculated based on these, and d, If D (P1) and D (Pk) are given, the coordinates of the point B ′ can be calculated.

In calculating the coordinates of the point B ′, the convergence angle ψ between the left-eye imaging unit 12 and the right-eye imaging unit 13 can be used instead of the distance D (P1) described above. FIG. 26 is a diagram showing the relationship between the distance D (P1) and the convergence angle ψ. Referring to FIG. 26, it can be seen that the following formula 19 holds.
[Equation 19]

Therefore, the component for calculating the coordinates of the point B ′ is
d, ψ, D (Pk), ωx (k), ωy (k)
It is also possible to paraphrase that.

  According to the first embodiment, since an image distorted into a trapezoid by projective transformation can be corrected to a rectangle (trapezoid correction), in the vicinity of the left-eye image and the right-eye image when performing stereoscopic viewing. The shift can be reduced, and stereoscopic viewing can be normally performed regardless of the position of the observation object.

  In the above description, a stereoscopic image display device that integrally includes a pair of imaging units, a distance measuring unit, an image processing unit, and a pair of display units has been described as an example. There may be another body as appropriate. For example, a pair of imaging unit and distance measuring unit are installed in a place where an observation object exists, a pair of display units are arranged in a place remote from this place, and these are connected by a communication line etc. In other words, the configuration may be such that stereoscopic observation can be performed. In this configuration, the image processing unit may be in a place where a pair of imaging units and a distance measuring unit are present, may be in a place where a pair of display units are present, or other places other than these However, it may be in a place connected by a communication line or the like. Thus, the stereoscopic image display device of the present invention is not limited in the location of the constituent elements.

  Although the stereoscopic image display apparatus has been mainly described above, a stereoscopic image display method applied to the stereoscopic image display apparatus having the above-described configuration may be used.

  The present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the components without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, you may delete some components from all the components shown by embodiment. Furthermore, the constituent elements over different embodiments may be appropriately combined. Thus, it goes without saying that various modifications and applications are possible without departing from the spirit of the invention.

The figure which shows the stereoscopic vision image display apparatus in Embodiment 1 of this invention from the front side. The figure which shows the stereoscopic vision image display apparatus in the said Embodiment 1 from the right side surface side (left eye side). The figure which shows the stereoscopic vision image display apparatus in the said Embodiment 1 from the back side. In the said Embodiment 1, the figure which shows the structural example which fixed the two imaging parts to the predetermined convergence angle. The figure which shows the mode of the image obtained by fixing the convergence angle of an imaging part according to a close observation object in the said Embodiment 1, and imaging this close observation object by a pair of right and left imaging parts. In the said Embodiment 1, the figure which shows the mode of the image obtained by fixing the convergence angle of an imaging part according to a close observation object, and image | photographing a long distance observation object with a pair of left and right imaging parts. In the said Embodiment 1, the perspective view which shows the imaging | photography range when an imaging | photography part for right eyes image | photographs a close observation object and a long distance observation object with a rectangle, respectively. The figure which shows the shape of the trapezoid distortion which has arisen in the image part centering on an observation object in the picked-up image obtained by imaging the long distance observation object in the said Embodiment 1. FIG. The figure which shows the shape of the trapezoid distortion which has arisen in the image part centering on an observation object in the picked-up image obtained by imaging the long-distance observation object in the said Embodiment 1. FIG. In the said Embodiment 1, the figure which shows a mode when the trapezoid distortion shown in FIG. 8 and the trapezoid distortion shown in FIG. 9 are observed with both eyes. FIG. 3 is a block diagram illustrating a circuit configuration of the stereoscopic image display device according to the first embodiment. In the said Embodiment 1, the figure which shows arrangement | positioning when the imaging part for right eyes, the nearest gazing point P1, and the farthest gazing point P2 are seen from upper direction. The figure which shows arrangement | positioning when the imaging part for right eyes and the arbitrary gazing points Pk between the nearest gazing point P1 and the farthest gazing point P2 are seen from the top in the first embodiment. FIG. 6 is a diagram showing an arrangement when an imaging plane β ′ and an observation plane αk obtained by rotating the imaging plane β about the y axis by an angle δk from above in the first embodiment. In the said Embodiment 1, the top view which shows the positional relationship of imaging | photography plane (beta) whose distance from the point O is Lw, and imaging | photography plane (beta) "which is Lw". In the said Embodiment 1, the perspective view which shows the positional relationship of imaging | photography plane (beta) whose distance from the point O is Lw, and imaging | photography plane (beta) "which is Lw". In the said Embodiment 1, the perspective view which shows the positional relationship of the point O, imaging | photography plane (beta), and observation plane (alpha) k. In the said Embodiment 1, (A) front view, (B) top view, (C) right view which shows the positional relationship of the point O and imaging | photography plane (beta). In the said Embodiment 1, (A) front view which shows the positional relationship of the point O and observation plane (alpha) k, (B) top view, (C) right view. In the said Embodiment 1, the figure which shows an example of the plane chart as an observation thing. In the said Embodiment 1, the perspective view which shows the positional relationship of the point O on the observation plane (alpha) k, and the point B on the imaging | photography plane (beta). FIG. 3 is a diagram showing a pixel configuration on an observation plane αk in the first embodiment. FIG. 3 is a diagram illustrating a pixel configuration on an imaging plane β in the first embodiment. FIG. 6 is a diagram illustrating a state of an imaging plane β ′ obtained by further rotating the imaging plane β on which the observation plane αk is projected by projection around the point O by the angle δk around the y axis in the first embodiment. FIG. 4 is a diagram showing a point B ′ and four pixel positions located in the vicinity thereof in the first embodiment. In the said Embodiment 1, the figure which shows the relationship between distance D (P1) and the convergence angle (psi). The figure which shows a mode that a convergence angle differs when observing the observation object in the vicinity, and the observation object in the distance. The figure which shows a mode that the degree of crossing of an eyeball becomes large when observing the observation thing in the vicinity. The figure which shows a mode that the degree of crossing of an eyeball becomes small when observing the observation object in a distant place. The figure which shows the example of the left-right paired imaging part comprised so that the convergence angle might be adjusted according to the distance to an observation object conventionally.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Stereoscopic image display apparatus 2 ... Main-body part 3 ... Left temple part 4 ... Right temple part 5 ... Frame 6 ... Left eye part 7 ... Right eye part 8 ... Bridge part 11 ... Control part 12 ... Left eye imaging part 13 ... Imaging unit for right eye 14 ... Display unit for left eye 15 ... Display unit for right eye 16 ... Distance measuring unit 21 ... Processor 22 ... Non-volatile memory 23 ... Image processing unit for left eye 24 ... Image processing unit for right eye

Claims (4)

A pair of imaging units fixed in left-right mirror symmetry so as to have a predetermined convergence angle at a predetermined interval;
A distance measuring unit that substantially measures the distance from the midpoint of the pair of imaging units to the observation object;
An image processing unit that performs image processing on each pair of captured images obtained from the pair of imaging units, and generates a pair of display images corresponding to the pair of imaging units,
A pair of display units for displaying the pair of display images generated by the image processing unit on each display screen;
Comprising
The image processing unit includes an interval between the pair of imaging units, a distance from a midpoint of the pair of imaging units substantially measured by the ranging unit to an observation object, and a convergence angle of the pair of imaging units. Or, based on the distance from the intersection of the optical axes of the pair of imaging units to the midpoint of the pair of imaging units, a pair of captured images in a range corresponding to each display screen of the pair of display units, A stereoscopic image display device that generates the pair of display images by correcting a keystone so as to be bilaterally symmetric with respect to a mirror surface that is bilaterally mirror-symmetric. In a space in which the pair of imaging units, the distance measuring unit, and the observation object exist, a pair of observation planes corresponding to the pair of display images, a pair of imaging planes corresponding to the pair of captured images, When each is set,
The image processing unit
The pixel position on the pair of observation planes is obtained based on the interval between the pair of imaging units and the distance from the midpoint of the pair of imaging units substantially measured by the ranging unit to the observation object. ,
The distance between the pair of imaging units, the distance from the midpoint of the pair of imaging units substantially measured by the ranging unit to the observation object, the convergence angle of the pair of imaging units, or the pair of imaging units The position of the point on the pair of imaging planes corresponding to the pixel position on the pair of observation planes based on the distance from the intersection of the optical axes to the midpoint of the pair of imaging units,
The pixel value of the point for which the position is obtained is estimated based on the pixel value of the photographing pixel located in the vicinity of the point for which the position on the pair of imaging planes is obtained, and the estimated pixel value is calculated on the pair of observation planes The stereoscopic image according to claim 1, wherein the trapezoid correction is performed by setting the pixel value of the pixel position to all the pixel positions on the pair of observation planes. Display device. The image processing unit
Processing for causing the pair of display images to be displayed in an enlarged or reduced manner by changing ranges corresponding to the pair of display screens in the pair of captured images based on a value of an enlargement / reduction parameter. And what to do,
In addition to the pixel position on the pair of observation planes, in addition to the distance between the pair of image pickup units and the distance from the midpoint of the pair of image pickup units substantially measured by the distance measuring unit to the observation object The stereoscopic image display apparatus according to claim 2, further obtained based on a value of the enlargement / reduction parameter. A pair of imaging units fixed in a mirror symmetry so as to have a predetermined convergence angle at a predetermined interval; and a distance measuring unit that substantially measures the distance from the midpoint of the pair of imaging units to the observation object; A three-dimensional display comprising a pair of display units for displaying on each display screen a pair of display images corresponding to the pair of imaging units generated based on the pair of captured images obtained from the pair of imaging units. A stereoscopic image display method applied to a visual image display device,
The distance between the pair of imaging units, the distance from the midpoint of the pair of imaging units substantially measured by the ranging unit to the observation object, the convergence angle of the pair of imaging units, or the pair of imaging units Based on the distance from the intersection of the optical axes to the midpoint of the pair of imaging units, a pair of photographed images in a range corresponding to each display screen of the pair of display units is placed on the mirror surface that is symmetrical to the left and right mirror surfaces. A stereoscopic image display method comprising generating the pair of display images by correcting a keystone so as to be symmetrical with respect to the left and right.

Images