JP2015056796A - Stereoscopic image processing apparatus, imaging apparatus, stereoscopic image processing method and stereoscopic image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a parallax image capable of displaying a stereoscopic image that is easy to view without varying greatly a feeling of project and a feeling of depth.SOLUTION: A stereoscopic image processing apparatus 100 acquires an input parallax image and photographing conditions for the image, acquires a distance to a nearest point in a photogenic subject, acquires observation conditions for a stereoscopic image and acquires an acceptable parallax range corresponding to the observation conditions. Horizontal shift amounts of an output parallax image relative to an input parallax image on a display screen and display parameters including display magnification are calculated so that an infinite point of a three-dimensional space reproduced from the output parallax image displayed on a stereoscopic display device matches a limit value at a depth side in the acceptable parallax range and the nearest point matches the limit value at a projection side in the acceptable parallax range. Further the input parallax image is subjected to image processing using the display parameters to generate the output parallax image.

Description

  The present invention relates to a stereoscopic image processing technique for performing image processing for observing a favorable stereoscopic image on a parallax image acquired by stereoscopic imaging.

  In the case where the observer observes a stereoscopic image by presenting parallax images having parallax to the left and right eyes of the observer, the limit of the amount of parallax that can be fused with the observer's brain has been exceeded. When a parallax image having parallax is presented, a double image is recognized instead of a stereoscopic image. In this case, fatigue unique to stereoscopic vision is caused to the observer.

  The fusion limit of a human three-dimensional image will be described with reference to FIGS. FIG. 7A shows a state where the left and right eyes 701 of the observer are simultaneously observing the subject images P1 and P2 displayed on the display screen 702 of the display device. The parallax amounts on the display screen 702 of the subject images P1 and P2 are d1 and d2, respectively, and the pop-out position and the depth position where the subject images P1 and P2 are reproduced are determined by these parallax amounts d1 and d2.

  In a state where the observer gazes at the subject image P1 and fuses it as a stereoscopic image, let P2 be the limit point on the depth side where the subject image at that position can also be fused as a stereoscopic image. It is known that there is a limit range that can be fused in the visual cortex of the brain in the distance between P1 and P2 on the retina, and if the distance between P1 and P2 exceeds the limit range, a double image is recognized. Yes. The fusion limit distance range on the retina is represented by a parallax angle (ab) that is a difference between convergence angles a and b at which the left and right eyes 701 look at the points P1 and P2, and 2 ° for a human eye. It is about ~ 3 °. This range of 2 ° to 3 ° is referred to as an allowable parallax angle capable of fusion.

  In addition, since the focus of the observer's eyes when observing the display device is in alignment with the display screen 702, if the stereoscopic image protrudes excessively from the display screen 702, the convergence direction and the focus direction do not coincide with each other. Fatigue occurs due to factors other than the double image. For this reason, for example, the allowable parallax angle 2 ° is divided into a pop-up-side parallax angle 1 ° and a depth-side parallax angle 1 ° with the display screen 702 as a boundary, and a fusion that balances the pop-up side and the depth side is achieved. It is set as the limit range (allowable parallax range) that can be imaged.

  FIG. 7B shows the relationship between the size of the display screen (horizontal axis) and the allowable parallax amount range d1 to d2 (vertical axis) obtained by converting the allowable parallax angle into the parallax amount d on the display screen. Yes. d1 is a parallax amount for fusing a stereoscopic image popping out from the display screen (hereinafter also referred to as an allowable parallax amount on the pop-out side), and d2 is a parallax amount (hereinafter referred to as depth for fusing a stereoscopic image that is deep on the display screen). Also referred to as an allowable parallax amount on the side). In general, the standard viewing distance varies depending on the size of the display screen. In the case of a high-definition television with a ratio of horizontal dimension to vertical dimension of 16: 9, the distance of 3 times the vertical dimension of the display screen (3H) is the viewing distance. Recommended. When the viewing distance becomes longer as the display screen size increases, the converted parallax amount on the display screen tends to become wider as the display screen size becomes larger as shown in FIG. 7B. When the display screen size is 80 inches or more, the parallax amount d2 on the depth side exceeds the standard interocular distance of 65 mm, and the eyeball moves in the divergent direction, causing fatigue. For this reason, the depth-side parallax amount d2 is limited to be within a standard interocular distance of 65 mm (dmax). If the overall allowable parallax angle is kept constant at 2 °, a margin is generated for the amount of parallax d1 on the protruding side, and the amount of parallax d1 on the protruding side can be increased as shown in FIG. 7B.

  Regarding the allowable parallax amount in such stereoscopic image display, the “3DC safety guideline (3D consortium)” also shows a guideline as “comfortable parallax range”.

  Thus, by suppressing the parallax amount of the parallax image displayed on the display device within the allowable parallax range as much as possible, it is possible to cause the observer to perform a comfortable stereoscopic image observation with less fatigue.

  However, conventionally, when the parallax amount of the displayed parallax image exceeds the allowable parallax range, the user (observer or operator) manually displays the parallax image display condition (see FIG. The amount of horizontal shift, display magnification, etc.) was changed and the stereoscopic effect was adjusted subjectively. For this reason, it is not only a heavy burden for a user who is unfamiliar with the adjustment, but it is difficult to make an appropriate adjustment using the allowable parallax range effectively.

  Japanese Patent Laid-Open No. 2004-26883 describes whether or not the parallax amount of a parallax image to be displayed is within the permissible parallax range when an allowable parallax range capable of stereoscopic viewing is acquired and the stereoscopic image is enlarged or reduced. A stereoscopic image processing apparatus for determining is disclosed. In the apparatus, when it is determined that the parallax amount after enlargement or reduction is not within the allowable parallax range, the parallax amount is automatically adjusted to be within the allowable parallax range. More specifically, when the amount of parallax is not within the allowable parallax range, the parallax image is leveled so that the parallax amount on the side of the pop-up side and the depth side that exceeds the allowable parallax range falls within the allowable parallax range. The amount of parallax is adjusted by shifting in the direction. For example, when it is determined that the depth side is outside the allowable parallax range when the parallax image is displayed, the farthest subject photographed in the parallax image is reproduced at the limit position on the depth side of the allowable parallax range. Adjust the amount of parallax.

JP 2004-349736 A

  However, there are various parallax images in various shooting distance ranges. In some cases, the distance to the farthest point is 4 to 5 m as in indoor imaging, and in the case of taking a landscape photograph. It can be infinitely far away. As described above, when various parallax images having different shooting distance ranges are sequentially displayed, if the parallax amount is adjusted by the stereoscopic image processing apparatus disclosed in Patent Document 1, the following problems occur. That is, in the apparatus of Patent Document 1, the amount of parallax is adjusted so that the farthest subject in each parallax image is reproduced at the limit position on the depth side of the allowable parallax range regardless of the actual depth distance. . For this reason, even when the parallax amount of each parallax image is adjusted so as to be within the allowable parallax range, in the case where observation is performed while sequentially switching a plurality of parallax images after the adjustment, a feeling of popping out or a feeling of depth is generated at each switching. There is a risk that a difficult-to-see stereoscopic image display with a large fluctuation will be performed.

  The present invention provides a stereoscopic image processing apparatus and a stereoscopic image that can generate a parallax image that enables easy-to-see stereoscopic image display without greatly changing the feeling of popping out and the feeling of depth even when parallax images with different shooting distance ranges are sequentially displayed. A processing method and a stereoscopic image processing program are provided.

  The stereoscopic image processing apparatus as one aspect of the present invention performs image processing for displaying on the stereoscopic display apparatus on the input parallax images for the left eye and the right eye acquired by stereoscopic imaging. The stereoscopic image processing apparatus acquires an input parallax image and imaging information acquisition means for acquiring imaging condition information in stereoscopic imaging, and acquires a distance from the imaging condition to the closest point of the subject, or a subject obtained from the input parallax image Nearest point distance obtaining means for calculating a distance to the nearest point using the amount of parallax, and observation condition obtaining means for obtaining observation condition information including the observation distance of the stereoscopic display device and the size of the display screen in the stereoscopic display device; The allowable parallax acquisition means for acquiring the allowable parallax range corresponding to the observation condition, and the infinity point of the stereoscopic space reproduced from the output parallax image displayed on the stereoscopic display device are set to the limit value on the depth side of the allowable parallax range. And the horizontal shift amount on the display screen with respect to the input parallax image and the table so that the nearest point coincides with the limit value on the pop-up side of the allowable parallax range. Characterized in that it has a display parameter calculating means for calculating a display parameters including magnification, the image producing means for producing an output parallax image by performing image processing using the display parameters for the input parallax image.

  An imaging system that performs stereoscopic imaging to generate input parallax images for the left eye and right eye, and the stereoscopic image processing apparatus that performs image processing for displaying the input parallax images on a stereoscopic display device; An imaging device having the above constitutes another aspect of the present invention.

  Also, a stereoscopic image processing method according to another aspect of the present invention is a method of performing image processing for displaying on a stereoscopic display device on input parallax images for the left eye and right eye acquired by stereoscopic imaging. It is. The stereoscopic image processing method acquires input parallax images and imaging condition information in stereoscopic imaging, acquires the distance from the imaging conditions to the closest point of the subject, or uses the parallax amount of the subject obtained from the input parallax image. The distance to the nearest point is calculated, information on the observation condition including the observation distance of the stereoscopic display device and the size of the display screen on the stereoscopic display device is acquired, the allowable parallax range corresponding to the observation condition is acquired, and the stereoscopic display device Output so that the point at infinity in the three-dimensional space reproduced from the output parallax image displayed on the screen matches the limit value on the depth side of the allowable parallax range, and the closest point matches the limit value on the protruding side of the allowable parallax range An image using display parameters for the input parallax image by calculating display parameters including the horizontal shift amount and display magnification on the display screen for the input parallax image of the parallax image And generating an output parallax image by performing sense.

  Furthermore, a stereoscopic image program according to another aspect of the present invention performs, on a computer, image processing for displaying an input parallax image for the left eye and right eye acquired by stereoscopic imaging on a stereoscopic display device. A computer program to be executed. The program obtains input parallax images and information on imaging conditions in stereoscopic imaging, obtains the distance from the imaging conditions to the closest point of the subject, or uses the parallax amount of the subject obtained from the input parallax image A process for calculating a distance to a point, a process for obtaining observation condition information including the observation distance of the stereoscopic display device and the size of the display screen in the stereoscopic display apparatus, and a process for obtaining an allowable parallax range corresponding to the observation condition And the point of infinity in the three-dimensional space reproduced from the output parallax image displayed on the stereoscopic display device matches the limit value on the depth side of the allowable parallax range, and the nearest point becomes the limit value on the pop-up side of the allowable parallax range. A process of calculating display parameters including a horizontal shift amount and a display magnification on the display screen with respect to the input parallax image of the output parallax image so as to match, and the input parallax image Characterized in that to execute a process of generating an output parallax image by performing image processing using the display parameters for the computer.

  In the present invention, the display parameter is calculated so that the infinity point of the three-dimensional space to be reproduced matches the limit value on the depth side of the allowable parallax range, and the closest point in the subject matches the limit value on the pop-out side, Image processing is performed on the input parallax image using the display parameters. For this reason, according to the present invention, it is possible to display an easy-to-see stereoscopic image that maintains a moderate pop-out feeling and has little fluctuation in the depth feeling even when displaying images while sequentially switching parallax images with different shooting distances.

1 is a block diagram illustrating a configuration of a stereoscopic image processing apparatus that is Embodiment 1 of the present invention. FIG. The figure explaining the geometric relationship between the three-dimensional imaging in each Example, and a three-dimensional display. The figure which shows the relationship between the convergence point distance and horizontal shift amount in each Example. 3 is a flowchart illustrating the entire processing of the stereoscopic image processing apparatus according to the first embodiment. 6 is a flowchart illustrating an allowable parallax amount setting process of the stereoscopic image processing apparatus according to the first embodiment. 3 is a flowchart illustrating display parameter calculation processing of the stereoscopic image processing apparatus according to the first embodiment. The figure which shows the parallax angle which can be fused, and the graph which shows the relationship between screen size and the amount of parallax which can be fused. The block diagram which shows the structure of the stereo image processing apparatus which is Example 2 of this invention. 10 is a flowchart illustrating an allowable parallax setting process of the stereoscopic image processing apparatus according to the second embodiment. FIG. 6 is a block diagram illustrating a configuration of an imaging apparatus that is Embodiment 3 of the present invention.

  Embodiments of the present invention will be described below with reference to the drawings.

  First, an outline of matters common to the first and second embodiments of the present invention will be described. The stereoscopic image processing device of each embodiment is for displaying on the stereoscopic display device parallax images for left eye and right eye (hereinafter referred to as input parallax images) acquired by stereoscopic imaging of the imaging space. Perform image processing. The display parameters used for this image processing are set so that the infinity point in the imaging space matches the limit value on the depth side of the allowable parallax range in the stereoscopic display device, that is, the depth amount at the infinity point is maximized. To decide. Moreover, the display parameter is set so that the closest point, which is the position of the closest subject in the imaging space, matches the limit value on the pop-up side of the allowable parallax range in the stereoscopic display device, that is, the pop-out amount of the image of the closest subject is maximum. To be determined. For the determination of the display parameters, information on the observation conditions including the display screen size of the stereoscopic display device, which is the observation conditions, and the observation distance from the observer to the display screen is used.

  If the observation conditions are known in advance, the parallax can be obtained by appropriately adjusting the imaging conditions of the stereoscopic imaging in accordance with the observation conditions, so that a desired stereoscopic effect (amount of popping out or depth) can be obtained when displayed on the stereoscopic display device. Images can be acquired. However, when the observation conditions are not determined in advance and the parallax image has already been acquired by stereoscopic imaging under an arbitrary imaging condition, depending on the actual observation conditions under which the parallax image displayed on the stereoscopic display device is observed The stereoscopic effect given to the observer varies.

  In each embodiment, an observer can obtain a desired stereoscopic effect under actual observation conditions based on the imaging conditions at the time of stereoscopic imaging with respect to an input parallax image that has already been acquired by stereoscopic imaging under arbitrary imaging conditions. A virtual imaging condition for obtaining a virtual parallax image to be given to the subject by stereoscopic imaging is obtained. The virtual imaging condition includes the convergence point distance and the focal length of the imaging optical system. Then, in order to reproduce the stereoscopic effect obtained on the assumption that the virtual parallax image obtained under the virtual imaging conditions is displayed, the output parallax on the display screen corresponding to the convergence point distance and the focal distance at the time of stereoscopic imaging is obtained. Image processing is performed to adjust the horizontal shift amount and display magnification of the input parallax image. That is, by adjusting the horizontal shift amount of the output parallax image on the display screen, an effect equivalent to that of adjusting the convergence point distance during stereoscopic imaging can be obtained. Further, by adjusting the display magnification on the display screen, the same effect as that obtained by adjusting the focal length of the imaging optical system at the time of stereoscopic imaging can be obtained.

  In the image processing performed in each embodiment, as the imaging conditions at the time of stereoscopic imaging, the distance to the closest point that is the position of the subject closest to the imaging device and the convergence that is the distance from the imaging device (imaging optical system) to the convergence point It is assumed that the point distance and the imaging baseline length that is the distance between the left and right imaging optical axes of the imaging optical system are known. Further, as imaging conditions during stereoscopic imaging, the focal length of the imaging optical system and the horizontal width of the imaging surface (the light receiving surface of an imaging element such as a CCD sensor or a CMOS sensor) in the imaging device are also known.

  Note that information on imaging conditions peculiar to stereoscopic imaging is obtained by taking captured images in a format in accordance with the standard “CIPA DC-006-2008 digital still camera stereo still image format” defined by the Camera and Imaging Products Association (CIPA). Is attached as a tag (additional information).

  Next, the geometric relationship between stereoscopic imaging and stereoscopic display (display of a parallax image for observing a stereoscopic image) in each embodiment will be described. FIG. 2A shows a diagram for geometrically explaining stereoscopic imaging, and FIG. 2B shows a diagram for geometrically explaining stereoscopic display.

  In FIG. 2B, reference numeral 205 denotes a display screen of the stereoscopic display device, and Ws denotes the horizontal width of the display screen 205. Ls is an observation distance as a distance from the observer (the left and right eyes 206) to the display screen 205. The observation distance is normally three times (3H) the vertical height of the display screen. The left and right eyes 206 of the observer have an interocular distance of 2 We (standard 65 mm), and the left-eye and right-eye parallax images displayed on the display screen 205 are separated by a separation device such as polarized glasses. Observe. The observer recognizes the stereoscopic image by fusing the left-eye and right-eye parallax images in the brain.

  The allowable parallax range in the stereoscopic display device is determined by the observation distance. First, the maximum projecting position Z01 ′ of the stereoscopic image from the display screen 205 is determined. Then, the maximum depth position Z02 ′ is obtained at the end of the range satisfying an allowable parallax angle (standard value is 2 °) that enables human fusion in the depth direction with reference to the maximum protrusion position Z01 ′. As a result, the allowable parallax range from the maximum protruding position Z01 ′ to the maximum depth position Z02 ′ is determined. The amount of parallax on the display screen 205 corresponding to the maximum pop-out position Z01 ′ is d01. In addition, the amount of parallax on the display screen 205 corresponding to the maximum depth position Z02 ′ is d02.

  In image processing for stereoscopic display, the maximum pop-out position Z01 'and the maximum depth position Z02' determined geometrically as described above are set as the limit value on the pop-out side and the limit value on the depth side of the allowable parallax range, respectively. It may be used. However, in each embodiment, image processing with a predetermined amount of margin with respect to Z01 ′ and Z02 ′, that is, the amount of parallax appropriately suppressed so that as many viewers as possible can view a stereoscopic image comfortably. The limit values on the protruding side and the depth side of the allowable parallax range are set. The limit value (maximum pop-up position) on the pop-up side in this image processing is set to Zmin, and the corresponding parallax amount is set to the allowable parallax amount d1 on the pop-up side. Further, the limit value (maximum depth position) on the depth side is set to Zmax, and the corresponding parallax amount is set to the allowable parallax amount d2 on the depth side.

  In the image processing performed in each embodiment, the limit value on the protruding side and the depth side of the allowable parallax range may be set as a value having a predetermined width instead of a single value. In this case, “coincides with limit values (maximum pop-out position and maximum depth position)” described later means that the value falls within a predetermined width set as the limit value.

  In each embodiment, the maximum pop-out position Zmin and the maximum depth position Zmax corresponding to the observation condition may be calculated each time based on the observation condition information acquired at the time of stereoscopic display. Further, the maximum protrusion position Zmin and the maximum depth position Zmax of the allowable parallax range corresponding to various observation conditions are calculated in advance, and the data is stored in the memory as table data, and the data corresponding to the actual observation conditions is stored in the memory. You may read and use from memory. In either case, this corresponds to obtaining the allowable parallax range (its limit value) from the observation conditions.

  Next, the stereoscopic imaging in each embodiment will be geometrically described with reference to FIG. In each embodiment, the parallax image is acquired by actual stereoscopic imaging under existing imaging conditions (hereinafter referred to as actual imaging conditions). In the following, a process for obtaining a virtual imaging condition such that the parallax amount of the virtual parallax image described above falls within the allowable parallax range in the image processing of the stereoscopic display device, and further obtaining a display parameter of the output parallax image from the virtual imaging condition Will be described.

  Reference numerals 201 and 202 denote left and right imaging lenses (imaging optical systems) of an imaging apparatus used for stereoscopic imaging, and 203 and 204 denote left and right imaging elements (imaging surfaces) provided corresponding to the left and right imaging lenses 201 and 202. ). The baseline length (hereinafter referred to as the camera baseline length), which is the distance between the imaging optical axes of the left and right imaging lenses 201 and 202, is a fixed value of 2Wc, and this camera baseline length 2Wc is the same as the camera baseline length under actual shooting conditions. To do. Point P is a convergence point where the imaging optical axes of the left and right imaging lenses 201 and 202 intersect. Note that the intersection point at which the imaging optical axes of the imaging lenses 201 and 202 intersect with the imaging elements (imaging surfaces) 203 and 204 is the center of the parallax images for the left eye and the right eye. The nearest point distance Z1, which is the distance from the imaging device (imaging lenses 201 and 202) to the nearest point in the range of the subject actually captured by the imaging device (imaging range), is known from the incidental information of the input parallax image. It can be acquired as information. In each embodiment, the farthest point distance Z2 that is the distance to the farthest point in the actual imaging range is not used, and the nearest point and the infinity point (∞) are the maximum pop-out position Zmin and the maximum in the stereoscopic display device. The horizontal shift amount and the display magnification described above are determined so as to coincide with the depth position Zmax.

  The horizontal shift amount can be calculated using a convergence point distance (hereinafter referred to as a virtual convergence point distance) Lc, which is one of the virtual imaging conditions. When a parallax image obtained by stereoscopic imaging of a subject is displayed on the display screen of a stereoscopic display device, the pop-out position and depth position of the reproduced subject stereoscopic image are determined by the amount of parallax of the subject in the parallax image. Further, the parallax amount of the subject is determined by the imaging conditions for stereoscopic imaging and the conditions (display parameters) including the horizontal shift amount and display magnification when displaying the parallax image on the display screen.

  In each embodiment, the infinity point of the imaging space to be stereoscopically captured is displayed by being converged to the maximum depth position Zmax of the stereoscopic space reproduced by the parallax image, and the closest point of the imaging space (position of the closest point distance Z1). Calculates the display parameter displayed at the maximum pop-out position Zmin.

  In other words, the output parallax is such that the infinity point of the stereoscopic space reproduced from the output parallax image displayed on the stereoscopic display device matches the maximum depth position Zmax, and the closest point of the imaging space matches the maximum pop-up position Zmin. Display parameters for the image are calculated. This means that the parallax amount at the infinity point in the imaging space matches the allowable parallax amount d2 corresponding to the maximum depth position Zmax and the parallax amount at the closest point in the imaging space corresponds to the maximum pop-out position Zmin. It can also be said that the display parameter is calculated so as to coincide with the amount d1.

The virtual convergence point distance Lc is calculated by the following equation (1).
Lc = (d2-d1) / d2 · Z1. . . (1)
The horizontal shift amount S on the display screen is calculated by the following equation (2) using the virtual convergence point distance Lc.
S = Ls · (1 / Lc0−1 / Lc) · 2Wc. . . (2)
However, Ls is the above-mentioned observation distance, Lc0 is a convergence point distance (hereinafter referred to as an actual convergence point distance) included in the actual imaging conditions, and 2Wc is the above-described camera base line length.

  FIG. 3 shows the relationship between the convergence point distance and the horizontal shift amount. Reference numerals 301 and 302 denote left and right cameras (imaging apparatuses) respectively configured by the left and right imaging lenses 201 and 202 and the left and right imaging elements 203 and 204 shown in FIG. Reference numeral 303 denotes a virtual imaging surface obtained by projecting the imaging surface of the imaging device of the left camera 301 at the actual convergence point distance Lc0 to the same position as the observation distance. On the virtual imaging surface 303 of the left camera 301, the center of the input parallax image acquired by stereoscopic imaging at the actual convergence point distance Lc0 is represented by a black circle 304. On the other hand, the center of the virtual parallax image acquired by stereoscopic imaging at the virtual convergence point distance Lc is represented by a white circle 305 that is shifted from the black circle 304 in the horizontal direction. That is, the center of the virtual parallax image with respect to the virtual convergence point distance Lc is shifted in the horizontal direction with respect to the center of the input parallax image corresponding to the actual convergence point distance Lc0.

  Therefore, if the input parallax image for the left eye acquired by the left camera 301 is horizontally shifted in the direction of the arrow 306 by S / 2, which is half of the horizontal shift amount S determined as described below, the virtual parallax is obtained. The same parallax image as the left-eye virtual parallax image corresponding to the convergence point distance Lc is obtained. Similarly, if the input parallax image for the right eye acquired by the right camera 302 is horizontally shifted in the direction opposite to the arrow 306 by S / 2, the virtual parallax image for the right eye corresponding to the virtual convergence point distance Lc. The same parallax image is obtained.

  The display magnification is a focal length of the left and right imaging lenses 201 and 202 which is another virtual imaging condition (a value common to the left and right imaging lenses 201 and 202, hereinafter referred to as a virtual lens focal length) f. It can be calculated.

The following relational expression (3) is obtained from the condition that the parallax amount at the infinity point of the imaging space matches the parallax amount d2 corresponding to the maximum depth position Zmax.
d2 = Ws / Wf · 2Wc · f / Lc. . . (3)
Here, Wf is the horizontal width of the subject image formed on the imaging surface of the image sensor (that is, the horizontal width of the imaging surface), and Ws is the horizontal width of the display screen described above. Lc is the virtual convergence point distance Lc obtained by Expression (1).

The camera base length 2Wc and the horizontal width Wf of the subject image formed on the imaging surface are determined as the specifications of the imaging apparatus. The horizontal width Ws of the display screen is also known as display screen size information of the stereoscopic display device used for stereoscopic display.
The virtual lens focal length f is calculated by the following equation (4) obtained by modifying the equation (3).
f = d2 / 2Wc · Lc · Wf / Ws. . . (4)
The display magnification A is calculated by the following equation (5) using the focal length of the imaging lens (hereinafter referred to as the actual lens focal length) f0, which is one of the actual imaging conditions.
A = f / f0. . . (5)
This display magnification A is obtained when an input parallax image is displayed when a virtual parallax image acquired by imaging at the virtual lens focal length f (that is, an output parallax image finally generated) is displayed on the display screen. Is a display parameter for adjustment to obtain the same display size.

  When the horizontal shift amount S on the display screen is calculated from the virtual convergence point distance Lc in this way, in each embodiment, image processing for shifting the left-eye and right-eye input parallax images by S / 2 in the horizontal direction. The horizontal image shift process is performed. When the display magnification A on the display screen is calculated from the virtual lens focal length f, image processing for enlarging or reducing the left-eye and right-eye input parallax images by the display magnification A is performed. As a result, when stereoscopic display is performed, the closest point of the subject matches the limit value on the pop-up side of the allowable parallax range (maximum pop-up position), and the limit point on the depth side of the allowable parallax range (maximum depth position) ) Can be generated.

  FIG. 1 shows the configuration of a stereoscopic image processing apparatus that is Embodiment 1 of the present invention. An observation condition input setting unit (observation condition acquisition means) 101 is a storage medium such as a memory or a hard disk that receives information about observation conditions including the display screen size and the observation distance of a stereoscopic display device that displays parallax images, or is input by a user. To get through. You may acquire observation conditions by communication with the stereoscopic display apparatus connected to the stereoscopic image processing apparatus of a present Example. Normally, the observation distance is a standard value that is three times the vertical height of the display screen (3H). Therefore, only one of the display screen size and the observation distance is acquired, and the other is acquired by calculation. You may do it.

  The observation condition includes the distance between the eyes of the observer. In this embodiment, 65 mm which is a predetermined standard distance between the eyes is used. However, other interocular distance information may be acquired through the observation condition input setting unit 101.

  The allowable parallax setting unit (allowable parallax acquisition unit) 102 calculates the allowable parallax range from the display screen size and the observation distance acquired from the observation condition input setting unit 101. Further, the allowable parallax setting unit 102 calculates allowable parallax amounts d1 and d2 corresponding to the limit values (maximum protruding position Zmin and maximum depth position Zmax) on the protruding side and the depth side of the allowable parallax range.

  An image information input unit (imaging information acquisition unit) 103 receives an input parallax image to be subjected to image processing in the stereoscopic image processing apparatus, that is, a parallax image acquired by stereoscopic imaging by an imaging apparatus such as a digital camera. And read via the above-mentioned storage media. In the input parallax image, additional information is attached as information on imaging conditions (real imaging conditions) at the time of stereoscopic imaging. The additional information is formed on the imaging surface of the imaging element, the distance to the nearest point (nearest point distance Z1), the actual convergence point distance Lc0, the camera base length 2Wc, the actual lens focal length f0, and the subject to be stereoscopically imaged. The horizontal width of the subject image (the horizontal width of the imaging surface) Wf is included.

  If the additional information is not attached to the input parallax image, the additional information may be acquired separately from the input parallax image through communication with the imaging device or a storage medium.

  The nearest point distance obtaining unit 104 obtains the nearest point distance Z1 from the additional information attached to the input parallax image read from the image information input unit 103.

  The display parameter calculation unit 105 displays the horizontal shift amount that is a display parameter for performing image processing on the input parallax image and the display from the closest point distance Z1, the actual shooting condition indicated by the additional information, and the allowable parallax amounts d1 and d2. Calculate the magnification.

  The image processing unit (image generation unit) 106 performs image processing (horizontal shift processing and enlargement / reduction) on the input parallax image using the calculated horizontal shift amount and display magnification, and generates an output parallax image. Then, the output parallax image is output to the stereoscopic display device.

  Next, a flow of a series of processes (stereoscopic image processing method) performed by the stereoscopic image processing apparatus of the present embodiment will be described with reference to the flowcharts of FIGS. 4, 5, and 6. 4 shows the entire process, FIG. 5 shows the allowable parallax amount setting process, and FIG. 6 shows the display parameter calculation process. When the stereoscopic image processing apparatus is configured by a computer, the processing described below is executed by the computer according to a stereoscopic image processing program as a computer program.

  In step S101, the observation condition input setting unit 101 acquires the above-described observation conditions through a user interface (not shown) or via a recording medium, and further through communication with a stereoscopic display device.

  Next, in step S102, the allowable parallax setting unit 102 calculates an allowable parallax range corresponding to the obtained viewing condition, and further, an allowable parallax amount d1 corresponding to the maximum pop-out position Zmin and the maximum depth position Zmax in the allowable parallax range. , D2 is calculated. This allowable parallax amount setting process will be described in more detail and specifically with reference to FIG.

  In step S <b> 201, the allowable parallax setting unit 102 acquires the observation conditions (the horizontal width Ws of the display screen and the observation distance Ls) through the observation condition input setting unit 101.

  Next, in step S202, the allowable parallax setting unit 102 determines the amount of parallax on the display screen from which the pop-up-side parallax angle is −1 ° as the allowable parallax angle (geometry shown in FIG. 2). The parallax amount (dologically determined) d01 is calculated.

  Next, in step S203, the allowable parallax setting unit 102 calculates a parallax amount d02 on the display screen in which the depth-side parallax angle is + 1 ° as the allowable parallax angle from the acquired viewing conditions.

  Next, in step S204, the allowable parallax setting unit 102 determines from the parallax amounts d01 and d02 the maximum pop-out position Zmin having a predetermined margin with respect to the limit value of the allowable parallax range corresponding to the allowable parallax angle ± 1 °. Allowable parallax amounts d1 and d2 corresponding to the maximum depth position Zmax are calculated. Then, the allowable parallax amounts d1 and d2 are output to the display parameter calculation unit 105.

  Returning to the flowchart of FIG. 4, in step S <b> 103, the image information input unit 103 acquires an input parallax image and additional information attached thereto. The contents of the additional information are as described above.

  Next, in step S <b> 104, the closest point distance acquisition unit 104 acquires the closest point distance Z <b> 1 from the additional information acquired from the image information input unit 103. If the additional point information does not include the closest point distance information, a parallax amount map of the input parallax image is created using information on other actual imaging conditions, and the parallax amount of the closest point is obtained from the parallax amount map. The nearest point distance may be calculated from this result.

  Next, in step S105, the display parameter calculation unit 105 acquires information acquired or calculated by the observation condition input setting unit 101, the allowable parallax setting unit 102, the image information input unit 103, and the closest point distance acquisition unit 104. Then, using these pieces of information, a horizontal shift amount S and a display magnification A, which are display parameters for image processing, are calculated. This display parameter calculation process will be described in more detail with reference to FIG.

  First, in step S301, the display parameter calculation unit 105 acquires the observation conditions (Ws, Ls), the allowable parallax amounts d1, d2, the actual shooting conditions (Z1, Lc0, 2We, f, Wf), and the closest point distance Z1.

  Next, in step S302, the display parameter calculation unit 105 calculates the virtual convergence point distance Lc by substituting the closest point distance Z1 and the actual convergence point distance Lc0 into equation (1).

Next, in step S303, the display parameter calculation unit 105 matches the virtual convergence point distance Lc with the observation distance Ls to the display screen, that is, the horizontal shift amount S on the display screen for setting the parallax amount to zero. calculate. Specifically, the horizontal shift amount S is calculated by substituting the observation distance Ls, the actual convergence point distance Lc0, the virtual convergence point distance Lc, and the camera baseline length 2We into Expression (2).

  Next, in step S304, the display parameter calculation unit 105 sets the virtual lens focal length f to the depth-side allowable parallax amount d2, the camera baseline length 2We, the virtual convergence point distance Lc, and the horizontal width of the imaging surface in Expression (4). It is calculated by substituting Wf and the horizontal width Ws of the display screen.

  Next, in step S305, the display parameter calculation unit 105 calculates the display magnification A by substituting the actual lens focal length f0 and the virtual lens focal length f into equation (5).

  Finally, in step S306, the display parameter calculation unit 105 outputs the horizontal shift amount S and the display magnification A obtained by the above processing to the image processing unit 106.

  Returning to FIG. 4, in step S <b> 106, the image processing unit 106 performs horizontal image shift processing on the input parallax image using the horizontal shift amount S, and enlargement or reduction processing of the input parallax image using the display magnification A. To generate an output parallax image.

  In step S107, the image processing unit 106 outputs the output parallax image to the stereoscopic display device.

  As described above, according to the present embodiment, the closest point of the stereoscopically imaged subject is displayed at the maximum protruding position of the allowable parallax range, and the infinity point is displayed at the maximum depth position of the allowable parallax range. . For this reason, the distance range of the entire three-dimensional space can be accommodated in the allowable parallax range while effectively using the allowable parallax range. Even when displaying parallax images with different shooting distances while sequentially switching them, it is possible to display an easy-to-see 3D image with a moderate pop-out feeling and little fluctuation in the depth feeling.

  Next, a stereoscopic image processing apparatus that is Embodiment 2 of the present invention will be described with reference to FIGS.

  FIG. 8 shows the configuration of the stereoscopic image processing apparatus of the present embodiment. Similarly to the stereoscopic image processing apparatus of the first embodiment, the stereoscopic image processing apparatus of the present embodiment also includes an observation condition input setting unit 101, an image information input unit 103, a closest point distance acquisition unit 104, a display parameter calculation unit 105, and image processing. Part 106. Each of these units has the same function as that of the first embodiment and performs the same processing. The stereoscopic image processing apparatus according to the present embodiment is different from the first embodiment in that an adjustment condition input unit (adjustment unit) 807 is added, and accordingly, processing performed by the allowable parallax setting unit 102 ′ is changed.

  The allowable parallax range changes according to the maximum pop-up position that is set, but if the pop-out amount at the maximum pop-out position is too large, the observer's eye adjustment function cannot follow the three-dimensional image pop-out / depth movement. In addition, eye fatigue may increase. In addition, it is preferable that the allowable parallax range can be adjusted by the difference in display screen size and observation distance, and also by individual differences among observers. For this reason, an adjustment condition input unit 807 is provided so that the user can determine the maximum protrusion position Zmin (nearest point display position) and the maximum depth position Zmax (infinity display position) according to other conditions such as the observation condition and the viewer's preference. ) Can be arbitrarily changed (adjustable).

  The flowchart of FIG. 9 shows the flow of allowable parallax amount setting processing in the present embodiment. This allowable parallax amount setting process is executed instead of the allowable parallax amount setting process described in the first embodiment (FIG. 5) according to a switching input by a user who wants to adjust the allowable parallax range.

  First, in step S401, the allowable parallax setting unit 102 ′ acquires the observation conditions (the horizontal width Ws of the display screen and the observation distance Ls) through the observation condition input setting unit 101.

  Next, in step S402, the allowable parallax setting unit 102 ′ acquires the closest point display position and the infinity display position as the adjustment conditions set by the user from the adjustment condition input unit 807.

  Next, in step S403, the allowable parallax setting unit 102 ′ sets the acquired closest point display position as the maximum pop-out position Zmin, and calculates a pop-out allowable parallax amount d1 corresponding to the maximum pop-out position Zmin.

  Next, in step S404, the allowable parallax setting unit 102 ′ sets the acquired infinity display position as the maximum depth position Zmax, and calculates the allowable parallax amount d2 on the depth side corresponding to the maximum depth position Zmax.

  In step S <b> 405, the allowable parallax setting unit 102 ′ outputs the allowable parallax amounts d <b> 1 and d <b> 2 to the display parameter calculation unit 105.

  According to the present embodiment, the maximum pop-out position and the maximum depth position can be adjusted according to observation conditions, individual differences among observers, etc., so that a stereoscopic image display more comfortable for the observer can be performed. The adjustable position may be at least one of the maximum protruding position and the maximum depth position, and it is particularly preferable to set the maximum protruding position that is easily influenced by the observation distance, the individual difference of the observer, and the like.

  FIG. 10 shows a configuration of a stereoscopic imaging apparatus that is Embodiment 3 of the present invention. This stereoscopic imaging apparatus includes an imaging system that generates input parallax images for the left eye and right eye by stereoscopic imaging, and the stereoscopic image processing apparatus 100 according to the first embodiment or the stereoscopic image processing apparatus 800 according to the second embodiment.

  The imaging system includes left and right imaging lenses (imaging optical systems) 201 and 202, and left and right imaging elements 203 and 204 that capture subject images having parallax formed by the left and right imaging lenses 201 and 202, respectively. Further, the imaging system includes a parallax image generation unit 300 that generates input parallax images for the left eye and the right eye using outputs from the imaging elements 203 and 204. The left and right imaging lenses 201 and 202 may be detachable (exchangeable).

  As described above, the stereoscopic image processing apparatuses according to the first and second embodiments may be configured as an apparatus different from the imaging apparatus, or may be provided integrally with the imaging apparatus as in the present embodiment.

  Each embodiment described above is only a representative example, and various modifications and changes can be made to each embodiment in carrying out the present invention.

  A stereoscopic image display apparatus capable of generating a parallax image for performing comfortable stereoscopic image display can be provided.

100,800 Stereoscopic image processing apparatus 101 Observation condition input setting unit 102, 102 'Allowable parallax setting unit 103 Image information input unit 104 Nearest point distance acquisition unit 105 Display parameter calculation unit 106 Image processing unit

Claims (12)

A stereoscopic image processing device that performs image processing for displaying on a stereoscopic display device on input parallax images for the left eye and right eye acquired by stereoscopic imaging,
Imaging information acquisition means for acquiring information of imaging conditions in the input parallax image and the stereoscopic imaging;
A closest point distance acquisition means for acquiring a distance from the imaging condition to the closest point of the subject, or calculating a distance to the closest point using a parallax amount of the subject obtained from the input parallax image;
Observation condition acquisition means for acquiring observation condition information including the observation distance of the stereoscopic display device and the size of the display screen in the stereoscopic display device;
An allowable parallax acquisition means for acquiring an allowable parallax range corresponding to the observation condition;
The point at infinity in the three-dimensional space reproduced from the output parallax image displayed on the stereoscopic display device matches the limit value on the depth side of the allowable parallax range, and the nearest point is the limit on the protruding side of the allowable parallax range Display parameter calculation means for calculating a display parameter including a horizontal shift amount and a display magnification on the display screen with respect to the input parallax image of the output parallax image so as to match a value;
A stereoscopic image processing apparatus comprising: an image generation unit configured to generate the output parallax image by performing the image processing using the display parameter on the input parallax image.   The stereoscopic image processing apparatus according to claim 1, wherein the limit value on the depth side and the limit value on the pop-out side are values each having a predetermined width.   The stereoscopic image processing apparatus according to claim 1, further comprising an adjusting unit that enables the user to adjust each of the limit values.   The display parameter calculation means is configured such that the infinity point of the stereoscopic space reproduced by the observation distance and the horizontal width of the display screen in the stereoscopic display device matches the limit value on the depth side, and the nearest point Finds a virtual imaging condition for acquiring a virtual parallax image displayed on the stereoscopic display device by the stereoscopic imaging so as to match the limit value on the pop-out side, and calculates the display parameter using the virtual imaging condition The three-dimensional image processing apparatus according to any one of claims 1 to 3, wherein The virtual imaging condition includes a convergence point distance and a focal length of an imaging optical system in an imaging apparatus used for the stereoscopic imaging,
The stereoscopic image processing apparatus according to claim 4, wherein the display parameter calculation unit calculates the horizontal shift amount from the convergence point distance, and calculates the display magnification from the focal distance. The display parameter calculation means includes the convergence point distance Lc included in the virtual imaging condition, the parallax amount d1 of the virtual parallax image corresponding to the limit value on the pop-out side, and the limit value on the depth side. Using the parallax amount d2 of the virtual parallax image and the distance Z1 to the nearest point,
Lc = (d2−d1) / d2 · Z1
The three-dimensional image processing apparatus according to claim 5, wherein the calculation is performed by: The display parameter calculation means includes the horizontal shift amount S, the observation distance Ls, a convergence point distance Lc0 included in the imaging condition in the stereoscopic imaging, and the convergence point distance Lc included in the virtual imaging condition. Using the baseline length 2Wc between the left and right imaging optical axes in the imaging optical system,
S = Ls · (1 / Lc0−1 / Lc) · 2Wc
The stereoscopic image processing apparatus according to claim 5, wherein the three-dimensional image processing apparatus is calculated by: The display parameter calculation means calculates the focal length f included in the virtual imaging condition, the parallax amount d2 of the virtual parallax image corresponding to the limit value on the depth side, and the left and right imaging optical axes in the imaging optical system. Baseline length 2Wc, the convergence point distance Lc included in the virtual imaging condition, the horizontal width Wf of the subject image formed on the imaging surface of the imaging device, and the horizontal width of the display screen in the stereoscopic display device Using Ws,
f = d2 / 2Wc · Lc · Wf / Ws
The stereoscopic image processing apparatus according to claim 5, wherein the three-dimensional image processing apparatus calculates the stereoscopic image processing apparatus according to claim 5. The display parameter calculation means uses the display magnification A of the output parallax image by using the focal length f included in the virtual imaging condition and the focal length f0 included in the imaging condition in the stereoscopic imaging.
A = f / f0
The stereoscopic image processing apparatus according to claim 5, wherein the three-dimensional image processing apparatus is calculated by: An imaging system that performs stereoscopic imaging and generates input parallax images for the left and right eyes;
An imaging apparatus comprising: the stereoscopic image processing apparatus according to claim 1, which performs image processing for displaying the input parallax image on a stereoscopic display apparatus. A stereoscopic image processing method for performing image processing for displaying on a stereoscopic display device on input parallax images for the left eye and right eye acquired by stereoscopic imaging,
Obtaining information on the input parallax image and imaging conditions in the stereoscopic imaging;
Obtain the distance from the imaging condition to the closest point in the subject, or calculate the distance to the closest point using the parallax amount of the subject obtained from the input parallax image,
Obtaining observation condition information including the viewing distance of the stereoscopic display device and the size of the display screen in the stereoscopic display device;
Obtaining an allowable parallax range corresponding to the viewing condition;
The point at infinity in the three-dimensional space reproduced from the output parallax image displayed on the stereoscopic display device matches the limit value on the depth side of the allowable parallax range, and the nearest point is the limit on the protruding side of the allowable parallax range A display parameter including a horizontal shift amount and a display magnification on the display screen with respect to the input parallax image of the output parallax image so as to match a value;
A stereoscopic image processing method, wherein the output parallax image is generated by performing the image processing using the display parameter on the input parallax image. A computer program for causing a computer to perform image processing for displaying on a stereoscopic display device on input parallax images for the left eye and right eye acquired by stereoscopic imaging,
Processing for obtaining information of the input parallax image and imaging conditions in the stereoscopic imaging;
A process of acquiring a distance from the imaging condition to the closest point in the subject, or calculating a distance to the closest point using a parallax amount of the subject obtained from the input parallax image;
Processing for obtaining information of observation conditions including an observation distance of the stereoscopic display device and a size of a display screen in the stereoscopic display device;
Processing for obtaining an allowable parallax range corresponding to the observation condition;
The point at infinity in the three-dimensional space reproduced from the output parallax image displayed on the stereoscopic display device matches the limit value on the depth side of the allowable parallax range, and the nearest point is the limit on the protruding side of the allowable parallax range Processing for calculating a display parameter including a horizontal shift amount and a display magnification on the display screen with respect to the input parallax image of the output parallax image so as to match a value;
A stereoscopic image processing program that causes the computer to execute a process of generating the output parallax image by performing the image processing using the display parameter on the input parallax image.