JP2012182738A - Stereo image pickup apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a stereo image pickup apparatus that generates stereo images in pan focus with a multifocal lens group.SOLUTION: In a stereo image pickup apparatus 1 having an image pickup element 4 for acquiring photographic images 51, 52, an imaging optical system 2 for directing the light of a first optical path L1 and the light of a second optical path L2 incident at a predetermined reference distance D to the image pickup element 4, and image processing means 6 for processing the photographic images 51, 52 picked up by the image pickup element 4, the imaging optical system 2 comprises a multifocal lens group 2 having at least a first lens part having a first focal length and a second lens part having a second focal length, and the image processing means 6 generates a left eye parallax image 61 and a right eye parallax image 62 from the first photographic image 51 by the light of the first optical path L1 and the second photographic image 52 by the light of the second optical path L2 picked up by the image pickup element 4 via the multifocal lens group 2.

Description

  The present invention relates to a technique for capturing a parallax image for stereoscopic viewing, and more particularly to a stereo image capturing apparatus employing a multifocal lens.

  In recent years, televisions, personal computers, mobile phones, and the like that can three-dimensionally view video content (including moving images and still images) have started to be widely used. The principle of stereoscopic vision is based on human binocular parallax, and these three-dimensional compatible devices generate binocular parallax left-eye and right-eye parallax images (hereinafter also referred to as “stereo images”). When given to the left and right eyes, a human recognizes a stereoscopic image with depth by reconstructing the parallax image as a stereoscopic image in the brain. As an apparatus for capturing a stereo image, a stereo adapter that divides an optical path into two by a mirror or a prism and captures left-eye and right-eye parallax images on one image sensor is known (for example, a patent). Reference 1). There is also a twin-lens imaging device that captures left-eye and right-eye parallax images independently using two imaging optical systems.

  In Patent Document 1, a stereo adapter mounted in front of an imaging lens divides a light beam incident on the imaging lens into two parts so as to have parallax, and each of the two divided light beams is divided into half of one frame by the left eye. The image is taken as an image for use and an image for the right eye.

  FIG. 14 is an explanatory diagram of a stereo image generated by a conventional stereo adapter. When shooting with a normal camera without a stereo adapter attached, an image 100 is obtained in which the length in the height direction of the frame corresponds to the vertical angle of view φ and the length in the width direction corresponds to the horizontal angle of view θ. When the adapter is attached to the camera, a left-eye parallax image 101 and a right-eye parallax image 102 are obtained. When these parallax images are viewed with an appropriate stereoscopic viewer or the naked eye, a stereoscopic image 103 is formed in the human brain.

  By the way, a lens employed in a camera is generally a single focal point, but in the field of vision correction instruments such as eyeglasses and contact lenses, a bifocal lens for both near and near is used. The bifocal lens has, for example, a lens unit having a short focal length suitable for imaging a subject located at a short distance, and a lens unit having a long focal length suitable for imaging a subject located at a long distance. Are integrally formed on the same lens surface. An image pickup apparatus employing such a bifocal lens and an image modification processing method thereof have been proposed (see, for example, Patent Document 2).

  In Patent Document 2, in an imaging apparatus employing a bifocal lens, an image modification process is performed on a captured image in which an image focused on a short-distance subject and an image focused on a long-distance subject overlap. By performing the above, a clear image focused on both the short-distance subject and the long-distance subject is acquired.

JP-A-5-158169 Special table 2008-516299 gazette

  As described above, three-dimensional stereoscopic vision based on stereo images depends on human brain processing. For this reason, in the right and left parallax images, if the parallax range, depth (protrusion, retraction) range, subject focus, and the like are not set appropriately, the formation of stereoscopic vision is hindered, and the viewer experiences significant visual fatigue and discomfort. There is a risk of (sickness). One condition for the viewer to enjoy stereoscopic images comfortably is a pan-focus property that is in focus from the subject that is the object of observation at the short distance (front) to the subject that is the object of observation at the long distance (back). It is necessary to pick up a high parallax image.

  In addition, when an observer views three-dimensional content, the viewpoint may sequentially shift to a subject group at a short distance, a medium distance, and a long distance depending on the scene. Also, there are times when the 3D content producer wants to change the subject to be noticed for each scene. For this reason, when generating a stereo image, it is necessary to consider focusing in a range (intermediate distance) between a short-distance subject and a long-distance subject. Furthermore, it may be necessary in some cases to focus only on a subject group with the observer's viewpoint and blur other subject groups.

  According to the technique described in Patent Literature 1, the optical path incident on the imaging optical system (lens system) of the camera can be divided into two by the stereo adapter, and the parallax image for the left eye and the parallax image for the right eye Can be imaged simultaneously on film. However, with a conventional stereo adapter employing a single focus lens, as shown in images 101 and 102 in FIG. 14, a focused image A is captured for a subject located at the depth of field of the single focus lens. However, for a subject that is out of the depth of field, an out-of-focus image B is captured, and it is difficult to obtain a parallax image with high pan focus. Note that when a single focus lens is used for the stereo adapter, in order to obtain a wide-focused parallax image from a short distance to a long distance, for example, it is conceivable to increase the F value by closing the aperture. However, in this case, the optical path may not be properly separated, and a black band may appear at the central joint between the left and right parallax images. Also, if the F value is increased to obtain a deep depth of field, the amount of incident light will decrease, resulting in underexposure, and a moving image or a moving subject that requires a fast shutter speed will be captured. Not suitable for.

  Further, in Patent Document 1 described above, the area of the imaging surface of the imaging element (corresponding to one frame) is divided into two, and the left-eye parallax image and the right-eye area are divided into the divided left and right areas. A parallax image is captured. For this reason, each parallax image is an image whose horizontal size (horizontal angle of view) is approximately half that of normal (capturing without using a stereo adapter), that is, a vertically long image. That is, as shown in FIG. 14, the left-eye parallax image 101 and the right-eye parallax image 102 have a horizontal angle of view approximately half (θ / 2) compared to the reference normal image 100, and are vertically long images. End up. In general, the viewing angle of a human is wider in the horizontal direction than in the vertical direction, so that a horizontally long image increases the power and presence, but the stereo adapter described in Patent Document 1 has a sense of presence. When a landscape or the like is imaged, a horizontally long (panoramic) parallax image cannot be generated.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a stereo image capturing apparatus capable of generating a stereo image with high pan focus. Furthermore, another object of the present invention is to provide a stereo image capturing apparatus capable of modifying left and right captured images acquired by an image sensor so as to be suitable for stereoscopic vision. It is another object of the present invention to provide a mirror mechanism for optical path division that can acquire a panoramic type parallax image suitable for stereoscopic vision in a stereo image capturing apparatus. In addition, a mirror mechanism suitable for a portable information terminal is provided.

  In order to solve the above-described problem, a stereo image capturing apparatus of the present invention includes an image sensor that acquires a captured image, and light of a first optical path that is incident with a predetermined baseline length that provides binocular parallax for stereoscopic viewing. In a stereo image capturing apparatus, comprising: an imaging optical system that guides light in a second optical path to an imaging surface of the imaging element; and an image processing unit that processes a captured image captured by the imaging element. A multifocal lens group including at least a first lens unit having a first focal length and a second lens unit having a second focal length different from the first focal length, and the image processing means includes: A left-eye parallax image based on a first captured image by the light in the first optical path and a second captured image by the light in the second optical path, which are captured by the image sensor through the multifocal lens group, and right And generating the use parallax images.

  In the stereo image capturing apparatus, a distance from the imaging surface to a position where a point light source positioned on the optical axis of the imaging surface is imaged by the second lens unit for long-distance imaging, and short-distance imaging It is preferable that the ratio of the distance from the imaging surface to the position imaged on the object side by the first lens unit is 100 or less.

  In the stereo image capturing apparatus, the image processing unit modifies the first and second captured images by a modification filter, and a first range corresponding to a depth of field of the first lens unit, Correcting the blur signal component of the second range corresponding to the depth of field of the second lens and the third range farther than the first range or the second range and different from the first and second ranges; It is preferable to generate a parallax image for the left eye and a parallax image for the right eye.

  Further, in the stereo image pickup device, the modification by the modification filter is performed by extracting a signal on the horizontal scanning line in the third range and performing a Fourier analysis to obtain a cumulative value of an alternating current component of a spatial frequency. It is preferable that the left-eye parallax image and the right-eye parallax image are larger than the first and second captured images.

  Further, in the stereo image pickup device, the modification filter is based on a focusing point spread function by light passing through the focusing lens portion and light passing through the other lens portion of the first and second lens portions. It is preferable that the image forming apparatus is configured based on a representative point spread function of the multifocal lens group obtained using a blurred point spread function.

  In the stereo image capturing apparatus, the modification filter may be configured to determine a ratio between the focused point spread function and the blurred point spread function, and an area ratio between the first lens unit and the second lens unit. Are preferably set to different ratios. It is preferable that the point spread function of an ideal lens is applied to at least a part of the point spread function for focusing.

  In the stereo image capturing apparatus, the image processing unit detects a shift amount of a subject in each parallax image by comparing the parallax image for the left eye and the parallax image for the right eye, and the detection It is preferable to identify the relative positional relationship of the subject in the optical axis direction based on the amount of deviation. Further, the image processing means sets a focus area and a blur area for each captured image or each parallax image based on the relative positional relationship of the subject, and sets each of the set It is preferable to perform different image modification processing on the region.

  Further, in the stereo image capturing apparatus, the image processing means is in a selected specific subject area, a subject area that is substantially equidistant from the specific subject, or a distance closer to the specific subject. It is preferable to set the area of the subject to the in-focus area. Further, it is preferable that the specific subject is selected based on information on an observer's viewpoint. In addition, it is preferable that the specific subject is selected based on information on a contour of the subject or information on a designated area input by an observer.

  The stereo image capturing apparatus may further include a mirror mechanism configured to be able to guide the light of the first optical path and the light of the second optical path to the imaging surface of the imaging element via the imaging optical system, respectively. The first lens unit and the second lens unit are divided by a linear boundary line, and are preferably arranged so as to be orthogonal to the optical path division direction by the mirror mechanism.

  In the stereo image capturing apparatus, it is preferable that the mirror mechanism divides light in a vertical direction to form the first optical path and the second optical path. The mirror mechanism includes a first mirror group including inner and outer mirrors for guiding the first optical path to the imaging surface, and inner and outer mirrors for guiding the second optical path to the imaging surface. Each of the inner mirrors is arranged to be stepped from each other in the vertical direction, and each of the inner mirrors is inclined at a first angle with respect to the vertical direction. It is preferable to incline each with a second angle. The first angle is preferably larger than the second angle, and a difference between the first angle and the second angle is preferably ¼ of a vertical field angle corresponding to the imaging surface. Furthermore, it is preferable that the entrance pupil of the imaging optical system is closer to the subject side than the rear end of each outer mirror in the optical axis direction.

  According to the present invention, since the multifocal lens is employed in the stereo image pickup apparatus, it is possible to generate parallax images for the left eye and the right eye that are wide in focus from the front to the back and have high pan focus characteristics. Further, it is possible to execute an image modification process suitable for a parallax image for stereoscopic viewing. Since the amount of shift of each parallax image can be calculated, the approximate front-to-back distance relationship of the subject group can be estimated, and a specific range where the observer's viewpoint can be modified in the stereo image. Furthermore, since a mirror mechanism that can divide the optical path vertically is provided, a panoramic type realistic stereo image can be acquired. Other effects will be described in the mode for carrying out the invention.

1 is a schematic configuration diagram of a stereo image pickup device employing a mirror mechanism according to an embodiment of the present invention. (A) (B) is a figure which shows an example of arrangement | positioning of a bifocal lens and an image pick-up element. Explanatory drawing of the 1st example of a stereo image Illustration of the original image Explanatory drawing showing the state of image modification Explanatory diagram showing how noise is reduced Flow chart showing image playback processing Explanatory drawing of deviation detection processing (A), (B), and (C) are a schematic top view configuration diagram, front configuration diagram, and side configuration diagram of a mirror mechanism according to an embodiment of the present invention (A) (B) is explanatory drawing which shows the inclination of each mirror in a mirror mechanism, and the one part state of the optical path of a perpendicular direction (A) (B) is a figure showing other examples of arrangement of a bifocal lens and an image sensor. Explanatory drawing of the 2nd example of a stereo image 1 is a schematic configuration diagram of a binocular stereo image capturing device according to an embodiment of the present invention. Illustration of stereo images with a conventional stereo adapter

  Embodiments of the present invention will be described below with reference to the drawings. In addition, this invention is not limited to the following examples.

  The stereo image pickup apparatus of the present invention uses the light of the first optical path L1 and the light of the second optical path L2 incident with a predetermined baseline length D for giving binocular parallax, and the left and right binocular parallax with respect to the subject. Is a device that generates a stereo image (a parallax image for the left eye and a parallax image for the right eye), and includes an imaging optical system configured as a multifocal lens group. As a specific stereo image pickup device, a mirror mechanism may be adopted, or a twin-lens type may be used.

  The stereo image capturing apparatus of the present invention can be applied to a camera for capturing three-dimensional content (still image data and moving image data that can be stereoscopically viewed by appropriate means). Since this apparatus employs a multifocal lens in the optical imaging system, an image with high pan focus can be obtained without setting a large aperture value (F value). For this reason, it is possible to image a subject with a relatively high exposure, which is particularly suitable for capturing moving images. Further, the stereo image pickup device of the present invention may be mounted on various information terminals such as a mobile phone, a personal computer, a tablet terminal, and a portable game machine. Note that it is also possible to switch between capturing a two-dimensional image and capturing a three-dimensional image by adopting a configuration in which the mirror mechanism is detachable or only one of the two-lens lenses can be used.

  FIG. 1 is a schematic configuration diagram of an embodiment of a stereo image pickup apparatus employing a mirror mechanism. A stereo image capturing apparatus 1 according to the present embodiment includes an image capturing optical system 2, an image sensor 4, an image processing unit 6, and a mirror mechanism 8. In the present specification, for the sake of explanation, it is assumed that the stereo image pickup device 1 is placed substantially horizontally, and the vertical direction (height direction) is the X direction and the horizontal direction (width direction) with respect to the optical axis L0. ) Is the Y direction, and the direction (depth direction) of the optical axis L0 is the Z direction. The direction of the baseline length D is parallel to the Y direction. The baseline length D is preferably a length close to the human interpupillary distance (about 5 to 6.5 cm) for stereoscopic viewing with less burden, but is shorter depending on the device to which the present invention is applied. Or it is good also as long length (for example, several mm-1 meter).

  The imaging optical system 2 according to the present invention is a multifocal lens group, and forms an image of a subject located at the focusing distance on the objective side on the imaging surface of the imaging element 4. Hereinafter, in the present specification, the imaging optical system 2 may be simply referred to as the multifocal lens group 2 in the description. The multifocal lens group 2 may be composed of a single multifocal lens 20, or one or a plurality of lenses having partially different focal points, such as a semicircular lens and a multifocal lens. The lens group 25 may be a combination of lenses. The multifocal lens group 2 includes at least a first lens unit 21 having a first focal length and a second lens unit 22 having a second focal length on the object side. Hereinafter, for the sake of simplicity, the multifocal lens group 2 will be treated as a single multifocal lens 20, and the multifocal lens 20 will be described as having a first lens portion 21 and a second lens portion 22. The configuration of the multifocal lens group 2 is not limited to this. The first lens unit 21 is suitable for imaging the subject A in the first range on the optical axis L0, and the second lens unit 22 is suitable for imaging the subject B in the second range. . The first range corresponds to the depth of field of the first lens unit 21, and the second range corresponds to the depth of field of the second lens unit 22. In the present specification, an arbitrary region other than the first range and the second range is defined as the third range at a distance farther than the first range or the second range on the optical axis L0. In FIG. 1, an intermediate range between the first range and the second range is defined as a third range.

  For example, the first lens unit 21 may be configured as a short-distance imaging lens having a short focal length, and the second lens unit 22 may be configured as a long-distance imaging lens having a long focal length. The configuration of the multifocal lens is not limited to this, and a short-distance imaging lens and a mid-range imaging lens may be combined, or a middle-distance imaging lens and a long-distance imaging lens are combined. May be. That is, the multifocal lens 20 used in the present invention only needs to have the first lens portion 21 and the second lens portion 22 having different focal lengths. In the following embodiments, the multifocal lens 20 will be described focusing on a bifocal lens having two different focal lengths, but may have three or more different focal lengths. Hereinafter, the multifocal lens 20 may be simply referred to as a bifocal lens 20. A specific configuration of the bifocal lens 20 will be described later (see FIGS. 2 and 11).

  In this embodiment, the imaging optical system (multifocal lens group) 2 is configured as a fixed-focus lens group. However, the imaging optical system (multifocal lens group) 2 can also be configured as a varifocal lens or zoom lens that can change the focal length. Note that the distance ranges corresponding to the first to third ranges also differ depending on the focal length of each lens unit. Although it is preferable that the first range and the second range do not overlap each other, they may partially overlap.

  When the mirror mechanism is employed, the bifocal lens 20 is divided into a region 201 where the light of the first optical path L1 is incident and a region 202 where the second optical path L2 is incident. The second lens unit 22 is arranged so as to be equally divided into each region. That is, the area of the first lens unit 21 in the region 201 is equal to the area of the first lens unit 21 in the region 202, and the area of the second lens unit 22 in the region 201 is equal to the second area in the region 202. It arrange | positions so that the area of the lens part 22 may become equal. For example, in the case of the bifocal lens 20 composed of the semicircular first lens unit 21 and the second lens unit 22, the direction of the boundary between the first lens unit 21 and the second lens unit 22 is the division of light by the mirror mechanism. It arrange | positions so that it may orthogonally cross a direction (refer FIG.2 and FIG.11 (A)). In the case of the bifocal lens 20 in which the first lens portion 21 and the second lens portion 22 are concentrically arranged, the bifocal lens 20 may be divided in any direction as long as it passes through the center of the bifocal lens 20 (FIG. 11B). reference).

  The imaging element 4 is a sensor that converts light incident through the imaging optical system 2 into an electrical signal, and has a light receiving region (imaging surface) 40 having a predetermined size. As the image sensor 4, a CMOS (Complementary Metal-Oxide Semiconductor) sensor, a CCD (CCD: Charge Coupled Device) sensor, or the like can be used. In order to generate left and right parallax images for stereoscopic viewing, the first imaging surface 41 on which the light of the first optical path L1 is incident and the second imaging surface 42 on which the light of the second optical path L2 is incident are formed on the imaging surface 40. Is formed. In the stereo image pickup device employing the mirror mechanism, the first image pickup surface 41 and the second image pickup surface 42 are arranged in parallel in the same direction as the optical path division direction by the mirror mechanism 8. Specific configurations of the first imaging surface 41 and the second imaging surface 42 in the imaging surface 40 will be described later with reference to FIGS. 2 and 11. In FIG. 1, the image pickup surface 40 of one image pickup device 4 is divided into two to provide a first image pickup surface 41 and a second image pickup surface 42. However, the image pickup device on which light in the first optical path L1 is incident Alternatively, an imaging element on which light in the second optical path L2 is incident may be provided, and the imaging surfaces of the imaging elements may be the first imaging surface 41 and the second imaging surface 42.

  The image processing unit 6 acquires the captured images 51 and 52 captured by the imaging element 4 as signals, performs image modification processing on the captured images 51 and 52, and performs the left eye and right eye processing. It is a device that outputs parallax images 61 and 62. As for the hardware configuration of the image processing means 6, for example, as described in FIG. 1 or FIG. 8 of Patent Document 2, a front end for amplifying an analog signal, an AD converter for analog-digital conversion, The image forming apparatus may include a modification filter unit for image modification processing, a correction unit that corrects an image signal, a DA converter for digital-analog conversion, and a control unit that controls each of these components. it can.

  The mirror mechanism 8 divides the light emitted from the short-distance subject A and the long-distance subject B (shown as a point but having a size for the sake of explanation), and the divided light is passed through the imaging optical system 2. This mechanism guides the first imaging surface 41 and the second imaging surface 42 of the imaging device 4. The mirror mechanism 8 includes a first mirror group 81 for guiding the light in the first optical path L1, and a second mirror group 82 for guiding the light in the second optical path L2. The first mirror group 81 includes an inner mirror 81a and an outer mirror 81b, and the second mirror group 82 includes an inner mirror 82a and an outer mirror 82b. As a specific configuration of the mirror mechanism 8, the light is divided in the horizontal direction (left and right) as in the prior art, and the light is divided in the vertical direction (up and down) as will be described later with reference to FIGS. 9 and 10. A method etc. can be used. If the mirror mechanism 8 and the imaging optical system 2 are arranged close to each other, the size can be reduced.

  Further, the left-eye and right-eye parallax images generated by the stereo image capturing device of the present embodiment may be output to a display device (not shown). As the display device, for example, a liquid crystal display, a plasma display, an organic EL display, or the like can be used. If the display device is compatible with three-dimensional display, an observer can view a stereoscopic image with the naked eye. It is also possible to view stereoscopic images using an appropriate viewer (stereoscopic glasses).

  The stereo image capturing apparatus of the present invention can be configured so that the mirror mechanism is detachable, and can be switched between capturing a two-dimensional image and capturing a three-dimensional image. Moreover, it is good also as a structure which can fold a mirror by simple operation.

  Hereinafter, the operation of the stereo image pickup apparatus in which the optical path is divided in the horizontal direction (left and right) will be described in detail with reference to FIGS. 1 to 3.

  FIG. 2 is a diagram illustrating an example of the arrangement of the bifocal lens and the image sensor in the stereo image imaging apparatus in which the optical path is divided in the horizontal direction. When the bifocal lens 20 having the semicircular first lens portion 21 and the semicircular second lens portion 22 is used, the center of the bifocal lens 20 is located on the optical axis, and the first lens portion 21 and The bifocal lens 20 is arranged so that the arrangement of the second lens unit 22 is orthogonal to the optical path division direction by the mirror mechanism 8. That is, as shown in FIG. 2, in the bifocal lens 20, the first lens portion 21 and the second lens portion 22 are vertically arranged so as to be orthogonal to the lateral direction that is the direction of division of the optical path. The imaging element 4 is arranged such that the imaging surface 40 is positioned on the imaging plane of the imaging optical system 2 including the bifocal lens 20 (other lenses are omitted in FIG. 2). For this reason, the image pickup surface 40 of the image pickup device 4 is divided into the left and right in the same way as the optical path division direction, and the first image pickup surface 41 and the second image pickup surface 42 having a horizontal field angle of half (θ / 2) are formed. .

  The distance from the imaging surface 40 to the position P1 where the point light source P0 located on the optical axis L0 of the imaging surface 40 is imaged on the object side by the first lens unit 21 for short-distance imaging is A1, and also for long-distance imaging When the distance from the imaging surface 40 to the position P2 imaged by the second lens unit 22 is A2, the first lens unit 21 and the second lens of the multifocal lens group 2 so that A2 / A1 is 100 or less. It is preferable to design the portion 22. If designed in this way, the focus can be adjusted even in the third range between the first range and the second range by image modification, so that an image with high pan focus property can be obtained. In particular, in order to obtain pan focus to an object at infinity, the second lens unit 22 is designed so that the position P2 imaged by the second lens unit 22 for long-distance imaging becomes the position of the hyperfocal distance. Is preferred. The hyperfocal distance is an object distance when focusing is performed so that the farthest point of the depth of field extends to infinity. In the present embodiment, the imaging optical system (multifocal lens group) 2 is configured as a fixed-focus lens group, but may be configured as a varifocal lens or zoom lens capable of changing the focal length. In this case, if the focal length is changed, the value of A2 / A1 also changes, but it is sufficient to include a configuration in which A2 / A1 is 100 or less.

  In FIG. 2, an example in which two semicircular lenses are combined to form a circular bifocal lens has been described. However, the present invention is not limited to this, and each lens unit is formed by combining semi-elliptical or polygonal lens units. May be arranged so as to be orthogonal to the dividing direction of the optical path, or concentric bifocal lenses (in this case, the arrangement of the lens portions is not particularly limited) may be used.

  A part of the light emitted from the short-distance subject A and the long-distance subject B enters the opening of the first mirror group 81 of the mirror mechanism 8 along the first optical path L1 indicated by a one-dot chain line, and the other part is , And enters the opening of the second mirror group of the mirror mechanism 8 along the second optical path L2 indicated by the two-dot chain line. The light incident from the opening of the first mirror group 81 is reflected by the outer mirror 81 b, is then reflected by the inner mirror 81 a, and enters the imaging optical system 2 including the region 201 of the bifocal lens 20. Then, it converges on the first imaging surface 41 of the imaging device 4 via the imaging optical system 2. Similarly, the light incident from the opening of the second mirror group 82 is reflected by the outer mirror 82b and the inner mirror 82a, and the second imaging of the imaging element 4 is performed by the imaging optical system 2 including the region 202 of the bifocal lens 20. Converge on surface 42. The light converged on the first imaging surface 41 and the second imaging surface 42 is converted into an electrical signal and output to the image processing means 6 as a first captured image 51 and a second captured image 52, respectively.

  FIG. 3 is an explanatory diagram of a stereo image divided in the horizontal direction. When a mirror mechanism that divides the optical path into left and right is used, the imaging surface 40 is equally divided into left and right, and the vertically long first imaging surface 41 and the second imaging that become half the normal horizontal field angle (θ / 2). A surface 42 is formed (see FIG. 2). Therefore, as shown in FIG. 3, the captured images 51 and 52 to be captured have a vertically long shape (horizontal angle of view θ / 2). The light that has passed through the first lens portion 21 and the light that has passed through the second lens portion 22 of the bifocal lens 20 converge on the first imaging surface 41 and the second imaging surface 42. Therefore, the first captured image 51 includes an image component 51a focused on a short distance formed by the first lens unit 21, and a signal component 51b focused on a long distance formed by the second lens unit 22. Is a signal that is superimposed. Similarly, the second captured image 52 is a signal obtained by superimposing the signal component 52a and the signal component 52b.

  Each captured image is subjected to image modification processing by the image processing means 6, and the left-eye parallax image 61 and the right-eye parallax image 62 that are wide in focus from the near subject to the far subject and have high pan-focus properties. It is output as (horizontal angle of view θ / 2). Details of the image modification processing will be described later with reference to FIGS.

  The left-eye parallax image 61 and the right-eye parallax image 62 are displayed by an applicable display device. The observer can recognize these as the stereoscopic image 70 with the naked eye or using an appropriate viewer.

  Hereinafter, the image modification processing of the present invention will be described. In the present invention, since the bifocal lens is employed in the imaging apparatus, as described above, the captured image has a signal component in focus in the first range (short distance) and a focus in the second range (far distance). The matched signal component and the blurred signal component out of focus in any range are included. For this reason, since the synthesized image is unclear as it is, it is necessary to execute an appropriate image modification process. For the image modification processing, the image modification processing method (FIGS. 1, 4, 5, 6, and 11) described in Patent Document 2 can be used. Is preferably used. Note that the stereo image capturing apparatus may be configured to allow the user to select an image modification processing method from among a plurality of image modification processing methods.

  In Patent Document 2, a reforming filter unit composed of a digital filter or the like has a focused image out of a captured image including a focused image component and a focused image component output from an image sensor. It is described that components are removed and a focused image component is extracted. The reforming filter unit includes a reforming filter coefficient calculation unit that calculates a reforming filter coefficient sequence. The reforming filter coefficient calculation unit is based on a multi-focal lens point spread function (hereinafter also simply referred to as “PSF”) for a subject disposed at a predetermined distance from the multi-focal lens (bifocal lens). The inverse transfer function (Fourier transform: FT) or the inverse fast Fourier transform (FFT) is performed on the inverse function of the transfer function, and the coefficient sequence of the inverse function, that is, A reforming filter coefficient sequence is calculated. Since the transfer characteristic due to the cascade connection between the multifocal lens and the reforming filter unit is 1, as a result, the output image passing through the reforming filter unit becomes equal to the subject image, and the out-of-focus blur caused by the multifocal lens is lost. Equivalent to removing the component.

The transfer function (hereinafter referred to as “representative PSF”) of a multifocal lens (hereinafter referred to as “representative PSF”) generally focuses either one of the first and second lens portions constituting the multifocal lens. And the other is a blur lens part, the point spread function by light passing through the focusing lens part (hereinafter referred to as “in-focus PSF”) is “PSF_F”, and the point spread function by light passing through the blur lens part ( (Hereinafter referred to as “blurred PSF”) is “PSF_B”, the amount of light (%) of the focused lens unit is “Φf”, and the amount of light (%) of the blurred lens unit is “Φb”. It is expressed as an equation.
PSF_Double = Φf × PSF_F + Φb × PSF_B (1)

Φf and Φb are expressed as follows, assuming that the amounts of light passing through the in-focus lens portion and the blurred lens portion are “Lf” and “Lb”, respectively, and the total light amount is “L” = Lf + Lb. .
Φf = Lf / L, Φb = Lb / L (2)

  According to the equations (1) and (2), since the total amount of light passing through the bifocal lens is normalized to be 1, the transfer function (representative PSF) “PSF_Double” of the bifocal lens is the amount of light. Do not depend.

Further, in a normal bifocal lens, the first lens portion and the second lens portion are configured to have substantially the same area, so the area ratio of the first lens portion and the second lens portion is 50%: 50%. That is, the ratio of the amount of light passing through each lens unit is also 50%: 50%. That is, as shown in the following formula, the representative PSF includes the in-focus PSF and the blurred PSF at a ratio of 50%: 50%.
PSF_Double = 0.5 × PSF_F + 0.5 × PSF_B (3)

  “PSF_Double” obtained in this way is a transfer function of the multifocal lens, and the imaging apparatus described in Patent Document 2 described above performs inverse FT or inverse FFT on the inverse function of this transfer function, An inverse function coefficient sequence, that is, a reforming filter coefficient sequence was calculated. In the imaging device described in Patent Document 2, when the first lens unit and the second lens unit constituting the bifocal lens are equally semicircular and a point light source is arranged at each focal point, The same focused image and out-of-focus image are projected respectively, and the in-focus PSF and out-of-focus PSF with respect to the focal position of the first lens unit, and the in-focus PSF and out-of-focus with respect to the focal position of the second lens unit are projected. It becomes a function of characteristics close to PSF. Therefore, according to the representative PSF, the signal component of the blurred image formed through either the first lens unit or the second lens unit is modified.

  According to the above-described image modification processing, not only the signal component corresponding to the subject in the first range and the second range is modified, but also the signal component corresponding to the subject in the third range is considerably improved. Has been confirmed by experiments by the applicant of the present application. This is because the amount of information increases by combining the focused signal components in the first and second ranges, and the degree of contrast and color modulation increases in the third range, and the blur signal components are corrected. This is probably because of this.

  However, when a reforming filter coefficient sequence is obtained based on the transfer function (representative PSF) of a bifocal lens using a ratio of 50%: 50% as in the past, the reforming process is executed using the reforming filter coefficient sequence. In some cases, random noise, which is different from the incident light signal, is emphasized and increased due to processing on the image sensor. In particular, when the imaging conditions are bad (for example, when the exposure is low due to insufficient illuminance), the problem of increasing random noise is significant, which adversely affects stereoscopic viewing. In the image modification processing of the present invention, the representative PSF setting of the bifocal lens is changed in consideration of the balance between image sharpness and random noise, and a modification filter suitable for a stereo image for stereoscopic viewing is provided. Constitute. Hereinafter, a reforming filter configured based on the setting of a conventional representative PSF is referred to as a standard reforming filter, and is distinguished from a reforming filter with noise countermeasures.

  The noise countermeasure reforming filter (hereinafter referred to as “first reforming filter”) of the present embodiment is a ratio in which the ratio of the in-focus PSF and the blurred PSF is different from the area ratio of the first lens unit and the second lens unit. It is configured based on the representative PSF set to. In particular, as a noise countermeasure, it is preferable to make the ratio of the in-focus PSF larger than the ratio of the blurred PSF. For example, the representative PSF is determined by setting the in-focus PSF to 60% (Φf = 0.6) and the blurred PSF to 40% (Φb = 0.4). In the standard reforming filter, each coefficient of the 3 × 3 matrix at the center of the reforming filter (vertical v × horizontal h-tap reforming filter 331 in FIG. 5 of Patent Document 2) is, for example, −0.003, − In the present embodiment, the values are set to 0.003, -0.003, -0.003, +2.000, -0.003, -0.003, -0.003, and -0.003. In the noise reduction filter, the corresponding matrix coefficients are -0.0025, -0.0025, -0.0025, -0.0025, +1.6644, -0.0025, -0.0025,- It is set to 0.0025 and -0.0025. According to the first reforming filter, the random noise can be reduced by about 1 dB compared to the standard reforming filter (see FIG. 6).

  Strictly speaking, the lens has aberration, and the light does not gather at one point, so it has a certain spread. That is, the in-focus PSF includes a small blur. Therefore, an improved lens PSF (hereinafter referred to as “ideal PSF”) “PSF_Δ” having no aberration in all or a part of the in-focus PSF can be applied to form a modified filter.

  Therefore, the noise reduction reforming filter (hereinafter referred to as “second reforming filter”) of the present embodiment is configured based on a representative PSF using an ideal PSF for a part or all of the focused PSF. The standard reforming filter is configured to match the performance of the lens using the PSF value calculated during the actual lens design, while the second reforming filter is part of the focused PSF value. Alternatively, the PSF value of the ideal lens was applied to all. For example, the representative PSF was obtained by setting the ratio of the ideal PSF and the ratio of the blurred PSF to 50%: 50%. According to the second reforming filter, the random noise can be reduced by about 3 dB compared to the standard reforming filter (see FIG. 6).

  The first reforming filter and the second reforming filter may be combined. For example, the representative PSF may be obtained by setting the ideal PSF to 40%, the in-focus PSF to 20%, and the blurred PSF amount to 40%, or the ideal PSF to 60% and the blurred PSF amount to 40%. .

  FIG. 4 is a landscape image (hereinafter referred to as “original image”) obtained by imaging from the first range (front, short distance) to the second range (back, long distance) through the third range (intermediate distance) in the park. . The original image in FIG. 4 is obtained by an image (first range (depth of field): 292 mm to 310 mm) picked up by a lens focusing on a distance of about 300 mm suitable for a close subject and a lens focusing on 10 m. This is a composite of a captured image (second range (depth of field): 5 m to far-off). This original image has a feature amount that is substantially the same as an image captured by a bifocal lens that focuses at 300 mm and infinity. The original image has a vertical size of 1148 images and a horizontal size of 1528 pixels. In FIG. 4, the positions of the horizontal scanning lines of the 530th, 620th, 710th, 800th, and 890th rows from the top are shown. The distances to the subject are 2100 mm, 1200 mm, 700 mm, 450 mm, and 350 mm, and are included in the third range (intermediate distance).

  FIG. 5 shows the spatial frequency AC component of the signal of each horizontal scanning line shown in the original image of FIG. 4. The spatial frequency AC component of the original image, the image modification processing by the standard modification filter, and FIG. The result shows the result of the image modification processing by the first modification filter. The horizontal axis in FIG. 5 corresponds to the position of the horizontal scanning line of the original image shown in FIG. 4 and corresponds to the distance to the subject. The vertical axis represents a spatial frequency alternating current component indicating the signal power of each horizontal scanning line. Here, as an index for evaluating the image quality of the image, the alternating component of the spatial frequency was calculated as follows. First, a signal on each horizontal scanning line of the original image is extracted and subjected to Fourier analysis to calculate a spatial frequency that is a superposition of sin signals. Next, the AC component excluding the frequency 0 (DC component) is integrated for the spatial frequency component, and the cumulative value of the AC component is calculated. The accumulated value of the AC component corresponds to the image quality and definition of the signal, and by comparing them, it can be used as an index for evaluating the signal modification level.

  The lowermost line in FIG. 5 is the AC component of the signal of the original image, and the line connecting the square marks is the AC component of the signal that has been subjected to image modification processing by the standard modification filter. Thus, according to the standard reforming filter, the signal can be reformed even for the intermediate distance. That is, according to this image modification process, a blurred signal component is removed, so that a focused signal component (for example, 51a and 51b in FIG. 3) is extracted, and a signal component at an intermediate distance is also corrected. It is possible to generate an image with high pan focus that is widely focused from a short distance to a long distance.

  A line connected with a triangle mark in FIG. 5 is an AC component of the signal subjected to image modification processing by the first modification filter. As shown in FIG. 5, when the ratio of the in-focus PSF to the blurred PSF is 60%: 40%, the signal reforming level slightly decreases. However, random noise was reduced by about 1 dB (see FIG. 6).

  FIG. 6 is a list of image modification results in the image modification process using the standard modification filter and the first and second modification filters. In FIG. 6, an evaluation chart for evaluating the sharpness of an image is used, and this is imaged by a camera using a vertically divided type bifocal lens. The evaluation chart is a chart in which a wedge-shaped pattern in which a plurality of black lines are arranged radially is arranged. In this experiment, as a chart for evaluation, two wedge-shaped patterns (9 lines) are vertically used in the same manner as a test chart (registration chart) used for evaluating television image quality and adjusting distortion of printed matter. The thing of the structure arranged in the direction and the horizontal direction was used. Further, a response function (MTF: Modulation Transfer Function) was used as an index for evaluating the sharpness of the image. Examples of the MTF measurement method include a contrast measurement method (rectangular wave chart method), a Fourier transform method (slit method), and the like. In this experiment, image modification processing using the standard modification filter, the first modification filter, and the second modification filter is performed on the evaluation chart, and the MTF is measured by the contrast measurement method in the evaluation chart after each process. The noise level was calculated based on the evaluation chart before processing. In the contrast measurement method, the difference between the maximum signal value and the minimum signal value due to the black part of the wedge-shaped pattern was calculated, and the difference was compared with the signal value of the white part.

  In the evaluation chart before the image modification process, the MTF is 16% and the noise level is the reference 0 dB. In the image modification process using the standard modification filter, the MTF is improved to 83%, but the noise level is also increased to 5.1 dB compared to the unprocessed evaluation chart.

  In the image reforming process using the first reforming filter (ratio of focused PSF and blurred PSF: 60%: 40%), the MTF was reduced to 78% compared to 83% of the standard reforming filter, but the noise level was Compared with 5.1 of the standard reforming filter, the reduction was about 1 dB. In the image reforming process using the second reforming filter (ratio of ideal PSF and blur PSF: 50%: 50%), the MTF was reduced to 72% compared to 83% of the standard reforming filter, but the noise level was low. Was reduced by about 3 dB compared to 5.1 of the standard reforming filter.

  As described above, according to the image reforming process using the first and second reforming filters, although the reforming level is lower than that of the standard reforming filter, random noise that is not preferable for stereoscopic vision is reduced. Can do.

  As described above, according to the image modification process using the standard modification filter, the image quality can be modified even in a range where each lens unit constituting the bifocal lens is out of focus. It is possible to generate a parallax image with high pan focus suitable for stereoscopic viewing. In addition, by changing the setting of the representative PSF of the bifocal lens and configuring the reforming filter in which the coefficient sequence of the reforming filter is changed, noise can be reduced and a parallax image with less glare can be generated. Can do.

  Furthermore, in the present invention, since the left and right parallax images are acquired, it is possible to modify these images by correlating these parallax images or to change the display of the images. For example, the left and right parallax images can be compared to reduce noise and increase sharpness.

  Since the left and right parallax images have been subjected to image modification processing separately, the nature of the noise is also different. When noise is included only in one of the parallax images, it may adversely affect the viewer's stereoscopic vision, and therefore noise can be identified and removed. Specifically, for example, when a signal that seems to be noise is included in one of the parallax images, the area including the signal is specified, and the other parallax image corresponding to the specified area and the specified area Compare the region with. And if the same signal is not contained in the area | region in the other corresponding parallax image, it can determine with noise and the noise can be removed.

  In addition, by using the amount of deviation between the displayed right and left parallax images, it is possible to specify the observation target of the observer and to specify the relative positional relationship and distance of the subject in the image. Furthermore, the parallax image can be modified and displayed according to the viewpoint of the observer.

  FIG. 7 is a flowchart showing the image display processing of the present invention. First, in step 701, the image processing means 6 acquires the captured images 51 and 52 output from the image sensor 4. For the acquired captured images 51 and 52, the above-described standard modification filter, first Image modification processing is executed using a modification filter, a second modification filter, or the like, and the parallax images 61 and 62 having high pan focus properties are generated. In order to display a normal stereo image, the parallax images 61 and 62 may be displayed using the right-eye display means and the left-eye display means. Steps 702 to 706 relate to the specification of the positional relationship using the shift amount and the image modification process. The image processing means 6 includes a storage unit including a memory for storing various data and programs, a control unit including a processor, etc., and realizes the processing of each step in FIG. 7 by cooperating the hardware and the program. Configured as possible.

  In addition, in step 701 and other steps, the image processing unit 6 may reproduce at least one of the generated parallax images 61 and 62 as a two-dimensional image on the display device on an appropriate display device (not shown).

  In step 702, the image processing means 6 selects a specific subject or a region including the subject in the parallax image. The specific subject is, for example, a subject facing the observer's viewpoint. In this case, the image processing unit 6 may acquire information on the viewpoint of the observer and specify the subject that the observer is observing based on the acquired information on the viewpoint. Information on the viewpoint of the observer is obtained by tracking a movement of the observer's eyes (line of sight, convergence angle, movement, etc.) by a camera (not shown). Further, the image processing means 6 may select this area assuming that the area near the center of the image is a viewpoint, or use a well-known image recognition technique to select a characteristic part (for example, human It is also possible to extract a contour of a face or the like and select a region cut out by the contour.

  In addition, since one of the parallax images is reproduced by the image reproducing unit, the image processing unit 6 acquires information on the subject or the region including the subject specified by the observer in the reproduced parallax image, and acquires the acquired region. Can also be selected as a specific region. In particular, when the stereo image pickup apparatus of the present invention is mounted on an information terminal (for example, a smartphone) provided with a touch panel sensor, it is preferable because the observer can intuitively operate by tracing the contour of the subject that he wants to stereoscopically view. is there. The image processing means 6 may acquire the contour information and select a specific region as it is, or may apply a rough contour region or a designated region such as a rough circle or semicircle traced by the observer. Then, a known image recognition technique may be applied to extract the contour of the subject more accurately.

  In step 703, the image processing means 6 detects the amount of deviation between the parallax images 61 and 62. Hereinafter, a process for detecting the deviation amount will be described.

  FIG. 8 is an explanatory diagram of the shift amount detection process. The left and right parallax images 61 and 62 include a short-distance subject A, a medium-distance subject C, and a long-distance subject B. Since each parallax image is captured with a predetermined baseline length D, between the corresponding subjects, the width direction (Y direction) depends on the distance in the depth direction (Z direction) where the subject is located. Deviation occurs. In step 703 in FIG. 7, the image 63 obtained by superimposing the parallax images can be used to detect the shift amount ΔY1 of the subject A, the shift amount ΔY2 of the subject B, and the shift amount ΔY3 of the subject C by image analysis. The detected shift amount corresponds to the distance from the imaging optical system to the actual position of the subject.

  In addition, as shown in the image 64, the left and right parallax images 61 and 62 are slid so that the contour of the specific subject is overlapped and matched with the specific subject having the observer's viewpoint selected in step 702 as a reference. You may let them. In this case, the degree of modulation of the superimposed specific subject is high, and the degree of modulation is low for subjects in front and back in the Z direction with respect to the specific subject. Therefore, the deviation amount is detected based on the difference in the degree of modulation. You can also

  In step 704 of FIG. 7, the image processing means 6 specifies the relative positional relationship of the subject in the image based on the detected deviation amount. Specifically, the magnitude of the shift amount of each subject is compared, it is determined that the subject with the larger shift amount is closer, the subject with the smaller shift amount is determined to be farther away, and the Z of each subject is determined. A relative positional relationship with respect to the direction can be specified. In addition, when a certain subject with a viewpoint is overlapped, the amount of deviation is 0, and if there is a subject with an amount of deviation of 0 in other regions, it can be determined that the subject is also at the same distance, and there are other amounts of deviation. It can be determined that the subject is either in front of or behind the specific subject.

  In step 705, the image processing unit 6 sets the area including the specific subject selected in step 702 as the in-focus area, and sets the area other than the in-focus area as the out-of-focus area. In this case, not only the specific subject but also a region including a subject equidistant from the specific subject or a region closer to the specific subject may be set as the in-focus region.

  In step 706, the image processing means 6 executes image reproduction processing. For example, in the parallax image after the modification process, the modified pixel data (modified component) is used as it is for the in-focus region, and the image data including the blur in the captured image ( The left and right parallax images are displayed on the display device in a state in which the blur component other than the specific subject with the viewpoint is blurred. In step 705, since the in-focus area and the out-of-focus area are specified, for example, using the captured image before the reforming process, the in-focus area is reformed to generate a parallax image. Then, the generated left and right parallax images may be displayed on the display device. Alternatively, the right and left parallax images may be displayed on the display device by appropriately combining the in-focus area in the parallax image after the modification process and the blurred area in the captured image before the modification process. In other words, it is only necessary to blur the area other than the specific subject using the blur component acquired by the bifocal lens. Accordingly, it is possible to generate a parallax image focused on only a subject or a predetermined area having the viewpoint of the observer and reproduce the left and right parallax images. Further, when the observer's viewpoint is moved, as in steps 702 to 705, the moved subject is set as a new in-focus area, the image in the in-focus area is modified, and the other areas are changed. It is preferable to display a blurred image.

  When the three-dimensional content is a moving image composed of continuous frames, the image processing means 6 determines in step 707 whether or not there is a next frame (parallax image) to be processed, and the next frame is If there is (Yes), the process of step 701 is executed. If there is no next frame (No), the process is terminated.

  Note that the flowchart shown in FIG. 7 is merely an example, and the processing order of each step is not limited to this, and may be appropriately changed as necessary. Depending on the playback mode (video content, still image content), some processes may be omitted or replaced with other processes. For example, in the case of moving image content, the same scene may last for several tens of frames. Therefore, in step 702, once a specific subject is selected, until the scene changes greatly, the identification that is mainly the observation target The subject may be determined not to be changed, and the subsequent step 702 may be omitted. Also, in the video content, since frames are continuously played and the observer's viewpoint must be continuously tracked, the information on the observer's viewpoint used in step 702 is the frame ( Information on the viewpoint of the observer specified in step 706) of the previous loop can also be used. Also, if the observer's viewpoint moves greatly up and down, left and right at short intervals, there is a risk that the observer will get sick of video, so the process of raising a specific subject in step 706 is stopped and played instead. Alternatively, a parallax image after a normal modification process with high pan focus may be reproduced. In this flowchart, the relative positional relationship of the subject is specified based on the shift amount of each parallax image after the reforming process, but triangulation is used based on the baseline length, and the distance from the imaging optical system to the subject is determined. The distance may be estimated. When displaying a three-dimensional moving image, it is preferable not to calculate an actual distance but to estimate only a relative positional relationship using a deviation amount in order to reduce a calculation load.

  Note that the relative positional relationship and distance of the subject may be estimated based on the feature amount of each captured image before the modification process. For example, in steps 701 and 703, the image processing unit 6 uses the post-reformation parallax image with a clear subject outline, but instead of the post-reformation parallax image, the captured image before the remodeling process. A signal is acquired, and an element pixel is cut out from the acquired captured image signal using a method of segmenting an image such as a line drawing extraction method or a finite element method, and an average position (distance) of the element pixel may be calculated. .

  Further, in the present embodiment, it has been described that the image processing unit 6 of the image capturing apparatus generates left and right parallax images that are focused on a specific subject, and the generated left and right parallax images are reproduced on the display device. Each processing shown in FIG. 7 may be realized by a processing unit different from the unit 6, for example, an image processing unit of the image display device. In addition, the image processing unit 6 outputs the left and right parallax images to the display device after generating the left and right parallax images focused on the specific subject, and executes the display processing of the parallax image acquired by the display device. It is good.

  As described above, according to the present invention, since the relative positional relationship of the subject in the image can be estimated from the parallax image or the captured image, the subject at a desired distance is focused or blurred. be able to. In other words, it is possible to perform a process that highlights a certain range of the observer's viewpoint in the parallax image for stereoscopic viewing. Also, by continuously tracking the observer's viewpoint, it is possible to continuously reproduce a parallax image in which the subject to be observed is in focus. Also, it is possible to change the in-focus range according to the scene in order to guide the viewpoint. In particular, in the present invention, the captured image obtained by the multifocal lens has natural blur image information depending on the distance. By using this blur image, uniform blur processing by an image processing application is performed. Unlike, it can express natural blur.

  In the above embodiment, the stereo image pickup apparatus in which the optical path is divided in the horizontal direction has been described. However, the optical path can also be divided in the vertical direction (up and down). Hereinafter, the mirror mechanism of the stereo image pickup apparatus that divides the optical path in the vertical direction will be described with reference to FIGS. 9 and 10, and then the arrangement of the image pickup unit and the bifocal lens will be described with reference to FIG. 11. An image captured using FIG. 12 will be described. Note that the mirror mechanism in FIGS. 9 and 10 can capture a stereo image even using a single focus lens. The various processes described in FIGS. 4 to 8 can be applied to a stereo image pickup apparatus that splits the optical path in the horizontal direction, or a stereo that splits the optical path described below in the vertical direction. The present invention can also be applied to an image pickup apparatus. Further, the present invention can also be applied to a binocular stereo image pickup apparatus shown in FIG.

  9A, 9B, and 9C are a schematic top configuration diagram, front configuration diagram, and side configuration diagram of the mirror mechanism of the present embodiment. FIG. 9A shows a part of the optical path in the horizontal direction. The mirror mechanism 8 includes a first mirror group 81 for guiding the first optical path L1 to the first imaging surface 41, and a second mirror group for guiding the second optical path L2 to the second imaging surface. The first mirror group 81 includes an inner mirror 81a and an outer mirror 81b, and the second mirror group 82 includes an inner mirror 82a and an outer mirror 82b. Each mirror is arranged at an angle with a predetermined angle in the vertical direction (X direction) as shown in FIG. 10 in order to distribute the optical path in the vertical direction. The figure is omitted.

  As shown in FIG. 9A, the first mirror group 81 and the second mirror group 82 are arranged so as to be symmetric with respect to the optical axis of the imaging optical system 2. In each mirror group, the upper end portion of the inner mirror and the upper end portion of the outer mirror are arranged so as to be substantially parallel. As shown in FIG. 9B, at least a part of the first mirror group 81 and the second mirror group 82 are arranged in a vertical difference. As shown in FIG. 9C, the mirror mechanism of the present embodiment is provided between the rear end portions of the inner mirrors 81a and 82b (front side in the Z direction) and the rear end portion (front side of the Z direction) of the outer mirror. It is preferable that the entrance pupil P of the imaging optical system 2 is positioned. That is, the rear end portion of the inner mirror can be retracted and arranged closer to the subject side than the rear end portion of the outer mirror, and the entrance pupil P can be configured as close as possible to the mirror mechanism. Thus, the mirror mechanism can be miniaturized by bringing the entrance pupil P close to the mirror, which is suitable for mounting on a portable information terminal. However, if the entrance pupil P is too close to the mirror, the optical path cannot be separated, and the parallax image may be kicked (a black pixel region having no subject information).

  In the conventional stereo adapter, the horizontal angle of view is halved (θ / 2). However, in the mirror mechanism of the present embodiment, as shown in FIG. A stereo image with a horizontal angle of view θ can be acquired.

  Referring to FIG. 9A, for example, when the light of the optical path L21 (indicated by a two-dot chain line) is incident through the opening of the second mirror mechanism, the light is reflected by the outer mirror 82b and reflected. The reflected light is reflected by the inner mirror 82a and reaches the entrance pupil P. On the other hand, when light in the optical path L22 (shown by a long broken line) is incident through the opening of the second mirror mechanism, it is reflected by the outer mirror 82b, and the reflected light is reflected by the inner mirror 82a, and enters the entrance pupil P. To reach. In this case, the optical path from the entrance pupil P is spread, that is, the horizontal angle of view is θ. For the sake of explanation, the intersection of the optical path L21 and the extension line of the optical path L22 is assumed to be a virtual entrance pupil P ′ (the entrance pupil P is virtually moved). The spread of the optical path from the virtual entrance pupil P ′ is indicated by θ.

  FIG. 10A is an explanatory view showing the inclination of the first mirror group 81 in the mirror mechanism, and is a view of the S1 cross section shown in FIG. FIG. 10B is an explanatory diagram showing the inclination of the second mirror group in the vertical direction, and is a view of the S2 cross section shown in FIG. First, the inner mirror 82a of the second mirror group 82 is disposed so that the lower end of the inner mirror 82a is inclined at an angle α with respect to the vertical direction in a direction approaching the entrance pupil P. Further, the outer mirror 82b is disposed with an upper end inclined at an angle β with respect to the vertical direction in a direction approaching the entrance pupil P. Similarly, the inner mirror 81a and the outer mirror 81b of the first mirror group 81 are also inclined with respect to the vertical direction at angles α and β, respectively.

  FIG. 10B schematically shows a part of the optical path in the vertical direction in the mirror mechanism. In the mirror mechanism of this embodiment, since each mirror is arranged with an inclination, the optical distance differs depending on the optical path, and two reflection points are generated on the upper side and the lower side of each mirror. Specifically, when the light (two-dot chain line) in the upper optical path L21 is incident on the second mirror group 82, the light is first reflected by the upper reflection point 1 of the outer mirror 82b, and then the upper reflection point of the inner mirror 82a. 2 and is incident on the entrance pupil P.

  Further, when the light (long-chain line) in the lower optical path L22 enters the second mirror group 82, it is first reflected by the lower reflection point 3 on the outer mirror 82b, and then the lower reflection point 4 on the inner mirror 82a. And is incident on the entrance pupil P. Each mirror is arranged with a different angle α, β, that is, there is a relative angular difference | α-β |, so that the light in each optical path reflected by the outer and inner mirrors is | α-β | The incident angle changes by 2 times. Therefore, for example, | α−β | = 4 ° is set, the light in the upper optical path L21 is incident at an angle of about + 8 ° with respect to the optical axis L0, and the light in the lower optical path L22 is relative to the optical axis L0. When incident at an angle of about −8 °, when each light is reflected by the outer and inner mirrors, the light in the optical path L21 enters the entrance pupil P at 0 °, and the light in the optical path L22 enters the entrance pupil at −16 °. Incident to P.

  Thus, although the vertical angle of view φ / 2 (for example, 16 °) itself does not change, the range of the vertical angle of view from −8 ° downward to + 8 ° upward is -16 downward by the second mirror group 82. Changed from ° to 0 °. Similarly, the vertical mirror angle range of −8 ° upward to + 8 ° upward is changed from 16 ° to 0 ° upward by the first mirror group 81 arranged at least partially different from the second mirror group 82. Be changed. For the sake of explanation, the intersection of the extension line of the optical path L21 and the extension line of the optical path L22 is defined as a virtual entrance pupil P ′, and the angle of spread of the optical path in the vertical direction from the virtual entrance pupil P ′, that is, vertical The angle of view is indicated as φ / 2.

  FIGS. 11A and 11B are diagrams illustrating an example of the arrangement of the bifocal lens and the image sensor in the stereo image capturing apparatus in which the optical path is divided in the vertical direction. When the bifocal lens 20 having the semicircular first lens portion 21 and the semicircular second lens portion 22 is used, the center of the bifocal lens 20 is located on the optical axis, and the first lens portion 21 and The bifocal lens 20 is arranged so that the arrangement of the second lens unit 22 is orthogonal to the optical path division direction by the mirror mechanism 8. That is, as shown in FIG. 11A, in the bifocal lens 20, the first lens portion 21 and the second lens portion 22 are arranged on the left and right so as to be orthogonal to the longitudinal direction that is the direction of dividing the optical path. Yes. In addition, the imaging element 4 is centered on the optical axis, and the imaging surface 40 of the imaging element 4 is the imaging optical system 2 including the bifocal lens 20 (other lenses are omitted in FIG. 11). It arrange | positions so that it may be located in an image plane. For this reason, the image pickup surface 40 of the image pickup device 4 is divided into the upper and lower sides in the same way as the optical path division direction, and the first image pickup surface 41 and the second image pickup surface 42 whose vertical angle of view is half (φ / 2) are formed. . In FIG. 11A, an example in which two semicircular lenses are combined to form a circular bifocal lens is described. However, the present invention is not limited to this, and each semi-elliptical or polygonal lens unit is combined. The boundary may be arranged so as to be orthogonal to the dividing direction of the optical path.

  As the bifocal lens, a circular first lens portion 21 as shown in FIG. 11B and an annular (donut-shaped) second lens portion 22 are used, and the optical axes of the respective lens portions coincide with each other. If a concentric circular shape is adopted, it is not restricted by the dividing direction of the optical path. Further, a polygonal or elliptical first lens portion 21 may be disposed on the inner side, and an annular second lens portion 22 having an inner edge (and outer edge) of the same shape may be disposed on the outer side. The polygonal corner may have a rounded shape. The arrangement of the first lens unit 21 and the second lens unit 22 may be reversed.

  Further, for example, the light in the second optical path L2 is generated by the first lens unit 21 and the second lens unit by the second mirror group 82 (see FIG. 9) at least a part of which is disposed on the lower side of the mirror mechanism 8. 22 is converged on the second imaging surface 42 via the lower half surface 202 of the bifocal lens 20. Therefore, from the second imaging surface 42, the second captured image 52 includes a signal including an image focused on a short distance through the first lens portion 21 of the lower half surface 201 of the bifocal lens 20, and the second lens portion. Thus, the signal is output as a composite signal of a signal including an image in focus at a long distance through 22.

  In addition, when it is set as the structure which guides the optical path divided | segmented up and down by the mirror mechanism 8, the bifocal lens 20 has the circular 1st lens part 21 and the annular | circular shape (doughnut type) as shown in FIG.11 (B). The second lens unit 22 may be used, and a concentric structure in which the optical axes of the lens units coincide with each other may be used. Further, a polygonal or elliptical first lens portion 21 may be disposed on the inner side, and an annular second lens portion 22 having an inner edge (and outer edge) of the same shape may be disposed on the outer side. The polygonal corner may have a rounded shape. The arrangement of the first lens unit 21 and the second lens unit 22 may be reversed. In the present invention, in order to generate a horizontally long stereo image divided vertically, the mirror mechanism is configured so that the optical path can be divided vertically.

  FIG. 12 is an explanatory diagram of a stereo image divided in the vertical direction. When the mirror mechanism 8 that divides the optical path vertically is used, the imaging surface 40 is divided into upper and lower parts, and the horizontally long first imaging surface 41 and the second imaging surface that are half the normal vertical field angle (φ / 2). An imaging surface 42 is formed (FIG. 11). For this reason, as shown in FIG. 12, the captured images 51 and 52 to be captured have a horizontally long shape (vertical angle of view φ / 2). The first captured image 51 acquired on the first imaging surface 41 and the second captured image 52 acquired on the second imaging surface 42 are subjected to image modification processing by the image processing means 6, so that the left-eye parallax image is obtained. 61 and the right-eye parallax image 62 (vertical angle of view φ / 2) are output. The observer can recognize these as the stereoscopic image 70 with the naked eye or using an appropriate viewer.

  As described above, according to the mirror mechanism of the present embodiment, the optical path is divided up and down, the first optical path L1 is guided to the first imaging surface 41 of the imaging device 4, and the second optical path L2 is directed to the second imaging surface 42. Therefore, a horizontally long stereo image having a horizontal angle of view θ can be formed. In this embodiment, the arrangement of each mirror of the mirror mechanism (position, opening angle in the optical axis direction, inclination angle, etc.) is fixed, but depending on the function of the mounted lens (zoom function, etc.) The mirror arrangement may be variable. In addition, the first mirror group and the second mirror group are arranged so that the imaging range is substantially the same in order to capture a stereo image. For example, the first mirror group and the second mirror group spread in the spreading direction. By arranging in this manner, a mirror mechanism capable of capturing an ultra-wide-angle image can be configured. In this embodiment, two mirror groups having the same horizontal (vertical) angle of view are used. However, mirror groups having different angles of view may be used, or three or more mirror groups may be used. .

  As an example of the mirror mechanism of this embodiment, the following design was performed. Regarding the inner mirrors 81a and 82a in the first and second mirror groups, the horizontal (longitudinal direction) length was 8.0 mm and the vertical (short direction) length was 1.0 mm. Regarding the outer mirrors 81b and 82b, the horizontal (longitudinal direction) length was 32.0 mm, and the vertical (short direction) length was 4.0 mm. In the horizontal direction (Y direction), the interval between the entrance pupil P and the virtual entrance pupil P ′ was set to 15.18 mm. That is, the baseline length D is 30.36 mm. The horizontal angle of view θ is about 50.0 °.

  Further, the inclination angle α of the outer mirror was set to 5.10 °, and the inclination angle β of the inner mirror was set to 1.10 °. | Α−β | was 4 °, and the vertical angle of view φ / 2 was 16 °.

  As mentioned above, although the stereo image imaging device which employ | adopted the mirror mechanism was demonstrated as embodiment of this invention, this invention is not limited to this structure. As described below, the present invention may be configured as a binocular stereo image capturing apparatus as described below.

  FIG. 13 is a schematic configuration diagram of a twin-lens stereo image capturing apparatus according to an embodiment of the present invention. The same components as those of the stereo image pickup apparatus described in FIG. 1 are denoted by the same reference numerals, and detailed description thereof is omitted. As shown in FIG. 13, the binocular stereo image pickup device 10 includes two image pickup optical systems 2, two image pickup devices 4, and image processing means 6. The stereo image pickup device 1 of FIG. 1 has an optical path. The difference is that a mirror mechanism for dividing is not provided. The housing (not shown) is provided with a left opening 85 and a right opening 86 with a predetermined baseline length D. The light in the optical path L1 is incident from the left opening 85 and converges on the first imaging surface 41 of the left imaging element 4 via the left imaging optical system 2. Similarly, the light of the optical path L2 is incident from the right opening 86 and converges on the second imaging surface 42 of the right imaging element 4 via the right imaging optical system 2. On the first and second imaging surfaces, the converged light is converted into an electrical signal and output to the image processing means 6 as the first captured image 51 and the second captured image 52. The first captured image 51 and the second captured image 52 are subjected to image modification processing or the like by the image processing unit 6 and are output as a left-eye parallax image 61 and a right-eye parallax image 62 to a display device or the like (not shown), respectively. The

DESCRIPTION OF SYMBOLS 1 Stereo image imaging device 2 Imaging optical system (multifocal lens group)
4 Image sensor 6 Image processing means 8 Mirror mechanism 20 Multifocal lens (bifocal lens)
21 First lens unit 22 Second lens unit 25 Single focus lens 40 Imaging surface 41 First imaging surface 42 Second imaging surface 51 First captured image 52 Second captured image 61 Left-eye parallax image 62 Right-eye parallax image 81 First mirror group 81a Inner mirror 81b Outer mirror 82 Second mirror group 82a Inner mirror 82b Outer mirror L0 Optical axis L1 First optical path L2 Second optical path

Claims (17)

An imaging device that acquires a captured image, and imaging optics that guides the light in the first optical path and the light in the second optical path that are incident with a predetermined baseline length that gives binocular parallax for stereoscopic viewing to the imaging surface of the imaging device In a stereo image capturing apparatus comprising: a system; and image processing means for processing a captured image captured by the image sensor.
The imaging optical system is configured as a multifocal lens group having at least a first lens unit having a first focal length and a second lens unit having a second focal length different from the first focal length,
The image processing means is based on a first captured image by the light of the first optical path and a second captured image by the light of the second optical path, which are captured by the image sensor through the multifocal lens group. A stereo image capturing apparatus for generating an eye parallax image and a right eye parallax image.   The distance from the imaging surface to the position where the point light source located on the optical axis of the imaging surface is imaged by the second lens unit for long-distance imaging, and the object by the first lens unit for short-distance imaging The stereo image capturing apparatus according to claim 1, wherein a ratio with a distance from the imaging surface to a position formed on the side is 100 or less.   The image processing means modifies the first and second captured images by a modification filter, and a first range corresponding to a depth of field of the first lens unit, a depth of field of the second lens. And a blur signal component in the third range which is farther than the first range or the second range and different from the first and second ranges, and corrects the parallax image for the left eye and the right eye The stereo image imaging device according to claim 1, wherein a parallax image is generated.   In the modification by the modification filter, the accumulated value of the spatial frequency alternating current component obtained by extracting the signal on the horizontal scanning line in the third range and performing Fourier analysis is added to the first and second captured images. The stereo image capturing apparatus according to claim 3, wherein the left-eye parallax image and the right-eye parallax image are larger than each other.   The reforming filter uses a focusing point spread function by the light passing through the focusing lens portion and a blurring point spread function by the light passing through the other lens portion of the first and second lens portions. 5. The stereo image capturing apparatus according to claim 3, wherein the stereo image capturing apparatus is configured based on the representative point spread function of the obtained multifocal lens group.   In the modification filter, the ratio of the in-focus point spread function and the blurred point spread function is set to a ratio different from the area ratio of the first lens unit and the second lens unit. The stereo image pickup device according to claim 5, wherein:   7. The stereo image pickup apparatus according to claim 5, wherein an ideal lens point spread function is applied to at least a part of the focusing point spread function.   The image processing means detects a shift amount of a subject in each parallax image by comparing the parallax image for the left eye and the parallax image for the right eye, and based on the detected shift amount, The stereo image capturing apparatus according to claim 1, wherein a relative positional relationship of the subject in the axial direction is specified.   The image processing means sets an in-focus area and a blurred area for each captured image or each parallax image based on a relative positional relationship of the subject, and sets each of the set areas. The stereo image capturing apparatus according to claim 8, wherein different image modification processing is executed.   The image processing means may select a region of the selected specific subject, a region of a subject that is substantially equidistant from the specific subject, or a region of a subject that is closer to the specific subject than the region of the focus. The stereo image capturing device according to claim 9, wherein   The stereo image capturing apparatus according to claim 10, wherein the specific subject is selected based on information on an observer's viewpoint.   The stereo image capturing apparatus according to claim 10, wherein the specific subject is selected based on information on a contour of the subject or information on a designated area input by an observer. A mirror mechanism configured to be able to guide the light of the first optical path and the light of the second optical path to the imaging surface of the imaging element via the imaging optical system,
13. The first lens unit and the second lens unit are divided by a linear boundary line, and are arranged so as to be orthogonal to a direction in which an optical path is divided by the mirror mechanism. The stereo image imaging device according to any one of the above.   The stereo image capturing apparatus according to claim 13, wherein the mirror mechanism divides light in a vertical direction to form the first optical path and the second optical path. The mirror mechanism includes a first mirror group including inner and outer mirrors for guiding the first optical path to the imaging surface, and a second mirror including inner and outer mirrors for guiding the second optical path to the imaging surface. And having a group
The inner mirrors are arranged to be stepped from each other in the vertical direction, and are inclined at a first angle with respect to the vertical direction,
The stereo image capturing apparatus according to claim 14, wherein each of the outer mirrors is inclined at a second angle with respect to a vertical direction. The first angle is greater than the second angle;
The stereo image imaging apparatus according to claim 15, wherein a difference between the first angle and the second angle is ¼ of a vertical field angle corresponding to the imaging surface.   The stereo image imaging apparatus according to claim 15 or 16, wherein an entrance pupil of the imaging optical system is closer to a subject side than a rear end portion of each outer mirror in an optical axis direction.

Images