JP5455033B2 - Distance image input device and outside monitoring device - Google Patents

Description

  The present invention relates to a distance image input device and a vehicle exterior monitoring device that obtain parallax information from an image of a subject, detect a distance to the subject from the obtained parallax information, and recognize a distance to a subject in an imaging range.

  An image input device that captures an image of a subject to obtain an image of the subject is useful as an information acquisition unit in a portable device, an in-vehicle device, a medical device, an industrial device, or the like. In addition, there is an increasing demand for a distance image input device that simultaneously detects not only two-dimensional image information of a subject but also distance information such as a distance to the subject and a three-dimensional shape of the subject.

  As a conventional distance image input device for acquiring distance image information, a stereo camera is used as shown in Patent Document 1. In this distance image input device, two or more cameras are installed in a plane substantially perpendicular to the optical axis of the lens, and the distance to the subject and the three-dimensional shape of the subject based on the principle of triangulation from the parallax of the images from these cameras. To get.

  In order to make it possible to install this distance image input device at an arbitrary place, it is desired to reduce the size and thickness of the device. However, as shown in Patent Document 1, when two or more cameras are arranged, there is a limit to downsizing the apparatus, and a distance and shape acquisition error is likely to occur due to the temporal variation of the arrangement. Furthermore, since there are individual differences in the lens groups and image sensors constituting the camera, it is necessary to calibrate them. This calibration of camera arrangement and individual camera difference causes an increase in manufacturing cost and apparatus cost.

  In order to reduce the size of the distance image input device, the distance image input device disclosed in Patent Document 2 uses a plurality of integrally formed fixed-focus optical blocks to reduce the size of the device and ensure brightness. ing.

  In order to obtain accurate distance and image information with the distance image input device, it is essential to obtain a focused image. To obtain a focused image at any distance to the subject, it is necessary to focus the lens of the imaging device to the distance to the predetermined subject, especially in close-up shooting where the distance to the subject is short. Since the depth of field is shallow and the image quality variation due to the change in the distance to the subject increases, the necessity of the focusing function becomes higher. As shown in Patent Document 2, when a subject is imaged with a fixed focus optical block, a focused image may not be obtained depending on the distance to the subject, and it may be difficult to detect the distance and shape.

  In order to focus on the distance to different subjects, the imaging apparatus disclosed in Patent Document 3 uses a liquid lens or a liquid crystal lens as a lens, and changes the focal length of the lens by an input signal indicating a change in focal length. . In addition, the distance image input device disclosed in Patent Document 4 uses a compound eye imaging system having two lens systems having the same focal length and a compound eye imaging system having two lens systems having a longer focal length than the lens system. Focusing on the distances to different subjects, distance distribution information is obtained from near distances in the wide-angle area and far distances in the narrow-angle area.

  The image pickup apparatus disclosed in Patent Document 3 requires a physical movable part as a focus adjustment unit, and thus has a disadvantage in that there is a limit to downsizing and speeding up of the apparatus and an increase in cost. Further, this imaging apparatus does not have a distance detection function.

  The distance image input device disclosed in Patent Document 4 has limitations in miniaturization, and calibration problems associated with the arrangement of lens systems and individual differences in image sensors remain. Further, it cannot detect an object in a wide range at a distant distance, and has a disadvantage that it is not suitable for other applications such as machine vision even if it is sufficient as an application for a vehicle-mounted camera.

  The present invention improves such disadvantages, and can acquire a plurality of stereo image pairs having the same angle of view and focused at different distances to detect a distance image with high distance accuracy and position accuracy. An object of the present invention is to provide a distance image input device and a vehicle exterior monitoring device that can reduce costs.

The distance image input device of the present invention includes an imaging device and an arithmetic processing unit, and the imaging device images a subject and outputs a plurality of stereo images each focused at different distances, and the arithmetic processing unit Includes a parallax detection unit, a parallax synthesis unit, and a distance calculation unit, and the parallax detection unit calculates the parallax based on the unique features of each of the plurality of stereo images output from the imaging device, and obtains the parallax information. A plurality of parallax images, the parallax synthesis unit outputs a synthesized parallax image by synthesizing the plurality of parallax images output from the parallax detection unit, and the distance calculation unit outputs from the parallax synthesis unit calculates the distance from synthetic parallax image to the subject and outputs a distance image, the imaging apparatus, the imaging lens of the plurality sets for obtaining said plurality of stereo images, the imaging element Has the door, said plurality of sets of the imaging lens is characterized in that each other substantially equal between the set focal length of the image pickup element side.
The plurality of sets of imaging lenses have a plurality of apertures corresponding to each imaging lens in a one-to-one correspondence and having substantially equal effective diameters because the lens sag is the same and the curvature radius is different between the sets, and the object side has a different focal length. An aperture array is provided on the subject side of the plurality of sets of imaging lenses.

  The unique feature of calculating the parallax from a plurality of stereo images output from the imaging device by the parallax detection means is a range of distances at which the stereo images are in focus.

  In addition, the parallax detection unit performs a parallax calculation of each stereo image pair in a parallax range corresponding to a range of distances in focus.

  The imaging device forms an image by each of a lens array in which a plurality of lenses are composed of a plurality of stereo lenses each having a different radius of curvature, and the plurality of stereo lenses provided on the image plane side of the lens array. And an imaging element that captures a compound eye image that is a set of images of the subject.

  Further, the plurality of stereo lenses of the imaging apparatus are characterized in that the radius of curvature is changed while keeping the lens sag constant.

  A vehicle exterior monitoring device according to the present invention includes the distance image input device and a display device, and is mounted on a vehicle.

  The present invention outputs a plurality of stereo images each focused on different distances, calculates a parallax based on a unique feature of each of the plurality of output stereo images, and outputs a plurality of parallax images having parallax information Since the distance to the subject is calculated from the synthesized parallax image obtained by synthesizing the plurality of output parallax images, the distance image is output, so that a distance image with high accuracy of the position of the subject can be obtained.

  In addition, since each stereo image is in a distance range where the stereo image is focused as a unique feature for calculating the parallax from a plurality of stereo images, a distance image with high accuracy of the position of the subject and the distance can be obtained.

  Furthermore, since the parallax of each stereo image pair is calculated in the parallax range corresponding to the range of the in-focus distance, it is possible to obtain a range image with high accuracy of the position of the subject.

  In addition, since the imaging apparatus has a lens array in which a plurality of lenses are composed of a plurality of stereo lenses each having a different radius of curvature, and an imaging element that captures a compound eye image of the subject by each of the plurality of stereo lenses, the imaging device has different distances. It is possible to provide a low-cost and thin distance image input device that can be focused and can realize a deep depth of field without a movable part.

  Furthermore, by changing the radius of curvature of the plurality of stereo lenses of the image pickup device while keeping the lens sag constant, the effective diameter of each stereo lens can be made substantially constant, and problems due to differences in the effective lens diameter can be eliminated.

  In addition, by using the distance image input device of the present invention for an out-of-vehicle monitoring device, a stereo image of a short-distance subject can be obtained from a long-distance subject, and highly accurate distance measurement is performed over a wide distance range. be able to.

It is a block diagram of the distance image input device of this invention. FIG. 6 is a layout view of the distance image input device as viewed from the subject direction. It is a change characteristic figure of MTF by a stereo lens. It is a schematic diagram which shows a lens sag and the change of an effective diameter. It is a flowchart which shows the process which acquires a distance image from the image imaged with the distance image input device. It is a figure which shows the imaged compound eye image. It is a flowchart which shows the process which synthesize | combines a 4th parallax image from a 1st parallax image. It is a block diagram of the monitoring apparatus outside a vehicle using a distance image input device.

  FIG. 1 is a block diagram of a distance image input apparatus according to the present invention. The distance image input device 1 includes an imaging device 2 and an arithmetic processing unit 3.

  The imaging device 2 includes a lens array 4, a light shielding wall 5, an aperture array 6, an imaging device 7 mounted on a substrate 8, and a housing 9. The lens array 4 is an even number, for example, the first imaging lens 411, the second imaging lens 412, the third imaging lens 421, and the fourth imaging, as shown in the layout diagrams viewed from the subject direction in FIGS. Eight lenses, a lens 422, a fifth imaging lens 431, a sixth imaging lens 432, a seventh imaging lens 441, and an eighth imaging lens 442 are arranged in an array. The first imaging lens 411 to the eighth imaging lens 442 are composed of two surfaces, a lens surface 40a on the object side and a lens surface 40b on the image surface side, and the lens surface 40a and the lens surface 40b are set as an image of the object. Is imaged on the image plane. The light shielding wall 5 is made of a material opaque to imaging light such as metal or resin, and forms a light shielding partition for preventing crosstalk (crosstalk) of light between adjacent imaging lenses in the lens array 4. As shown in FIG. 2, rectangular holes are formed corresponding to the first imaging lens 411 to the eighth imaging lens 442 of the lens array 4, and the wall between the holes is Acts as a septum. The light shielding wall 5 is bonded to the image side surface of the lens array 4. The aperture array 6 is provided with circular holes 6 a corresponding to the first imaging lens 411 to the eighth imaging lens 442 in the plate member, and the apertures of the first imaging lens 411 to the eighth imaging lens 442 are provided. Acts as The aperture array 6 is bonded to the lens array 4 via a protrusion 4a provided on a plane portion of the surface of the lens array 4 on the subject side. The image sensor 7 is composed of, for example, a CMOS sensor, and outputs image data of a subject to be imaged by entering light that has passed from the first imaging lens 411 through the eighth imaging lens 442. The housing 9 is bonded to the surface of the lens array 4 on the object side and bonded to the surface of the substrate 5 on the image sensor 7 side while holding the lens array 4, the light shielding wall 5, and the aperture array 6. ing.

  The lens array 4 of the imaging device 2 will be described. The first imaging lens 411 to the eighth imaging lens 442 have lens surfaces 40a and 40b having different radii of curvature, and the radii of curvature are set so as to focus on different object distances. The first imaging lens 411 and the second imaging lens 412 constituting the lens array 4 have a lens surface 40a and a lens surface 40b having substantially the same radius of curvature, and the first imaging lens 411 and the second imaging lens. 412 is paired to form a first stereo camera lens, and together with the pixel area of the image sensor 7 corresponding to the first imaging lens 411 and the second imaging lens 412, the first stereo camera is configured. Yes. The third imaging lens 421 and the fourth imaging lens 422 also have a lens surface 40a and a lens surface 40b having substantially the same radius of curvature, and form a second stereo camera lens as a pair, and the corresponding imaging device 7 The fifth imaging lens 431 and the sixth imaging lens 432 also have a lens surface 40a and a lens surface 40b having substantially the same radius of curvature, and form a pair. A third stereo camera lens is configured, and a third stereo camera is configured together with the corresponding pixel region of the image sensor 7. The seventh imaging lens 441 and the eighth imaging lens 442 also have substantially the same radius of curvature. It has lens surfaces 40a and 40b and forms a fourth stereo camera lens as a pair, and together with the corresponding pixel region of the image sensor 7, it forms a fourth stereo camera.

  As shown in FIG. 2, the first imaging lens 411 and the second imaging lens 412 that constitute the first stereo lens of the lens array 4 are the same as the fifth imaging lens 431 that constitutes the third stereo lens. The third imaging lens 421 and the fourth imaging lens 422 that constitute the second stereo lens are arranged in a line at equal intervals D1 on both sides of the sixth imaging lens 432, and the fourth imaging lens 422 constitutes the fourth stereo lens. The seventh imaging lens 441 and the eighth imaging lens 442 are arranged in one row at equal intervals D1, and the first imaging lens 411 and the third imaging lens 421 arranged in one row also have the interval D1. They are spaced apart. The distance D2 = 3 * D1 between the first imaging lens 411 and the second imaging lens 412 is the baseline length of the first stereo camera, and the distance D2 between the third imaging lens 421 and the fourth imaging lens 422 is the first. And the distance D1 between the fifth imaging lens 431 and the sixth imaging lens 432 becomes the baseline length of the third stereo camera, and the seventh imaging lens 441 and the eighth imaging lens 442 The interval D1 is the baseline length of the fourth stereo camera. The first stereo camera constituted by the first imaging lens 411 and the second imaging lens 412 having the baseline length D2 and the second stereo camera constituted by the third imaging lens 421 and the fourth imaging lens 422 are far away. A fourth stereo composed of a third stereo camera composed of a fifth imaging lens 431 and a sixth imaging lens 432 having a baseline length D1, and a seventh imaging lens 441 and an eighth imaging lens 442. The camera is for short distance use.

  FIG. 3 shows an example of through focus MTF (Modulation Transfer Function) by the first to fourth stereo lenses constituting the lens array 4. In FIG. 3, A4, A3, A2, and A1 represent through-focus MTF curves corresponding to each stereo lens (in this case, MTF change according to subject distance). The subject distance at the peak is different. When the subject distance is shortened on the order of several millimeters, the depth of field becomes very shallow, and a slight defocus causes a large MTF to decrease and blur the image. As the subject distance increases, the depth of field increases, and the through focus MTF curve A1 does not blur the image to infinity beyond a certain distance. Therefore, as shown in FIG. 3, the fourth stereo lens is designed from the first stereo lens of the lens array 4 so that the through-focus MTF curves A1 to A4 intersect at a certain level threshold TH, and according to the subject distance. By selecting a reduced image (hereinafter referred to as a single-eye image) or pixels of a subject using a stereo lens having an appropriate MTF, the subject can be focused from a close range of several millimeters to infinity. The subject distances where the through focus MTF curves A1 to A4 intersect the threshold value TH are K, L, M, and N, respectively.

  The first imaging lens 411 to the eighth imaging lens 442 are designed so that the focal lengths are substantially equal in addition to the setting of the radius of curvature. Accordingly, since the optical magnification of the first imaging lens 411 to the eighth imaging lens 442 is approximately defined only by the subject distance, the image from the first imaging lens 411 to the eighth imaging lens 442 is in focus (blur). The size of the image is constant even if the levels are different from each other. Therefore, when selecting only the focused single-eye image from the compound eye image acquired by the imaging device 2 or extracting only the focused pixel, the first imaging lens 411 to the eighth eye It is not necessary to consider the difference in image size due to the imaging lens 442, and it is not necessary to consider the difference in image size even when some information recognition is performed using the acquired image.

  FIGS. 4A to 4C show how the lens sag and the lens effective diameter change when the curvature of the lens is changed while keeping the focal length of the lens substantially constant. As shown in FIG. 4A, when the curvature of the lens surface 40a and the lens surface 40b is changed from the state of the lens sag ta, the effective diameter da of the lens surface 40a, the lens sag tb of the lens surface 40b, and the effective diameter db, the effective diameter is changed. When da and db are not changed, it is necessary to change the lens sag ta and tb of the lens surface 40a and the lens surface 40b to lens sag ta1 and tb1, as shown in FIG. If the lens sag is changed in this way, the shape becomes complicated, such as providing a height difference based on the difference in lens sag between the lenses. On the other hand, when the lens sag ta and tb are not changed, it is necessary to change the effective diameters da and da to the effective diameters da1 and db1 as shown in FIG. Changing the effective diameter in this way changes the optical characteristics due to the effective diameter between lenses, for example, the cutoff frequency and brightness of the lens.Therefore, it is necessary to adjust the conditions after obtaining the image of the subject. It becomes complicated.

  Therefore, the lens array 4 changes the radius of curvature while keeping the lens sag of the first imaging lens 411 to the eighth imaging lens 442 constant, and provides an aperture array 6 having substantially the same effective diameter on the subject side of the lens array 4. In addition, the effective diameters of the first imaging lens 411 to the eighth imaging lens 442 are made substantially constant, so that problems caused by the difference in effective lens diameter are eliminated. Although the first imaging lens 411 to the eighth imaging lens 442 and the aperture array 6 of the lens array 4 are in contact with each other depending on the first imaging lens 411 to the eighth imaging lens 442, the lens array 4 The lens array 4 and the aperture array 6 are brought into contact with each other by the projection 4c provided on the flat surface portion, thereby eliminating problems such as the aperture array 6 being inclined and adhered to the lens array 4.

  When the imaging device 2 is observed from the subject direction, as shown in FIG. 2, the circles indicated by solid lines in the first imaging lens 411 to the eighth imaging lens 442 are circular holes (effective diameters) 6 a of the aperture array 6. And its diameter is constant. A circle indicated by a broken line represents the diameter of the lens surface 40a including the invalid area, and a circle indicated by a dotted line represents the diameter of the lens surface 40b. Since the radius of curvature is different from the first stereo lens to the fourth stereo lens and the lens sag of the first imaging lens 411 to the eighth imaging lens 442 is constant, the first imaging lens 411 The diameters of the circles indicated by the broken line and the dotted line in the eighth imaging lens 442 are different.

  The arithmetic processing unit 3 includes an image capture unit 11, an image correction unit 12, four sets of parallax detection units 13 a to 13 d corresponding to the first to fourth stereo cameras, a parallax synthesis unit 14, and a distance calculation unit 15. . The image capture unit 11 converts the compound eye image input from the image sensor 7 of the imaging device 2 into a first single-eye image 511 by light passing through the first imaging lens 411 and a first eye by light passing through the second imaging lens 412. The second single-eye image 512, the third single-eye image 521 by light passing through the third imaging lens 421, the fourth single-eye image 522 by light passing through the fourth imaging lens 422, and the fifth A fifth single-eye image 531 by light passing through the imaging lens 431, a sixth single-eye image 532 by light passing through the sixth imaging lens 432, and a first single-eye image 532 by light passing through the seventh imaging lens 441. 7 single-eye images 541 and an eighth single-eye image 542 by light that has passed through the eighth imaging lens 442 are output separately. The image correction unit 12 performs distortion correction on the first single-eye image 511 to the eighth single-eye image 542 input from the image capture unit 11 and outputs the result. The parallax detection unit 13a detects the parallax from the first single-eye image 511 and the second single-eye image 512 input from the image correction unit 12, obtains and outputs the first parallax image 51, and the parallax detection unit 13b The parallax is detected from the third single-eye image 521 and the fourth single-eye image 522 to obtain and output the second parallax image 52, and the parallax detection unit 13 c has the fifth single-eye image 531 and the sixth single-eye image 531. The parallax is detected from the eye image 532 to obtain and output the third parallax image 53, and the parallax detection unit 13d detects the parallax from the seventh single-eye image 541 and the eighth single-eye image 542, and outputs the fourth parallax. A parallax image 54 is obtained and output. The parallax synthesis unit 14 synthesizes the fourth parallax image 54 from the first parallax image 51 input from the parallax detection units 13 a to 13 d and outputs a parallax synthesis image. The distance calculation unit 15 calculates a subject distance from the parallax composite image input from the parallax synthesis unit 14 and outputs a distance image.

  Next, a process for obtaining a distance image from an image captured by the imaging device 2 of the distance image input device 1 will be described with reference to a flowchart of FIG.

  The light passing through the eighth imaging lens 442 from the first imaging lens 411 of the imaging device 2 is incident to acquire an image of the subject with the imaging device 7, and the acquired compound eye image 50 is obtained as an image capture unit of the arithmetic processing unit 3. 11 (step S1). As shown in FIG. 6, the image acquired by the imaging device 7 includes a first single-eye image 511, a second single-eye image 512, and a first single-eye image 512 corresponding to the first imaging lens 411 to the eighth imaging lens 442. 3 eye images 521, 4 eye images 522, 5 eye images 531, 6 eye images 532, 7 eye images 541, and 8 eye images 542. 50. In this case, a plane photograph is taken and the subject is at a certain distance. In FIG. 6, a black area around the first individual image 511 to the eighth individual image 542 is an area shaded by the light shielding wall 5. Since the focal lengths of the first imaging lens 411 to the eighth imaging lens 442 are substantially equal, the imaging ranges of the first single-eye image 511 to the eighth single-eye image 542 are substantially equal. On the other hand, since the subject distance in focus differs from the first single-eye image 511 to the eighth single-eye image 542 depending on the radius of curvature, the in-focus state of the first single-eye image 511 to the eighth single-eye image 542 is , A pair of the first imaging lens 411 and the second imaging lens 412, a pair of the third imaging lens 421 and the fourth imaging lens 422, and a pair of the fifth imaging lens 431 and the sixth imaging lens 432 The pair of the seventh imaging lens 441 and the eighth imaging lens 442 is different for each pair. Then, the first single-eye image 511 and the second single-eye image 512 captured by the pair of the first imaging lens 411 and the second imaging lens 412 are in focus. When photographing a three-dimensional object, a portion that is in focus and a portion that is not in focus are mixed in the first eye image 511 to the eighth eye image 542.

  The image capture unit 11 separates the compound eye image 50 input from the image sensor 7 from the first single-eye image 511 to the eighth single-eye image 542, and separates the first single-eye image 511 to the eighth single-eye image. The image 542 is output to the image correction unit 12 (steps S2 and S3). The image correction unit 12 corrects the distortion of the input first eye image 511 to the eighth eye image 542 using internal parameters, external parameters, etc. specific to the camera according to the working accuracy and assembly accuracy of the lens. The pair of the corrected first single-eye image 511 and the second single-eye image 512 is output to the parallax detection unit 13a, and the pair of the third single-eye image 521 and the fourth single-eye image 522 is detected. Output to the unit 13b, output a pair of the fifth single-eye image 531 and the sixth single-eye image 532 to the parallax detection unit 13c, and set a pair of the seventh single-eye image 541 and the eighth single-eye image 542 to It outputs to the parallax detection part 13d (step S4). The camera-specific internal parameters and external parameters for which distortion correction is performed by the image correction unit 12 are Zhang's technique ("A flexible new technique for camera calibration". IEEE Transactions on Pattern Analysis and Machine Intelligence, 22 (11): 1330-1334 , 2000) etc.

  The parallax detection unit 13 a detects parallax from the input first eye image 511 and the second eye image 512 and outputs the first parallax image 51 to the parallax synthesis unit 14. In the first single-eye image 511 and the second single-eye image 512 that have undergone image correction, the corresponding points of the subject in each image are shifted in the horizontal direction according to the distance of the subject. The parallax can be detected by detecting the shifted position and obtaining the shift amount. For this parallax calculation, a technique called block matching is used. In block matching, as shown in FIG. 6, a small area 511a of the first single-eye image 511, for example, 4 pixels in the vertical direction × 4 pixels in the horizontal direction is cut out, and the corresponding area of the second single-eye image 512 is searched. Therefore, a small area 512a of the second single-eye image 512 is cut out and an evaluation value is calculated. As the evaluation value, a sum of luminance differences (SAD: Sum of Absolute Difference) between two regions 511a and 512a, a sum of squares of luminance differences (SSD: Sum of Squared Difference), and the like are obtained. In this case, the position of the region 512a of the second single-eye image 512 is moved within a predetermined range, and a position shift between images is searched by searching for a cutout position that gives a minimum value of SAD or SSD at each position. It can be obtained in pixel units. Since parallax in units of pixels often lacks accuracy, parallax of less than a pixel (sub-pixel) is also estimated. For this estimation, it is common to use equiangular line fitting or parabolic fitting. In addition, the data used for calculating SAD and SSD may be a value obtained by differentiating luminance (for example, DOG) instead of luminance. In this way, the sub-pixel level parallax for all pixels is calculated and detected as a parallax image. Here, each pixel of the parallax image corresponds to each pixel of the first single-eye image 511. In addition, the parallax is set to zero for pixels for which parallax could not be detected.

  Similarly, the parallax detection unit 13b detects the parallax from the input third eye image 521 and the fourth eye image 522, and outputs the second parallax image 52 to the parallax synthesis unit 14, and the parallax detection unit 13c. Detects the parallax from the input fifth single-eye image 531 and the sixth single-eye image 532 and outputs the third parallax image 53 to the parallax synthesis unit 14, and the parallax detection unit 13 d receives the input seventh image The parallax is detected from the eye image 541 and the eighth single-eye image 542, and the fourth parallax image 54 is output to the parallax synthesizer 14 (steps S5 and S6). When parallax is detected by the parallax detectors 13a to 13b, the parallax is not calculated in exactly the same manner, but the range of the image cutout position for calculating the parallax is set to the range of the in-focus distance. For example, in the first single-eye image 511, the region 512a of the second single-eye image 512 for performing block matching on the region 511a having the size (dx, dy) centered on the position (x, y) is Let (x + s1-cx, y-cy) to (x + s2 + cx, y + cy) be a range of size (dx, du) centered on a point inside the rectangle with the diagonal vertex. Here, cx and cy are positive values, for example, 2. This value is necessary as a margin for detecting the minimum value. The settings of s1 and s2 are shown in the following table.

These s1 and s2 are the minimum value and the maximum value of parallax at which measurement is performed, respectively, and correspond one-to-one to the maximum value and minimum value of the subject distance that can be measured. Here, the parallax shown in the table is calculated using the following equation (1)) when the focal length is 1 mm, the base length D1 = 1 mm, the base length D2 = 3 mm, and the pixel size of the image sensor 7 is 6 μm.
Subject distance = baseline length x focal length / (parallax x pixel size) (1)
In this way, a pair of the first single-eye image 511 and the second single-eye image 512, a pair of the third single-eye image 521 and the fourth single-eye image 522, the fifth single-eye image 531, The parallax in the sixth single-eye image 532 pair and the seventh single-eye image 541 and the eighth single-eye image 542 pair can be calculated.

  The parallax synthesizer 14 synthesizes the first parallax image 51 to the fourth parallax image 54 input from the parallax detectors 13a to 13d and outputs them to the distance calculator 15 (step S7). When the first parallax image 51 to the fourth parallax image 54 are combined, the second parallax image 52 to the second parallax image 52 to the second parallax image 51 that are input from the parallax detection units 13 b to 13 d to the first parallax image 51 that is input from the parallax detection unit 13 a. The values of the four parallax images 54 are mapped. Specifically, as shown in the flowchart of FIG. 7, when the parallax at the coordinates (x, y) of the second parallax image 52 input from the parallax detection unit 13b is α (≠ 0), the parallax detection unit 13a When the parallax of the coordinates (x, y−α * D1 / D2) of the first parallax image 51 input from is zero, the value α is set to the coordinates (steps S11 to S15). After performing the above-described processing on all the pixels of the second parallax image 52 (step S16), the parallax at the coordinates (x, y) of the third parallax image 53 input from the parallax detection unit 13c is β (≠ 0). When the parallax of the coordinate (x + β) of the first parallax image 51 is zero, the value (β * D2 / D1) is set to that coordinate (steps S17 to S20). After performing the above-described processing for all the pixels of the third parallax image 53 (step S21), the parallax γ (≠ 0) at the coordinates (x, y) of the fourth parallax image 54 input from the parallax detection unit 13d Then, when the parallax of the coordinates (x + γ, y−γ) of the first parallax image 51 is zero, the value (γ * D2 / D1) is set to the coordinates (steps S22 to S25). When the above-described processing is performed on all the pixels of the fourth parallax image 54, the processing ends. However, since the parallaxes α, β, and γ have values after the decimal point, they are rounded off to the nearest whole number when used for coordinate calculation of the first parallax image 51. In this way, the first parallax image 51 to the fourth parallax image 54 can be synthesized.

  The distance calculating unit 15 calculates a subject distance from the combined parallax image input from the parallax combining unit 14 by the equation (1) to obtain a distance image. The distance detection resolution increases as the subject distance decreases and decreases as the subject distance increases. In addition, since the parallax increases as the subject distance decreases, the overlapping area between pairs of single-eye images decreases, and the field of view from which distance information can be acquired is narrowed. Considering these points, a pair of the first imaging lens 411 and the second imaging lens 412 having a larger parallax and higher distance resolution due to the long base line length of D2, and a third imaging lens The curvature radius is set so that the in-focus subject distance becomes long with respect to the pair of the fourth imaging lens 422 and the fourth imaging lens 422, and the fifth imaging lens 431 and the sixth imaging lens 432 having a short baseline length, The radius of curvature is set so that the subject distance in focus is shortened for the pair of the seventh imaging lens 441 and the eighth imaging lens 442. Therefore, a distance image can be acquired with a higher resolution for a distant subject, and a distance image can be acquired with a wide field of view for a nearby subject.

  A vehicle exterior monitoring device 20 using the distance image input device 1 is shown in FIG. The vehicle outside monitoring device 20 is mounted on the vehicle 21 to image an object within a predetermined range outside the vehicle, and recognize and monitor the object outside the vehicle from the captured image. The distance image input device 1, the computer 22, and the display device 23 are monitored. Have The distance image input device 1 is disposed at a position that does not block the driver's field of view, such as the rear stage of the rear mirror of the vehicle 21, and outputs a distance image from the captured subject image to the computer 22. The computer 22 detects a road shape and a three-dimensional position of a plurality of three-dimensional objects at high speed from the input distance image, identifies a preceding vehicle or an obstacle based on the detection result, performs a judgment process of a collision warning, and the like. When the detected object becomes an obstacle of the host vehicle 21, a warning is given to the driver by displaying it on the display device 23 installed in front of the driver. Further, by connecting an external device for controlling actuators (not shown), it is possible to perform automatic collision avoidance control of the vehicle body.

1; distance image input device, 2; imaging device, 3; arithmetic processing unit, 4; lens array,
5; light shielding wall, 6; aperture array, 7; imaging element, 8; substrate, 9;
11; Image capture unit, 12; Image correction unit, 13; Parallax detection unit,
14; parallax composition unit; 15; distance calculation unit; 20;
411; first imaging lens; 412; second imaging lens;
421; a third imaging lens, 422; a fourth imaging lens,
431; fifth imaging lens, 432; sixth imaging lens,
441; seventh imaging lens; 442; eighth imaging lens.

JP 2007-263669 A Japanese Patent Laid-Open No. 2003-143459 JP 2006-217131 A JP 2008-286527 A

Claims (7)

Having an imaging device and an arithmetic processing unit;
The imaging device images a subject and outputs a plurality of stereo images each focused at different distances;
The arithmetic processing unit includes a parallax detection unit, a parallax synthesis unit, and a distance calculation unit,
The parallax detection means calculates a parallax based on a unique feature of each of a plurality of stereo images output from the imaging device, and outputs a plurality of parallax images having parallax information,
The parallax synthesis unit synthesizes a plurality of parallax images output from the parallax detection unit and outputs a synthesized parallax image,
The distance calculating means calculates a distance from the synthesized parallax image output from the parallax synthesizing means to the subject and outputs a distance image ;
The imaging apparatus includes a plurality of sets of imaging lenses for obtaining the plurality of stereo images and imaging elements, and the plurality of sets of imaging lenses have substantially the same focal length between the imaging elements. A distance image input device characterized by the above.   The plurality of sets of imaging lenses have a plurality of apertures corresponding to each imaging lens in a one-to-one correspondence and having substantially equal effective diameters because the lens sag is the same and the curvature radius is different between the sets, and the object side has a different focal length. The range image input apparatus according to claim 1, wherein an aperture array is provided on a subject side of the plurality of sets of imaging lenses. The distance according to claim 1 or 2 , wherein the unique feature of calculating parallax from a plurality of stereo images output from the imaging device by the parallax detection means is a range of distances at which each stereo image is focused. Image input device. The distance image input device according to any one of claims 1 to 3, wherein the parallax detection unit performs a parallax calculation of each stereo image pair in a parallax range corresponding to a range of distances in focus. . The imaging device forms an image by each of a lens array in which a plurality of lenses are composed of a plurality of stereo lenses each having a different radius of curvature, and the plurality of stereo lenses provided on the image plane side of the lens array. distance image input device according to any one of claims 1 to 4, characterized in that it comprises an imaging device for imaging a compound eye image a set of the subject image that. 6. The distance image input device according to claim 5, wherein the plurality of stereo lenses of the imaging device have a curvature radius changed with a lens sag constant. An out-of-vehicle monitoring device comprising the distance image input device according to any one of claims 1 to 6 and a display device and mounted on a vehicle.

Images