WO2014054753A1

WO2014054753A1 - Image processing device and device for monitoring area in front of vehicle

Info

Publication number: WO2014054753A1
Application number: PCT/JP2013/076980
Authority: WO
Inventors: 山下　龍麿
Original assignee: アルプス電気株式会社
Priority date: 2012-10-04
Filing date: 2013-10-03
Publication date: 2014-04-10
Also published as: JP2015232442A

Abstract

　Provided is an image processing device which simultaneously acquires an image with a wide field of view, an image with a narrow field of view, and an image as a stereo camera, and which is capable of estimating the distance to an object. Also provided is a device for monitoring the area in front of a vehicle. The imaging device comprises: a first imaging means (10) for capturing first image information; a second imaging means (20) for capturing second image information having a narrow field of view and having an area in common with the first image information; and an image recognition means (40) that recognizes objects based on the image information. The image recognition means is equipped with: an image processing unit (42) that acquires first and second images obtained by correcting first image information and second image information, and extracts a third image which is made to correspond to the common area from the first image, and a distance estimating means (43) which extracts characteristic points contained in a third image as a stereo range object, searches for corresponding points contained in a second image that correspond to the characteristic points, and estimates the distance.

Description

Image processing apparatus and vehicle forward monitoring apparatus

The present invention relates to an image processing apparatus and a vehicle forward monitoring apparatus that can estimate a distance to an object whose image has been recognized.

Vehicle front monitoring by mounting a camera on the vehicle, extracting the vehicle and obstacles ahead from the image information, issuing a warning before colliding with them, and operating the brake as a danger avoidance operation The device is known.

In an apparatus that estimates the distance to an object in front of a vehicle based on image information, the distance to the object is calculated by applying the principle of triangulation based on image information captured by a pair of imaging means called stereo cameras. Calculated.

Since the accuracy of distance calculation using a stereo camera depends on the accuracy of detecting the relative change in the image position, the distance that can be estimated for a distant object is the amount of image information captured. It is limited by the optical characteristics of the imaging means. For this reason, as a vehicle front monitoring device applied when traveling at a speed of about 100 km / h, a long-distance camera is used which ensures accuracy by increasing the focal length of the camera and narrowing the visual field range.

On the other hand, when driving at low speed, it is necessary to be able to recognize the vehicle diagonally forward. Also, when detecting another vehicle that has entered the vehicle immediately before the host vehicle while traveling at a high speed, it is necessary to ensure image information in a wide field of view. For this reason, a short-distance camera was used in which the focal length of the camera was shortened to ensure a field of view range.

Patent Document 1 discloses an out-of-vehicle monitoring device having a structure in which a short-distance CCD camera and a long-distance CCD camera are respectively provided as stereo cameras, and image processing of either stereo camera is switched depending on the situation. Yes.

FIG. 10 is a front view of a vehicle 200 equipped with the vehicle exterior monitoring device described in Patent Document 1. FIG. As shown in FIG. 10, a total of four cameras, that is, a long-

distance CCD camera

201, 202 and a short-

distance CCD camera

211, 212 are mounted. The image is input to the stereo image processing apparatus, the amount of positional deviation on the left and right images is determined, and the distance to the object is calculated.

Patent Document 2 discloses a structure in which a lens for a short-distance shooting camera lens and a lens for a long-distance shooting camera are each used as a stereo camera lens, and images captured by the respective lenses are formed in separate imaging regions (imaging surfaces). An imaging apparatus is disclosed. Combining the lens of a stereo camera for short-distance shooting and the lens of a stereo camera for long-distance shooting, it is possible to obtain clear image information suitable for distance measurement from the vicinity of the vehicle to a distance, and acquire these images without switching. .

Therefore, even when another vehicle or the like has entered the vehicle just before the host vehicle from diagonally forward during high-speed driving, which is the operation of the stereo camera for long-distance shooting, the image information of the stereo camera for short-distance shooting It is possible to detect an interrupting vehicle and immediately connect to a warning or danger avoidance operation.

Japanese Patent Laid-Open No. 5-265547 JP 2010-271143 A

However, in the structure of the above prior art, at least four lenses are required because the lens of the short-distance shooting stereo camera and the lens of the long-distance shooting stereo camera are used. In addition, the image sensor is an aggregate of a large number of pixels, and it is necessary to increase the number of pixels in order to improve measurement accuracy. The increase in the number of pixels not only increases the size of the image sensor, but also affects the system scale, such as an increase in the amount of electronic information processing corresponding to the number of pixels and an increase in the processing speed required for it. As described above, an image pickup apparatus that simultaneously picks up images corresponding to two sets of stereo cameras doubles the number of pixels of the image pickup element, and thus tends to be expensive and large-scale as a system. Therefore, the size of the imaging device using a pair or two pairs of cameras that require four lenses becomes large, and the image processing device that constitutes the outside monitoring device (vehicle forward monitoring device) becomes expensive. End up.

The present invention is to solve the above-described problems, and in particular, by using a pair of imaging means, an image with a wide visual field range, an image with a narrow visual field range, and an image as a stereo camera that combines these images are simultaneously displayed. The present invention provides an image processing device and a vehicle forward monitoring device that can estimate the distance to an object that has been acquired and image-recognized.

An image processing apparatus according to the present invention includes a first imaging unit that captures first image information, a field of view that is narrower than the first imaging unit, and has a common area with the first image information. Second image pickup means for picking up image information of image 2 and image recognition means for recognizing an object based on the first image information and the second image information, wherein the image recognition means The first image information is corrected to obtain the first image and the second image information is corrected to obtain the second image, and the first image is made to correspond to the common area. An image processing unit that performs a process of cutting out the third image, and a feature point included in the third image as a stereo ranging object in the object, and the second corresponding to the feature point The corresponding points in the third image are searched for corresponding points included in the image of A distance estimation processing unit that performs a process of estimating a distance to the stereo distance measurement object based on the image position and the image position of the corresponding point in the second image. To do.

The image processing apparatus acquires a first image having a wide visual field range and a second image having a common area with the first image. Thereby, the stereo image of this 2nd image and the 3rd image cut out corresponding to the common area | region with the 2nd image among 1st images can be acquired. Furthermore, by extracting feature points included in the second image and the third image, the object can be recognized as a stereo distance measurement object, and the image positions of the feature points in the third image and the corresponding points in the second image can be recognized. The shift amount of the image position can be measured based on the image position. The distance from the imaging position of the image processing apparatus to the object can be estimated from the amount of image position deviation. Therefore, it is possible to simultaneously acquire an image with a wide visual field range, an image with a narrow visual field range, and an image as a stereo camera by using a pair of imaging units each having one lens, and estimate the distance to the recognized object. . Therefore, an image processing apparatus capable of performing equivalent image processing without using two sets of stereo cameras can be provided.

Furthermore, the image processing unit includes a coordinate conversion table storage unit in which a coordinate conversion table is stored in advance, and the image processing unit removes distortion using the coordinate conversion table for the first image information. Processing is performed to acquire the first image.

Since the first image pickup means has a wider field of view range than the second image pickup means, the picked-up image information is an image in which the center peculiar to the wide-angle lens is large and the periphery is distorted. If this distortion remains, it becomes complicated to compare the third image with the second image. For this reason, distortion removal processing is performed on the captured image information to convert the image information into a first image without distortion. Since this distortion removal processing is performed using a pre-stored coordinate conversion table, arithmetic processing using mathematical formulas is unnecessary, and high-speed processing is possible even when the number of pixels of the image data is large. Since high-speed processing is possible, the processing speed of image recognition is not reduced.

Furthermore, it is preferable that the image processing unit detect a color feature included in the first image and perform a process of recognizing the object as a color feature object related to the color feature. .

If the first image is acquired as a visible light color image, the image processor can detect the color characteristics of the object included in the first image. In this way, it is possible to quickly recognize the object whose color feature is detected by the first image. In addition, if the color feature object related to the color feature in visible light by the first image is the same as the stereo distance measurement object, the distance to the color feature object can be estimated.

When the object deviates from the visual field range of the second imaging means, it is preferable that the object is continuously tracked with the first image acquired by the first imaging means.

Once the feature point recognized as the target object deviates from the second image captured in the visual field range of the second imaging unit, the feature point continues to be recognized in the first image having a wide visual field range. be able to. Therefore, the extracted object is continuously tracked. Thereby, even after deviating from the visual field range of the second image pickup means, it is possible to detect a change in the image position of the feature point with time in the first image and to estimate the distance.

In the image processing apparatus of the present invention, the first imaging unit and the second imaging unit are mounted in front of the vehicle, and the first image includes a first long-distance image in front of the vehicle. The second image is a second long-distance image in front of the vehicle, and the information on the stereo ranging object included in the second image is related on the first image. It is characterized by that.

It is an image processing device that is mounted on a vehicle and images the front of the vehicle. Since the first image is a short-distance image in front of the vehicle and corresponds to a wide field of view, signals and signs positioned at a short distance in front of the vehicle are displayed. Signals and signs can be detected by capturing within the field of view. Since the first imaging element is a wide-angle lens and has a relatively deep focal depth, a distant object is also imaged in the first image, and the third image can be cut out. If the information on the stereo ranging object included in the second image and the third image is related on the first image, a signal or a sign located at a long distance in front of the vehicle approaches a short distance in front of the vehicle. But you can continue to recognize.

Furthermore, it is preferable that the first imaging unit and the second imaging unit are disposed on the same substrate. Since the first image pickup means and the second image pickup means are arranged on the same substrate, it is possible to perform optical adjustment before attaching to the place of use. Therefore, attachment is easy, and optical calibration after attachment is unnecessary compared with the arrangement of the left and right cameras separately.

The second imaging means can collect infrared light, the first imaging means can collect near-infrared light, and the second image and the third image are red. It is preferable that the image captured with external light is included. In this way, even at night or in a tunnel where the amount of visible light is small, it can be supplemented with an image by infrared light, and so-called night vision ability is improved.

Furthermore, the vehicle forward monitoring device of the present invention is a vehicle forward monitoring device including the above-described image processing device, and is characterized by notifying an occupant of forward information with a distance to an object estimated by the image processing device. Have.

The image processing apparatus acquires a first image having a wide visual field range and a second image having a common area with the first image. Thereby, the stereo image of this 2nd image and the 3rd image cut out corresponding to the common area | region with the 2nd image among 1st images can be acquired. Furthermore, by extracting feature points included in the second image and the third image, the object can be recognized as a stereo distance measurement object, and the distance to the stereo distance measurement object can be estimated. In this way, the vehicle forward monitoring device provided with the image processing device can estimate the distance to an object located at a long distance ahead of the vehicle and can be used for collision prevention warning and cruise control. In addition, since a signal or a sign located at a short distance in front of the vehicle can be captured in the field of view by an image with a wide visual field range, automatic signal recognition and automatic sign recognition are possible. Furthermore, lane detection is possible for short-distance images with a wide visual field range, and meandering detection, warning, and control are also possible. Therefore, it is possible to simultaneously acquire an image of a wide visual field range and an image as a stereo camera by a pair of imaging means each having one lens, and estimate the distance to the stereo distance measurement object recognized as an image, Warning and control with a wide field of view image is possible. Therefore, it is possible to provide a vehicle forward monitoring apparatus that can perform equivalent image processing without using two sets of stereo cameras.

According to the present invention, the image processing apparatus acquires a first image having a wide visual field range and a second image having a common area with the first image. Thereby, the stereo image of this 2nd image and the 3rd image cut out corresponding to the common area | region with the 2nd image among 1st images can be acquired. Furthermore, by extracting feature points included in the second image and the third image, the object can be recognized as a stereo distance measurement object, and the distance to the stereo distance measurement object can be estimated. If it carries out like this, it can be set as the image processing apparatus which can perform an equivalent image process by using a pair of imaging means each provided with one lens, without using two sets of stereo cameras.

Therefore, an image processing apparatus and a vehicle forward monitoring apparatus that can simultaneously acquire an image with a wide field of view and an image as a stereo camera by using a pair of imaging means and estimate the distance to the recognized stereo distance measurement object are provided. can do.

1 is a block diagram illustrating an image processing apparatus according to a first embodiment of the present invention. It is a model front view of the vehicle by which the image processing apparatus of 1st Embodiment is mounted. It is the bird's-eye view which looked at the vehicle carrying the image processing device of a 1st embodiment from the top. FIGS. 4A and 4B are explanatory diagrams showing image correction of the first image information. FIG. 4A is an image image before the image information is corrected, and FIG. 4B is an image image after the correction. It is a flowchart explaining the image processing procedure in 1st Embodiment. It is explanatory drawing which shows the 1st image acquired by the 1st imaging means, and the 2nd image acquired by the 2nd imaging means, (a) is description explaining a 1st image and a 3rd image. FIG. 4B is an explanatory diagram showing the second image. It is explanatory drawing which shows the 1st image acquired by the 1st imaging means, and the 2nd image acquired by the 2nd imaging means, (a) is explanatory drawing which shows a 1st image, (b) FIG. 6 is an explanatory diagram showing a second image. It is explanatory drawing of the spectral characteristic of a visible light filter and an infrared-light filter. It is a block diagram of the vehicle front monitoring apparatus provided with the image processing apparatus of 2nd Embodiment. It is a front view of a vehicle provided with the conventional exterior monitoring apparatus.

[First Embodiment]
The image processing apparatus 1 in the first embodiment will be described below.

FIG. 1 is a block diagram showing an image processing apparatus 1 in the first embodiment. The image processing apparatus 1 includes a camera unit 30, an image recognition unit 40, and a control unit 50, and is mounted on the vehicle so as to capture the front of the vehicle. The control unit 50 supplies power to the camera unit 30 and the image recognition unit (image recognition unit) 40 and outputs a control signal to control each state.

The camera unit 30 includes a wide-angle lens 11 and a first image sensor 12, and includes a first image capturing unit 10 that obtains first image information, a telephoto lens 21, and a second image sensor 22. The second image pickup means 20 for obtaining the second image information, and the image output section 31 connected to the first image pickup means 10 and the second image pickup means 20 are provided. The image output unit 31 is an electronic circuit that repeatedly acquires the first image information and the second image information at predetermined time intervals and outputs them to the image recognition means 40.

An image recognition means (image recognition unit) 40 recognizes an object based on the first image information and the second image information, and therefore, an image input unit 41, an image processing unit 42, a distance estimation processing unit 43, and A display image output unit 44. The image information transmitted from the image output unit 31 is input to the image input unit 41, and the image processing unit 42 corrects the acquired image information to acquire the first image and the second image, The third image is cut out from the first image. The image processing unit 42 outputs stereo image data to be sent to the distance estimation processing unit 43 and display image data to be sent to the display image output unit 44.

The distance estimation processing unit 43 includes the third image cut out from the first image corrected by the image processing unit 42 and the second image corrected by the image processing unit 42 in common. A process for recognizing the stereo distance measurement object and estimating the distance to the stereo distance measurement object is performed.

The display image output unit 44 adjusts the display image output unit 44 so as to be easily viewed by the user and outputs a display image. Moreover, the distance estimation process part 43 emphasizes the stereo ranging object and performs the process which displays the estimated distance.

FIG. 2 is a schematic front view of a vehicle equipped with the image processing apparatus 1 of the present embodiment. When viewed from the front of the vehicle, it can be visually confirmed that the camera unit 30 is mounted over the windshield. The camera unit 30 includes a wide-angle lens 11 and a telephoto lens 21. The wide-angle lens 11 and the telephoto lens 21 are positioned at the same height and in the left-right direction of the vehicle, and the optical axis of the lens is directed in the front direction of the vehicle. ing. The camera unit 30 includes the first imaging unit 10 and the second imaging unit 20 together with the image output unit 31 on one camera unit substrate (not shown) so that optical adjustment can be easily performed. It is. Further, the image recognition means 40 is disposed inside the vehicle, separated from the camera unit 30.

Since the image processing apparatus 1 of this embodiment is mounted on a vehicle, the display image output from the display image output unit 44 is visually recognized by the display means 60 disposed on the front side of the driver's seat of the vehicle. The display means 60 is, for example, a liquid crystal display device, and a display screen of a car navigation device or an audio device can be used. Further, the display means 60 may be provided exclusively in the vicinity of the driving steering wheel or on the windshield in front of the driver's seat. It is preferable to correct the dynamic range and gradation in the display image output unit 44 in accordance with the display characteristics of the display means 60 to be used.

Next, the imaging means of the image processing apparatus 1 of this embodiment will be described.

In the present embodiment, the first imaging unit 10 includes a wide-angle lens 11 and a first imaging element 12, while the second imaging unit 20 includes a telephoto lens 21 and a second imaging element 22. Have.

FIG. 3 is an overhead view of a vehicle on which the image processing apparatus 1 of the present embodiment is mounted as viewed from above. Schematically, the horizontal field angle (horizontal field angle A1) of the first imaging unit 10 and the horizontal field angle (horizontal field angle A2) of the second imaging unit 20 are represented.

The first image pickup device 12 used in the first image pickup means 10 has a large number of pixels and includes a visible light filter for color photography. As the first image pickup device 12, a solid-state image pickup device mounted on a video camera can be used. For example, a solid-state imaging device capable of capturing a horizontal 1920 × vertical 1080 color image at 30 frames / second is used. The wide-angle lens 11 was selected so that a visual field range of 40 degrees could be imaged as the horizontal field angle A1.

On the other hand, the second imaging element 22 used in the second imaging means 20 has a standard number of pixels called VGA and does not include a visible light filter. For example, a solid-state imaging device that can capture a black and white image of horizontal 640 × vertical 480 at 30 frames / second is used. The telephoto lens 21 was selected so that a visual field range of 20 degrees could be imaged as the horizontal field angle A2. As shown in FIG. 3, the visual field range of the second imaging unit 20 is within the visual field range of the first imaging unit 10.

<Image processing procedure>
Next, an image processing procedure in the present embodiment will be described. In the image processing apparatus 1, the image recognition unit (image recognition unit) 40 is not affected by the relative mounting posture between the first imaging unit 10 and the second imaging unit 20, and is included in the actual camera posture. A parallelization process (Rectification) is performed to remove the influence of the above. At this time, distortion removal processing is also performed at the same time.

Generally, the ideal arrangement state of the left and right image pickup means constituting the stereo camera is a state in which the left and right image pickup means are completely fixed in the same parallel posture. For this reason, the distance calculation process using a stereo camera targets a pair of left and right images taken in a parallel posture.

The ideal relative mounting orientation (parallel orientation) of a stereo camera is an orientation in which the camera coordinate system X axis (camera right direction) is parallel to the baseline and the camera optical axis (camera coordinate system Z axis) is parallel. That is. The base line in the stereo camera is a straight line connecting the left and right lens centers. The parallelization processing of the image information is achieved by converting the captured image information into an image that would have been captured when the imaging unit was in an ideal parallel posture. Here, the camera coordinate system is a coordinate system possessed by the camera, with the center of the image as the origin, the right direction is the positive direction of the X axis, the downward direction is the positive direction of the Y axis, and the front direction of the camera is the positive direction of the Z axis. Represents a direction.

In the image processing apparatus 1, the first imaging unit 10 and the second imaging unit 20 are incorporated in the camera unit 30, and are mechanically adjusted and fixed with respect to the mounting position and inclination. Thereafter, optical relative position data of the second imaging unit 20 with respect to the first imaging unit 10 is acquired by image information acquired by imaging a checkerboard in which white and black squares are alternately arranged. . That is, the relative position in the direction of the optical axis (camera coordinate system Z axis), the vertical and horizontal positions on a plane (plane of the camera coordinate system X axis and Y axis) orthogonal to the optical axis (camera coordinate system Z axis) of both lenses , And the roll, pitch, and yaw of the second imaging means 20 relative to the first imaging means 10 are grasped.

The relative position data obtained in this way is recorded in advance in the image processing unit 42, and conversion is performed to calibrate the captured image information to an image that would have been captured when the imaging means was in an ideal parallel posture. Applies to parallelization processing.

At this time, at least the distortion removal processing of the image information captured by the first imaging unit 10 is performed at the same time. That is, since the imaging means has an image height characteristic unique to the lens, the conversion is performed so that the XY coordinates on the image data coincide with the XY coordinates of the position in the real space. Since the XY coordinate data of the checkerboard is known, distortion removal processing data can be acquired from image information obtained by imaging the checkerboard. Since the first imaging unit 10 has a wider field of view than the second imaging unit 20, the captured image information is an image in which the center peculiar to the wide-angle lens 11 is greatly reflected and the periphery is distorted. If this distortion remains, it becomes complicated to compare the third image with the second image. For this reason, distortion removal processing is performed on the captured image information to convert the image information into a first image without distortion. FIG. 4 is an explanatory diagram illustrating image correction of the first image obtained by imaging the checkerboard. FIG. 4A shows an image before the image is corrected, and FIG. 4B shows an image after the correction. The distortion removal process for the image information captured by the second imaging unit 20 can be performed in the same manner. In this way, even if the first imaging unit 10 and the second imaging unit 20 have unique lens characteristics and are not in an ideal parallel posture, it can be corrected to an ideal image. .

In the present embodiment, these image corrections are performed by a conversion method called LUT (Look Up Table). In this method, the coordinates of each pixel in the captured image information are rearranged at the coordinate position to be imaged when there is no distortion, and this is a method of creating a coordinate conversion table in advance. In the present embodiment, a coordinate conversion table is created from image information obtained by imaging the checkerboard, and the coordinate conversion table is stored in advance in the coordinate conversion table storage unit 45 of the image processing unit 42. In the image processing unit 42, using the coordinate conversion table for the first image information stored in the coordinate conversion table storage unit 45, parallelization processing and distortion removal processing are performed on the first image information. A first image is acquired. Similarly, the second image information is subjected to a parallelization process and a distortion removal process using a second image information coordinate conversion table to obtain a second image. Note that if one image information is used as a reference, only the other image information needs to be performed for the parallelization processing. Generally, these image corrections can be performed by calculation using a calculation formula for each pixel of image information. However, when the number of pixels increases, the amount of calculation becomes enormous, which causes a problem that the processing speed decreases. In the present embodiment, since this distortion removal processing is performed using a coordinate conversion table stored in advance, arithmetic processing using mathematical formulas is unnecessary, and high-speed processing is possible even if the number of pixels of the image data is large. Since high-speed processing is possible, the processing speed of image recognition is not reduced.

FIG. 5 is a flowchart for explaining the image processing procedure in the first embodiment. As an image processing procedure in the image recognition unit 40, in step ST1, image information captured by the first imaging unit 10 and the second imaging unit 20 is input to the image processing unit 42, and image information is acquired. In step ST2, the above-described image correction is performed, the first image information is corrected to correct the first image, and the second image information is corrected to acquire the second image.

Further, in step ST3, a third image corresponding to the common area with the second image is cut out from the acquired first image, and the third image is acquired. The first image picked up by the first image pickup means 10 has a wide field of view since it has a horizontal angle of view of 40 degrees, and includes a common area with the second image picked up by the second image pickup means 20. In other words, since the second image captured by the second imaging unit 20 has a horizontal angle of view of 20 degrees, the visual field range is narrower than that of the first image, and a common area with the first image is formed. Have. Therefore, it is possible to cut out the third image corresponding to the common area with the second image from the first image.

The image processing unit 42 performs image processing for correcting the first image in step ST2 and cutting out the third image from the first image in step ST3. At this time, the cut out third image is converted into a grayscale image.

Next, the distance estimation processing unit 43 extracts the feature points of the object included in the third image corresponding to the common area, recognizes the stereo distance measurement object related to the feature points, Processing for estimating the distance from the imaging position of the processing device to the stereo distance measurement object is performed. In step ST4, the feature point of the target object included in the third image is extracted, and the corresponding second image is searched for the feature point shown in the third image. When the corresponding point of the second image corresponding to the feature point shown in the third image is searched, in step ST5, the image position of the feature point in the third image and the image of the corresponding point in the second image The shift amount of the image position is measured based on the position. For a stereo ranging object related to the feature point of the object included in the third image and the corresponding point included in the second image, the distance is determined according to the principle of triangulation by the amount of image position deviation. Calculated. This process is repeated in the same manner for the image information captured at the next timing until the image process is completed in step ST6.

FIG. 6 is an explanatory diagram showing a first image and a second image. In FIG. 6, the distance from the vehicle to the signal is assumed to be about 50 m. FIG. 6A is an explanatory diagram showing a first image and a third image (F region image) captured by the first imaging means 10. FIG. 6B is an explanatory diagram showing a second image captured by the second imaging unit 20. An image after the above-described correction processing is shown. In FIG. 6A, image processing for cutting out the image of the F region from the first image is performed corresponding to the common region with the second image. A signal far away in front of the vehicle is commonly included in the second image and the third image.

FIG. 7 is an explanatory diagram showing a first image and a second image. In FIG. 7, the distance from the vehicle to the signal is assumed to be about 15 m. FIG. 7A is an explanatory diagram illustrating a first image captured by the first imaging unit 10. FIG. 7B is an explanatory diagram illustrating a second image captured by the second imaging unit 20. In Fig.7 (a), it turns out that the signal ahead of a vehicle is contained in the 1st image. However, in FIG. 7B, the signal is not included in the second image. That is, when the distance to the signal shown in FIG. 6 is 50 m, both the first imaging means 10 and the second imaging means 20 can recognize the signal, but the distance to the signal is up to 15 m. In FIG. 7 that is approaching, the signal can be recognized only by the first imaging means 10 alone. This is because the visual field range of the first imaging means 10 is wide, but the image processing apparatus 1 uses the first image for objects around the visual field range, such as a signal when approaching the front of the vehicle. Can continue to recognize.

6 is recognized as an object common to the third image and the second image as in the signal shown in FIG. 6, and the relative displacement amount of the image position of the object in the left and right images is measured, and the image position displacement is detected. The distance is estimated by the quantity. Once the feature point recognized as the target object deviates from the second image captured in the visual field range of the second imaging unit 20, the feature point is recognized in the first image having a wide visual field range. You can continue. Therefore, the extracted object is continuously tracked. Thereby, even after deviating from the visual field range of the second imaging means 20, it is possible to estimate the distance to the object based on how the image position of the feature point changes with time in the first image. Therefore, if the information of the stereo ranging object included in the second image and the third image is related on the first image, a signal or a sign located at a long distance in front of the vehicle is displayed at a short distance in front of the vehicle. You can continue to recognize even when approaching.

Furthermore, since the first image sensor 12 is an image sensor for color photography, it is possible to detect the color characteristics of the object from the first image. In this case, the image processing unit 42 detects a color feature included in the first image, and performs a process of recognizing the object as a color feature object related to the color feature. For this reason, it is possible to quickly recognize an object whose color feature in visible light is extracted from the first image. The color feature of the feature point recognized as the stereo distance measurement object common to the third image and the second image can be detected and the object can be continuously recognized. In addition, if the color feature object related to the color feature in visible light by the first image is the same as the stereo distance measurement object common to the third image and the second image, The distance to the object can be estimated.

If the color feature of the signal tracked in FIG. 7A is recognized, when the signal changes from a red signal to a blue signal, it is recognized as a change in the color feature, and the fact that the color has changed is output to the display image. Or sound a signal.

Therefore, it is suitable for the image processing apparatus 1 that is mounted on a vehicle and images the front of the vehicle. In the image processing apparatus 1, the first imaging unit 10 and the second imaging unit 20 are mounted so as to image the front of the vehicle. The first image is a short-distance image including a first long-distance image in front of the vehicle, and the second image is a second long-distance image in front of the vehicle, and the stereo measurement included in the second image. Distance object information can be correlated on the first image. Since the first imaging element 12 is the wide-angle lens 11 and the depth of focus is relatively deep, a distant object can be imaged in the first image. Since signals and signs located at a short distance in front of the vehicle are captured within the field of view, automatic signal recognition and automatic sign recognition are possible.

Further, if the camera unit 30 in which the first image pickup means 10 and the second image pickup means 20 are arranged and incorporated on one camera unit substrate is arranged so as to image the front of the vehicle through the windshield of the vehicle. Easy to attach to the vehicle. Furthermore, since the first imaging means 10 and the second imaging means 20 are arranged on the same substrate as compared with the arrangement where the left and right cameras are separated, optical adjustment is performed before being attached to the vehicle. It is possible to leave. Therefore, it is easy to mount, and there is an effect that the optical calibration after the mounting is unnecessary as compared with the arrangement of the left and right cameras separately. In the small camera unit 30, the distance between the left and right lenses (base line length) is set to about 65 mm. On the other hand, when it is desired to ensure the distance estimation accuracy at a longer distance, the base line length may be about 300 mm.

In this embodiment, each of the first imaging unit 10 and the second imaging unit 20 uses only one lens, and thus there are only two lenses. Further, the imaging surface of the imaging device is an aggregate of a large number of pixels, and in order to improve measurement accuracy, it is necessary to increase the number of pixels, but it is not necessary to correspond to four lenses. The increase in the number of pixels of the imaging element affects not only the size of the imaging surface but also the system scale, such as an increase in the amount of electronic information processing corresponding to the number of pixels and an increase in the processing speed required for it. Therefore, the image processing apparatus 1 that only acquires an image corresponding to a pair of stereo cameras has a minimum number of pixels of the image sensor and is inexpensive as a system.

In the present embodiment, the first image sensor 12 is a color image sensor having a visible light filter, and the second image sensor 22 is a black and white image sensor having no visible light filter. However, it is preferable that the imaging device has sensitivity to near infrared light. For example, when the second imaging unit 20 can collect near-infrared light and the first imaging unit 10 can collect near-infrared light, the second image and the third image Includes an image captured with near-infrared light.

In the first image sensor 12, in addition to the visible light filter, an infrared light filter may be prepared, and pixels for imaging infrared light blocked by the visible light filter may be arranged. FIG. 8 is an explanatory diagram of spectral characteristics of the visible light filter and the infrared light filter. An RGB filter with a wavelength of red, green, and blue, respectively, and an IR filter that transmits near infrared light with a wavelength of 700 nm or more and reflects or absorbs visible light with a wavelength of 680 nm or less. is there. In this way, even at night or in a tunnel where the amount of visible light is small, it can be supplemented with an image acquired with infrared light, and so-called night vision ability is improved. When infrared light is included in the emission spectrum of the headlight, or when an auxiliary light for irradiating infrared light is used, an effect of further improving the night vision ability is achieved.

Since the second image sensor 22 does not include a visible light filter, it is sensitive to near infrared light without being blocked by the visible light filter. In addition, the infrared cut filter which cuts infrared light for adjustment with visibility is not used.

The second image sensor 22 may be a far-infrared image sensor that detects far-infrared light. In this case, although it is difficult to combine imaging with visible light, an image with far-infrared light can be acquired as the second image, so the sensitivity of detecting a human body at night is further improved. .

It should be noted that the image processing necessary for stereo distance measurement does not include extra peripheral image data, and therefore has a secondary effect of high processing speed.

As described above, the image processing apparatus 1 according to the present embodiment acquires a first image having a wide visual field range and a second image having a common area with the first image. Thereby, the stereo image of this 2nd image and the 3rd image cut out corresponding to the common area | region with the 2nd image among 1st images can be acquired. Furthermore, by extracting feature points included in the second image and the third image, the object can be recognized as a stereo distance measurement object, and the image position of the feature point in the third image and the corresponding point in the second image can be recognized. The shift amount of the image position can be measured based on the image position. The distance from the imaging position of the image processing apparatus 1 to the stereo distance measurement object can be estimated from the amount of image position deviation.

Therefore, a distance to a stereo distance measuring object that is obtained by simultaneously acquiring an image of a wide field of view, an image of a narrow field of view, and an image as a stereo camera by a pair of imaging means each having one lens. Can be estimated. Therefore, an image processing apparatus capable of performing equivalent image processing without using two stereo cameras can be provided.

The image processing apparatus 1 can be used for a security camera, a self-propelled robot, etc. in addition to being mounted on a vehicle.

[Second Embodiment]
An image processing apparatus 2 according to the second embodiment of the present invention will be described.

FIG. 9 is a block diagram of the vehicle front monitoring apparatus 100 including the image processing apparatus 2 according to the second embodiment. The parts common to the image processing apparatus 1 of the first embodiment are given the same reference numerals and the description thereof is omitted.

In the second embodiment, in the image processing apparatus 2, in addition to passing display information from the distance estimation processing unit 43 to the display image output unit 44, the front information is transmitted to the occupant with the estimated distance to the target object. Is sent to the warning determination unit 70.

In the warning determination unit 70, the distance estimation processing unit 43 estimates the distance to the object ahead of the vehicle and the object continuous tracking data recognized by the display image output unit 44. A decision is made to issue a warning. For example, an alarm is output to the warning means 71 that issues a warning before colliding with a preceding vehicle in front or an obstacle, or a command for operating the brake of the vehicle as a risk avoidance operation is output to the risk avoidance operation means 72. To do.

Note that, similarly to the image processing apparatus 1 of the first embodiment, the display data may be output from the display image output unit 44 to the display means 60 using a display screen such as a car navigation apparatus.

Hereinafter, the effects of the present embodiment will be described.

The first imaging means 10 can acquire the first image information in a wide visual field range in front of the vehicle. The second imaging unit 20 can acquire second image information having a common area with the first image. The image processing unit 42 corrects the acquired image information to acquire the first image and the second image, and performs a process of cutting out the third image from the first image. Therefore, the image processing apparatus 2 acquires a first image having a wide visual field range and a second image having a common area with the first image. Thereby, the stereo image of this 2nd image and the 3rd image cut out corresponding to the common area | region with the 2nd image among 1st images can be acquired. Furthermore, by extracting feature points included in the second image and the third image, the object can be recognized as a stereo distance measurement object, and the image positions of the feature points in the third image and the corresponding points in the second image can be recognized. The shift amount of the image position can be measured based on the image position. The distance from the imaging position of the image processing device 2 to the stereo distance measurement object can be estimated from the amount of image position deviation.

The vehicle forward monitoring device 100 including the image processing device 2 can estimate the distance to an object located at a long distance in front of the vehicle and can be used for collision prevention warning and cruise control. Since a signal or a sign located at a short distance in front of the vehicle can be captured in the field of view by an image having a wide field of view imaged by the image processing device 2, automatic signal recognition and sign recognition are possible. Since the signal and the sign can be recognized, the warning means 71 can be used for driving assistance that issues a voice or warning. Further, when the driver does not recognize the red light or the regulation sign, the warning by the warning means 71 and the brake operation by the danger avoiding operation means 72 can be performed. Furthermore, lane detection is possible for short-distance images with a wide visual field range, and meandering detection, warning, and control are also possible. Therefore, it is possible to simultaneously acquire an image of a wide visual field range and an image as a stereo camera by a pair of imaging means each having one lens, and estimate the distance to the stereo distance measurement object recognized as an image, Warning and control with a wide field of view image is possible.

Further, once the feature point recognized as the target object deviates from the second image captured in the visual field range of the second imaging unit 20, the feature point is detected in the first image having a wide visual field range. Can continue to recognize. Therefore, the extracted object can be continuously tracked within the visual field range of the display image data sent to the display image output unit 44.

As described above, the image processing apparatus 2 according to the present embodiment acquires the first image having a wide visual field range and the second image having the common area with the first image. Thereby, the stereo image of this 2nd image and the 3rd image cut out corresponding to the common area | region with the 2nd image among 1st images can be acquired. Furthermore, the objects included in the second image and the third image can be recognized as a stereo distance measurement object by extracting feature points, and the distance to the stereo distance measurement object can be estimated. Therefore, a pair of imaging means each having one lens simultaneously acquire an image with a wide field of view and an image as a stereo camera, and the front information is given to the occupant with the estimated distance to the stereo distance measurement object. It can be set as the vehicle front monitoring apparatus to alert | report.
Therefore, it is possible to provide a vehicle forward monitoring apparatus that can perform equivalent image processing without using two stereo cameras.

Therefore, an image processing apparatus capable of simultaneously obtaining an image of a wide visual field range, an image of a narrow visual field range, and an image as a stereo camera by a pair of imaging means, and estimating the distance to the stereo distance measurement object recognized as an image, and A vehicle front monitoring apparatus can be provided.

The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the scope of the invention. For example, the present invention can be modified as follows, and these embodiments also belong to the technical scope of the present invention.

The image processing apparatus may be provided with wireless communication means, and may be wirelessly communicated with a remote control device having the display means 60.

Alternatively, the display means 60 may be integrally provided in the image processing apparatus.

DESCRIPTION OF

SYMBOLS

1, 2 Image processing apparatus 10 1st imaging means 11 Wide angle lens 12 1st imaging device 20 2nd imaging means 21 Telephoto lens 22 2nd imaging device 30 Camera unit 31 Image output part 40 Image recognition means (image recognition) unit)
DESCRIPTION OF SYMBOLS 41 Image input part 42 Image processing part 43 Distance estimation process part 44 Display image output part 45 Coordinate conversion table memory | storage part 50 Control means 60 Display means 70 Warning determination part 71 Warning means 72 Risk avoidance operation means 100 Vehicle forward monitoring apparatus

Claims

First imaging means for imaging first image information;
A second imaging unit that captures second image information having a field of view narrower than the first imaging unit and having a common area with the first image information;
Image recognition means for recognizing an object based on the first image information and the second image information,
The image recognition means corrects the first image information to obtain a first image and a second image by correcting the second image information, and obtains the second image from the first image. An image processing unit that performs a process of cutting out a third image corresponding to the common area;
In the object, a feature point included in the third image as a stereo ranging object is extracted, and a corresponding point included in the second image corresponding to the feature point is searched, and the third object is searched. A distance estimation processing unit that performs a process of estimating a distance to the stereo distance measurement object based on the image position of the feature point in the image and the image position of the corresponding point in the second image; An image processing apparatus comprising:
The image processing unit has a coordinate conversion table storage unit in which a coordinate conversion table is stored in advance,
The image according to claim 1, wherein the image processing unit obtains the first image by performing distortion removal processing using the coordinate conversion table on the first image information. Processing equipment.
In the image processing unit, a color feature included in the first image is detected, and a process for recognizing the object as a color feature object related to the color feature is performed. The image processing apparatus according to claim 1.
The object is continuously tracked by the first image acquired by the first imaging means when the object is out of the visual field range of the second imaging means. The image processing apparatus described.
The first imaging means and the second imaging means are mounted so as to image the front of the vehicle,
The first image is a short distance image including a first long distance image in front of the vehicle,
The second image is a second long-distance image in front of the vehicle,
The image processing apparatus according to claim 1, wherein information on the stereo distance measurement object included in the second image is related on the first image.
2. The image processing apparatus according to claim 1, wherein the first imaging unit and the second imaging unit are disposed on the same substrate.
The second imaging means can collect infrared light, the first imaging means can collect near-infrared light, and the second image and the third image are red. The image processing apparatus according to claim 1, further comprising an image captured with external light.
A vehicle forward monitoring device comprising the image processing device according to any one of claims 1 to 7 on a vehicle, wherein the vehicle is in front of the occupant with a distance to the stereo ranging object estimated by the image processing device. A vehicle forward monitoring device for notifying information.