JP5951785B2 - Image processing apparatus and vehicle forward monitoring apparatus - Google Patents

Description

  The present invention relates to an image processing apparatus and a vehicle forward monitoring apparatus that can estimate a distance to an object whose image has been recognized.

  Vehicle front monitoring by mounting a camera on the vehicle, extracting the vehicle and obstacles ahead from the image information, issuing a warning before colliding with them, and operating the brake as a danger avoidance operation The device is known.

  In an apparatus that estimates the distance to an object in front of a vehicle based on image information, the distance to the object is calculated by applying the principle of triangulation based on image information captured by a pair of imaging means called stereo cameras. Calculated.

  Since the accuracy of distance calculation using a stereo camera depends on the accuracy of detecting the relative change in the image position, the distance that can be estimated for a distant object is the amount of image information captured. It is limited by the optical characteristics of the imaging means. For this reason, as a vehicle front monitoring device applied when traveling at a speed of about 100 km / h, a long-distance camera is used which ensures accuracy by increasing the focal length of the camera and narrowing the visual field range.

  On the other hand, when traveling at a low speed, it is necessary to be able to recognize the vehicle diagonally forward. Also, when detecting another vehicle that has entered the vehicle immediately before the host vehicle while traveling at a high speed, it is necessary to ensure image information in a wide field of view. For this reason, a short-distance camera was used in which the focal length of the camera was shortened to ensure a field of view range.

  Patent Document 1 discloses an out-of-vehicle monitoring device having a structure in which a short-distance CCD camera and a long-distance CCD camera are respectively provided as stereo cameras, and image processing of either stereo camera is switched depending on the situation. Yes.

  FIG. 13 is a front view of a vehicle 200 including the outside monitoring apparatus described in Patent Literature 1. FIG. As shown in FIG. 13, a total of four cameras, a long-distance CCD camera 201, 202 and a short-distance CCD camera 211, 212, are mounted. The image is input to the stereo image processing apparatus, the amount of positional deviation on the left and right images is determined, and the distance to the object is calculated.

  Patent Document 2 discloses a structure in which a lens for a short-distance shooting camera lens and a lens for a long-distance shooting camera are each used as a stereo camera lens, and images captured by the respective lenses are formed in separate imaging regions (imaging surfaces). An imaging apparatus is disclosed. Combining the lens of a stereo camera for short-distance shooting and the lens of a stereo camera for long-distance shooting, it is possible to obtain clear image information suitable for distance measurement from the vicinity of the vehicle to a distance, and to acquire these images without switching. .

  Therefore, even when another vehicle or the like has entered the vehicle just before the host vehicle from diagonally forward during high-speed driving, which is the operation of the stereo camera for long-distance shooting, the image information of the stereo camera for short-distance shooting It is possible to detect an interrupting vehicle and immediately connect to a warning or danger avoidance operation.

JP-A-5-265547 JP 2010-271143 A

  However, the long-distance shooting camera is required to be able to capture an image even on a road with few street lamps at night, and a camera capable of so-called night vision is required. Therefore, a camera that captures an image in the infrared light wavelength band using an infrared light filter or a camera that captures an image in the visible light wavelength band and the infrared light wavelength band without using an infrared cut filter is used. The In the structure of the above-described prior art, at least four lenses are required because the lenses of the short-distance shooting stereo camera and the long-distance shooting stereo camera are used.

  Further, these night vision camera images are not color images in the visible light wavelength band, and thus the color of the object cannot be acquired. Therefore, the color information could not be recognized. In order to recognize color information, in addition to a pair of stereo cameras that capture images in the infrared light wavelength band, a camera that captures images in the visible light wavelength band is prepared. Device), and a necessary image processing apparatus becomes expensive.

  The present invention is for solving the above-described problems, and in particular, by using a pair of imaging means, an image including color features, a night vision image, and an image as a stereo camera are simultaneously acquired, and the image recognition target is also obtained. It is possible to provide an image processing apparatus and a vehicle forward monitoring apparatus that can estimate the distance of the image and recognize the image of the color feature object.

The image processing apparatus of the present invention receives visible light and near-infrared light collected through a lens having a relatively wide angle of view, so that the front of the vehicle can be imaged with visible light and near-infrared light . A field-of-view range including a common area with the field of view of the first imaging unit in front of the vehicle by receiving far-infrared light collected through one imaging unit and a lens having a relatively narrow angle of view On the basis of the second image pickup means capable of picking up the image with far infrared light, the first image information picked up by the first image pickup means and the second image information picked up by the second image pickup means. Image recognition means for recognizing an object, wherein the image recognition means corrects the first image information and corrects the first image and the second image information to correct the second image information. white image, acquires a further corresponding to the common region and the second image from said first image The cut out third image, the by the first image by detecting the color characteristics of the object, an image of the object image recognizing processing is performed as the color characteristic objects that are related with the color feature A processing unit extracts feature points of the object included in the third image, recognizes the object as a stereo ranging object, and sets corresponding points corresponding to the feature points in the second image. A distance estimation processing unit that performs a process of estimating a distance from the feature point of the third image and the corresponding point of the second image to the target object, and the extracted feature point is Even after deviating from the range of the second and third images, the object continues to be recognized as the color feature object in the first image, and the vehicle occupant is informed of the object information as necessary. It is characterized by notifying .

The first imaging means can capture the first image information by receiving visible light and near-infrared light collected through a lens having a relatively wide angle of view . The second imaging means receives far-infrared light collected through a lens having a relatively narrow angle of view, so that the second image can be obtained in a visual field range including the visual field and the common area of the first imaging means. Information can be imaged. Accordingly, the image processing apparatus corrects the first image information and the second image information to acquire the first image and the second image, and detects the color feature of the object by the first image. Thus, the object is image-recognized as the color feature object related to the color feature. Further, based on the first image and the second image, the object can be recognized as a stereo distance measurement object, and the distance from the imaging position of the image processing apparatus to the stereo distance measurement object can be estimated. Accordingly, an image including a color feature and an image as a stereo camera can be simultaneously acquired by a pair of imaging units each including a single lens, and the distance to the recognized stereo ranging object can be estimated. The feature object can be recognized as an image. Therefore, an image processing apparatus capable of performing equivalent image processing without using two stereo cameras can be provided.

In addition , since the first image is acquired with visible light and near-infrared light, the object can be recognized by detecting the color characteristics of the object included in the first image, and the first image and An image as a stereo camera in the infrared wavelength band with the second image can be acquired. In this way, even at night or in a tunnel where the amount of visible light is small, it can be supplemented with an image by infrared light, and the night vision stereo ranging performance is improved.

Further, since acquiring an image of the far-infrared light as the second image, the sensitivity of detecting the human body at night, is further improved.

In addition , signals and signs located at a short distance in front of the vehicle can be captured in the field of view, and the signals and signs can be detected by color characteristics. Since the first imaging means has a wide visual field range, the first image can also capture a common area having the same visual field range as the second image, and a third image corresponding to the common area can be cut out. it can. If the information on the stereo ranging object included in the second image and the third image is related on the first image, a signal or a sign located at a long distance in front of the vehicle approaches a short distance in front of the vehicle. But you can continue to recognize.

  It is preferable that the first imaging unit and the second imaging unit are disposed on the same substrate. Since the first image pickup means and the second image pickup means are arranged on the same substrate, it is possible to perform optical adjustment before attaching to the place of use. Therefore, attachment is easy, and optical calibration after attachment is unnecessary compared with the arrangement of the left and right cameras separately.

  The image processing unit includes a coordinate conversion table storage unit in which a coordinate conversion table is stored in advance. In the image processing unit, the coordinate conversion is performed on the first image information and the second image information. It is preferable that a parallelization process using a table is performed. By so doing, it is possible to correct image information acquired in a state where the first imaging unit and the second imaging unit are not in an ideal parallel posture into an image acquired in a parallel posture. Since the image correction by the parallelization process is performed using a coordinate conversion table prepared in advance, high-speed processing is possible even if the number of pixels of the image data is large. Since high-speed processing is possible, the processing speed of image recognition is not reduced.

  Furthermore, the vehicle forward monitoring device of the present invention is a vehicle forward monitoring device including the image processing device described above, and the distance to the object estimated by the image processing device or the color feature that has been image-recognized. It has the feature of notifying the passenger of the front information with the object.

According to the present invention, by a pair of image pickup means having a their respective one lens, acquires the image as an image and a night vision image and a stereo camera including a color characteristic simultaneously, image recognition stereo ranging The distance to the object can be estimated, and the color feature object can be recognized. If it carries out like this, it can be set as the image processing apparatus in which an equivalent image processing is possible, without using two stereo cameras.

  Therefore, a pair of imaging means can simultaneously acquire an image including a color feature, a night vision image, and an image as a stereo camera, can estimate the distance to the recognized object, and can recognize the color feature object. An image processing apparatus and a vehicle front monitoring apparatus can be provided.

1 is a block diagram illustrating an image processing apparatus according to a first embodiment of the present invention. It is a model front view of the vehicle by which the image processing apparatus of 1st Embodiment is mounted. It is the bird's-eye view which looked at the vehicle carrying the image processing device of a 1st embodiment from the top. FIGS. 4A and 4B are explanatory diagrams showing image correction of the first image information. FIG. 4A is an image image before the image information is corrected, and FIG. 4B is an image image after the correction. It is a flowchart explaining the image processing procedure in 1st Embodiment. It is explanatory drawing which shows the 1st image and 3rd image which were acquired with the 1st imaging means. It is explanatory drawing which shows the 1st image acquired with the 1st imaging means, and the 2nd image acquired with the 2nd imaging means. It is explanatory drawing of the spectral characteristic of a visible light filter and an infrared-light filter. It is an example of an image of visible light. It is an example of an image of near infrared light. It is an example of a far-infrared light image. It is a block diagram of the vehicle front monitoring apparatus provided with the image processing apparatus of 2nd Embodiment. It is a front view of a vehicle provided with the conventional exterior monitoring apparatus.

[First Embodiment]
The image processing apparatus 1 in the first embodiment will be described below.

  FIG. 1 is a block diagram illustrating an image processing apparatus 1 according to the first embodiment. The image processing apparatus 1 includes a camera unit 30, an image recognition unit 40, and a control unit 50, and is mounted on the vehicle so as to capture the front of the vehicle. The control unit 50 supplies power to the camera unit 30 and the image recognition unit (image recognition unit) 40 and outputs a control signal to control each state.

  The camera unit 30 includes a wide-angle lens 11 and a first image sensor 12, and includes a first image capturing unit 10 that obtains first image information, a telephoto lens 21, and a second image sensor 22. The second image pickup means 20 for obtaining the second image information, and the image output section 31 connected to the first image pickup means 10 and the second image pickup means 20 are provided. The image output unit 31 is an electronic circuit that repeatedly acquires the first image information and the second image information at predetermined time intervals and outputs them to the image recognition means 40.

  An image recognition means (image recognition unit) 40 recognizes an object based on the first image information and the second image information, and therefore, an image input unit 41, an image processing unit 42, a distance estimation processing unit 43, and A display image output unit 44. The image information transmitted from the image output unit 31 is input to the image input unit 41, and the image processing unit 42 corrects the acquired image information to acquire the first image and the second image, The third image is cut out from the first image. The image processing unit 42 outputs stereo image data to be sent to the distance estimation processing unit 43 and display image data to be sent to the display image output unit 44.

  The distance estimation processing unit 43 includes the third image cut out from the first image corrected by the image processing unit 42 and the second image corrected by the image processing unit 42 in common. A process for recognizing the stereo distance measurement object and estimating the distance to the stereo distance measurement object is performed.

  The display image output unit 44 adjusts the display image output unit 44 so that the user can easily recognize it, and outputs a display image. Moreover, the distance estimation process part 43 emphasizes the stereo ranging object and performs the process which displays the estimated distance.

  FIG. 2 is a schematic front view of a vehicle on which the image processing apparatus 1 of the present embodiment is mounted. When viewed from the front of the vehicle, it can be visually confirmed that the camera unit 30 is mounted over the windshield. The camera unit 30 includes a wide-angle lens 11 and a telephoto lens 21. The wide-angle lens 11 and the telephoto lens 21 are positioned at the same height and in the left-right direction of the vehicle, and the optical axis of the lens is directed in the front direction of the vehicle. ing. The camera unit 30 includes the first imaging unit 10 and the second imaging unit 20 together with the image output unit 31 on one camera unit substrate (not shown) so that optical adjustment can be easily performed. It is. Further, the image recognition means 40 is disposed inside the vehicle, separated from the camera unit 30.

  Since the image processing apparatus 1 of this embodiment is mounted on a vehicle, the display image output from the display image output unit 44 is visually recognized by the display means 60 disposed on the front side of the driver's seat of the vehicle. The display means 60 is, for example, a liquid crystal display device, and a display screen of a car navigation device or an audio device can be used. Further, the display means 60 may be provided exclusively in the vicinity of the driving steering wheel or on the windshield in front of the driver's seat. It is preferable to correct the dynamic range and gradation in the display image output unit 44 in accordance with the display characteristics of the display means 60 to be used.

  Next, the imaging means of the image processing apparatus 1 of this embodiment will be described.

  In the present embodiment, the first imaging means 10 includes a wide-angle lens 11 and a first imaging element 12 and is configured to be able to collect visible light. On the other hand, the second imaging unit 20 includes a telephoto lens 21 and a second imaging element 22 and is configured to be able to collect infrared light.

  FIG. 3 is an overhead view of a vehicle on which the image processing apparatus 1 of the present embodiment is mounted as viewed from above. Schematically, the horizontal field angle (horizontal field angle A1) of the first imaging unit 10 and the horizontal field angle (horizontal field angle A2) of the second imaging unit 20 are represented.

  The first imaging device 12 used for the first imaging means 10 has a large number of pixels and includes a visible light filter for color photography. This is an RGB filter whose wavelength to be transmitted is a red region, a green region, and a blue region. For example, a solid-state image sensor mounted on a video camera can be used for the first image sensor 12. For example, a solid-state imaging device capable of capturing a horizontal 1920 × vertical 1080 color image at 30 frames / second is used. The wide-angle lens 11 was selected so that a visual field range of 40 degrees could be imaged as the horizontal field angle A1.

On the other hand, the second imaging element 22 used in the second imaging means 20 has a standard number of pixels called VGA and includes an infrared filter that transmits infrared light . For example, using the image can be captured solid-state imaging device monochrome images of horizontal 640 × vertical 480 at 30 frames / sec. The telephoto lens 21 was selected so that a visual field range of 20 degrees could be imaged as the horizontal field angle A2. As shown in FIG. 3, the visual field range of the second imaging unit 20 is within the visual field range of the first imaging unit 10.

<Image processing procedure>
Next, an image processing procedure in the present embodiment will be described. In the image processing apparatus 1, the image recognition unit (image recognition unit) 40 is not affected by the relative mounting posture between the first imaging unit 10 and the second imaging unit 20, and is included in the actual camera posture. A parallelization process (Rectification) is performed to remove the influence of the above. At this time, distortion removal processing is also performed at the same time.

  In general, the ideal arrangement state of the left and right imaging means constituting the stereo camera is a state in which the left and right imaging means are fixed in the same parallel posture. For this reason, the distance calculation process using a stereo camera targets a pair of left and right images taken in a parallel posture.

  The ideal relative mounting orientation (parallel orientation) of a stereo camera is an orientation in which the camera coordinate system X axis (camera right direction) is parallel to the baseline and the camera optical axis (camera coordinate system Z axis) is parallel. That is. The base line in the stereo camera is a straight line connecting the left and right lens centers. The parallelization processing of the image information is achieved by converting the captured image information into an image that would have been captured when the imaging unit was in an ideal parallel posture. Here, the camera coordinate system is a coordinate system possessed by the camera, with the image center as the origin, the right direction is the positive direction of the X axis, the downward direction is the positive direction of the Y axis, and the front direction of the camera is the positive direction of the Z axis. Represents a direction.

  In the image processing apparatus 1, the first imaging unit 10 and the second imaging unit 20 are incorporated in the camera unit 30, and are mechanically adjusted and fixed with respect to the mounting position and inclination. Thereafter, optical relative position data of the second imaging unit 20 with respect to the first imaging unit 10 is acquired by image information acquired by imaging a checkerboard in which white and black squares are alternately arranged. . That is, the relative position of both lenses in the direction of the optical axis (camera coordinate system Z axis), the vertical and horizontal positions on the plane (plane of the camera coordinate system X axis and Y axis) orthogonal to the optical axis (camera coordinate system Z axis) , And the roll, pitch, and yaw of the second imaging means 20 relative to the first imaging means 10 are grasped.

  The relative position data obtained in this way is recorded in advance in the image processing unit 42, and conversion is performed to calibrate the captured image information to an image that would have been captured when the imaging means was in an ideal parallel posture. Applies to parallelization processing.

  At this time, at least the distortion removal processing of the image information captured by the first imaging unit 10 is also performed at the same time. That is, since the imaging means has an image height characteristic unique to the lens, the conversion is performed so that the XY coordinates on the image data coincide with the XY coordinates of the position in the real space. Since the XY coordinate data of the checkerboard is known, distortion removal processing data can be acquired from image information obtained by imaging the checkerboard. Since the first imaging unit 10 has a wider field of view than the second imaging unit 20, the captured image information is an image in which the center peculiar to the wide-angle lens 11 is greatly reflected and the periphery is distorted. If this distortion remains, it becomes complicated to compare the third image with the second image. For this reason, distortion removal processing is performed on the captured image information to convert the image information into a first image without distortion. FIG. 4 is an explanatory diagram illustrating image correction of the first image obtained by imaging the checkerboard. FIG. 4A shows an image before the image is corrected, and FIG. 4B shows an image after the correction. The distortion removal process for the image information captured by the second imaging unit 20 can be performed in the same manner. In this way, even if the first imaging unit 10 and the second imaging unit 20 have unique lens characteristics and are not in an ideal parallel posture, it can be corrected to an ideal image. .

  In the present embodiment, these image corrections are performed by a conversion method called LUT (Look Up Table). In this method, the coordinates of each pixel in the captured image information are rearranged at the coordinate position to be imaged when there is no distortion, and this is a method of creating a coordinate conversion table in advance. In the present embodiment, a coordinate conversion table is created from image information obtained by imaging the checkerboard, and the coordinate conversion table is stored in advance in the coordinate conversion table storage unit 45 of the image processing unit 42. In the image processing unit 42, using the coordinate conversion table for the first image information stored in the coordinate conversion table storage unit 45, parallelization processing and distortion removal processing are performed on the first image information. A first image is acquired. Similarly, the second image information is subjected to a parallelization process and a distortion removal process using a second image information coordinate conversion table to obtain a second image. Note that if one image information is used as a reference, only the other image information needs to be performed for the parallelization processing. Generally, these image corrections can be performed by calculation using a calculation formula for each pixel of image information. However, when the number of pixels increases, the amount of calculation becomes enormous, which causes a problem that the processing speed decreases. In the present embodiment, since this distortion removal processing is performed using a coordinate conversion table stored in advance, arithmetic processing using mathematical formulas is unnecessary, and high-speed processing is possible even if the number of pixels of the image data is large. Since high-speed processing is possible, the processing speed of image recognition is not reduced.

  FIG. 5 is a flowchart illustrating an image processing procedure in the first embodiment. As an image processing procedure in the image recognition unit 40, in step ST1, image information captured by the first imaging unit 10 and the second imaging unit 20 is input to the image processing unit 42, and image information is acquired. In step ST2, the above-described image correction is performed, the first image information is corrected to correct the first image, and the second image information is corrected to acquire the second image.

  Further, in step ST3, a third image corresponding to the common area with the second image is cut out from the acquired first image, and the third image is acquired. The first image picked up by the first image pickup means 10 has a wide field of view since it has a horizontal angle of view of 40 degrees, and includes a common area with the second image picked up by the second image pickup means 20. In other words, since the second image captured by the second imaging unit 20 has a horizontal angle of view of 20 degrees, the visual field range is narrower than that of the first image, and a common area with the first image is formed. Have. Therefore, it is possible to cut out the third image corresponding to the common area with the second image from the first image.

  The image processing unit 42 performs image processing for correcting the first image in step ST2 and cutting out the third image from the first image in step ST3. At this time, the cut out third image is converted into a grayscale image.

  Next, the distance estimation processing unit 43 extracts the feature points of the object included in the third image corresponding to the common area, recognizes the stereo distance measurement object related to the feature points, Processing for estimating the distance from the imaging position of the processing device to the stereo distance measurement object is performed. In step ST4, the feature point of the target object included in the third image is extracted, and the corresponding second image is searched for the feature point shown in the third image. When the corresponding point of the second image corresponding to the feature point shown in the third image is searched, in step ST5, the image position of the feature point in the third image and the image of the corresponding point in the second image The shift amount of the image position is measured based on the position. For a stereo ranging object related to the feature point of the object included in the third image and the corresponding point included in the second image, the distance is determined according to the principle of triangulation by the amount of image position deviation. Calculated. This process is repeated in the same manner for the image information captured at the next timing until the image process is completed in step ST6.

  FIG. 6 is an explanatory diagram showing a first image and a second image. In addition, in order to make it easy to compare the first image and the second image, a description will be given using a general visible light monochrome image. In FIG. 6, the distance from the vehicle to the signal is assumed to be about 50 m. FIG. 6A is an explanatory diagram showing a first image and a third image (F region image) captured by the first imaging means 10. Here, the first image is originally a color image, and the third image is a black and white grayscale image. FIG. 6B is an explanatory diagram showing a second image captured by the second imaging unit 20. An image after the above-described correction processing is shown. In FIG. 6A, image processing for cutting out the image of the F region from the first image is performed corresponding to the common region with the second image. A signal far away in front of the vehicle is commonly included in the second image and the third image.

  FIG. 7 is an explanatory diagram showing a first image and a second image. Similar to FIG. 6, a visible light image is used for easy comparison. In FIG. 7, the distance from the vehicle to the signal is assumed to be about 15 m. FIG. 7A is an explanatory diagram illustrating a first image captured by the first imaging unit 10. FIG. 7B is an explanatory diagram illustrating a second image captured by the second imaging unit 20. In Fig.7 (a), it turns out that the signal ahead of a vehicle is contained in the 1st image. However, in FIG. 7B, the signal is not included in the second image. That is, when the distance to the signal shown in FIG. 6 is 50 m, the signal can be recognized by both the first imaging means 10 and the second imaging means 20, but the distance to the signal is up to 15 m. In FIG. 7 that is approaching, the signal can be recognized only by the first imaging means 10 alone. This is because the visual field range of the first imaging means 10 is wide, but the image processing apparatus 1 uses the first image for objects around the visual field range, such as a signal when approaching the front of the vehicle. Can continue to recognize.

  6 is recognized as an object common to the third image and the second image as in the signal shown in FIG. 6, and the relative displacement amount of the image position of the object in the left and right images is measured, and the image position displacement is detected. The distance is estimated by the quantity. Once the feature point recognized as the target object deviates from the second image captured in the visual field range of the second imaging unit 20, the feature point is recognized in the first image having a wide visual field range. You can continue. Therefore, the extracted object is continuously tracked. Thereby, even after deviating from the visual field range of the second imaging means 20, it is possible to estimate the distance to the object based on how the image position of the feature point changes with time in the first image. Therefore, if the information of the stereo ranging object included in the second image and the third image is related on the first image, a signal or a sign located at a long distance in front of the vehicle is displayed at a short distance in front of the vehicle. You can continue to recognize even when approaching.

  Furthermore, since the first image sensor 12 is an image sensor for color photography, it is possible to detect the color feature of the object from the first image that is a color image. In this case, the image processing unit 42 detects a color feature included in the first image, and performs a process of recognizing the object as a color feature object related to the color feature. For this reason, it is possible to quickly recognize an object whose color feature in visible light is extracted from the first image. The color feature of the feature point recognized as the stereo distance measurement object common to the third image and the second image can be detected and the object can be continuously recognized. In addition, if the color feature object related to the color feature in visible light by the first image is the same as the stereo distance measurement object common to the third image and the second image, The distance to the object can be estimated.

  If the color feature of the signal tracked in FIG. 7A is recognized, when the signal changes from a red signal to a blue signal, it is recognized as a change in the color feature, and the fact that the color has changed is output to the display image. Or sound a signal.

  Therefore, it is suitable for the image processing apparatus 1 that is mounted on a vehicle and images the front of the vehicle. In the image processing apparatus 1, the first imaging unit 10 and the second imaging unit 20 are mounted so as to image the front of the vehicle. The first image is a short-distance image including a first long-distance image in front of the vehicle, and the second image is a second long-distance image in front of the vehicle, and the stereo measurement included in the second image. Distance object information can be correlated on the first image. Since the first imaging element 12 is the wide-angle lens 11 and the depth of focus is relatively deep, a distant object can be imaged in the first image. Since signals and signs located at a short distance in front of the vehicle are captured within the field of view, automatic signal recognition and automatic sign recognition are possible.

  Further, if the camera unit 30 in which the first image pickup means 10 and the second image pickup means 20 are arranged and incorporated on one camera unit substrate is arranged so as to image the front of the vehicle through the windshield of the vehicle. Easy to attach to the vehicle. Furthermore, since the first imaging means 10 and the second imaging means 20 are arranged on the same substrate as compared with the arrangement where the left and right cameras are separated, optical adjustment is performed before being attached to the vehicle. It is possible to leave. Therefore, it is easy to mount, and there is an effect that the optical calibration after the mounting is unnecessary as compared with the arrangement of the left and right cameras separately. In the small camera unit 30, the distance between the left and right lenses (base line length) is set to about 65 mm. On the other hand, when it is desired to ensure the distance estimation accuracy at a longer distance, the base line length may be about 300 mm.

  In the present embodiment, each of the first imaging unit 10 and the second imaging unit 20 uses only one lens, so that there are only two lenses. Further, the imaging surface of the imaging device is an aggregate of a large number of pixels, and in order to improve measurement accuracy, it is necessary to increase the number of pixels, but it is not necessary to correspond to four lenses. The increase in the number of pixels of the imaging element affects not only the size of the imaging surface but also the system scale, such as an increase in the amount of electronic information processing corresponding to the number of pixels and an increase in the processing speed required for it. Therefore, the image processing apparatus 1 that only acquires an image corresponding to a pair of stereo cameras has a minimum number of pixels of the image sensor and is inexpensive as a system.

  Hereinafter, the effect by having set it as this embodiment is demonstrated.

  The first imaging means 10 can capture the first image information in the visible light wavelength band. The second image pickup unit 20 can pick up an image of the visual field range including the common field and the visual field of the first image pickup unit 10, and can pick up the second image information in the infrared wavelength band. Thereby, the image processing apparatus 1 corrects the first image information and the second image information to acquire the first image and the second image, and detects the color feature of the object by the first image. Then, the image of the object is recognized as the color feature object related to the color feature. Further, based on the first image and the second image, the object can be recognized as a stereo distance measurement object, and the distance from the imaging position of the image processing apparatus 1 to the stereo distance measurement object can be estimated. . Accordingly, an image including a color feature and an image as a stereo camera can be simultaneously acquired by a pair of imaging units each including a single lens, and the distance to the recognized stereo ranging object can be estimated. The feature object can be recognized as an image. Therefore, an image processing apparatus capable of performing equivalent image processing without using two stereo cameras can be provided.

In the above description , the first image pickup device 12 is a color image pickup device including a visible light filter. However, in practice, an image pickup device having sensitivity to near infrared light is used. used, the first image pickup means 10, to allow focusing the near-infrared light, to capture the first image information in the visible and near-infrared light.

In the first image sensor 12, an infrared light filter may be prepared in addition to the visible light filter, and pixels for imaging infrared light blocked by the visible light filter may be arranged. FIG. 8 is an explanatory diagram of spectral characteristics of the visible light filter and the infrared light filter. An RGB filter with a wavelength of red, green, and blue, respectively, and an IR filter that transmits near infrared light with a wavelength of 700 nm or more and reflects or absorbs visible light with a wavelength of 680 nm or less. is there. By so doing, the first imaging means 10 can capture the first image information with visible light and near infrared light .

  FIG. 9 shows an example of visible light images. FIG. 10 shows an example of near-infrared light imaged simultaneously with FIG. In FIG. 9, the far field and the lateral field of view are dark and difficult to visually recognize, but in FIG. 10, the far field and the lateral field of view are bright and an image called “night vision” is obtained. Therefore, even at night or in a tunnel where the amount of visible light is small, it can be supplemented with an image acquired with infrared light, and night vision stereo distance measurement performance is improved.

The second image sensor 22 includes an infrared filter for acquiring an image in the infrared wavelength band, and has sensitivity to infrared light . Na us, without using an infrared filter, image may be obtained without cutting the sensitivity of the entire wavelength band, but if it contains an image of the visible light wavelength band, specific to the image of the infrared light wavelength range Therefore, it is preferable to provide an infrared filter according to the purpose of use.

Second image sensor 22 is a far-infrared imaging device for detecting a far-infrared light. FIG. 11 shows an example of far-infrared light imaged simultaneously with FIG. Compared to FIG. 9 and FIG. 10, it can be seen that a far field of view is easily visually recognized. Further, when an image with far-infrared light is acquired as the second image, the human body that emits far-infrared light is also viewed brightly, so that the sensitivity for detecting the human body at night is further improved.

  In addition, since it is an image process that does not include extra peripheral image data in the image information necessary for stereo distance measurement, there is a secondary effect that the processing speed is high.

  The image processing apparatus 1 can be used for a security camera, a self-propelled robot, or the like in addition to being mounted on a vehicle.

[Second Embodiment]
An image processing apparatus 2 according to the second embodiment of the present invention will be described.

  FIG. 12 is a block diagram of the vehicle front monitoring apparatus 100 including the image processing apparatus 2 of the second embodiment. The parts common to the image processing apparatus 1 of the first embodiment are given the same reference numerals and the description thereof is omitted.

  In the second embodiment, in the image processing apparatus 2, in addition to passing display information from the distance estimation processing unit 43 to the display image output unit 44, the front information is transmitted to the occupant with the estimated distance to the target object. Is sent to the warning determination unit 70.

  In the warning determination unit 70, the distance estimation processing unit 43 estimates the distance to the object ahead of the vehicle and the object continuous tracking data recognized by the display image output unit 44. A decision is made to issue a warning. For example, an alarm is output to the warning means 71 that issues a warning before colliding with a preceding vehicle in front or an obstacle, or a command for operating the brake of the vehicle as a risk avoidance operation is output to the risk avoidance operation means 72. To do.

  Similar to the image processing device 1 of the first embodiment, the display data may be output from the display image output unit 44 to the display means 60 using a display screen such as a car navigation device.

  Hereinafter, the effect by having set it as this embodiment is demonstrated.

  The first imaging means 10 can capture the first image information in the visible light wavelength band in front of the vehicle. The second imaging means 20 can image the visual field range including the visual field of the first image and the common area, and can capture the second image information in the infrared light wavelength band. As a result, the image processing apparatus 2 corrects the first image information and the second image information to obtain the first image and the second image, and detects the color feature of the object by the first image. Then, the image of the object is recognized as the color feature object related to the color feature. Further, based on the first image and the second image, the object can be recognized as a stereo distance measurement object, and the distance from the imaging position of the image processing device 2 to the stereo distance measurement object can be estimated. . Accordingly, an image including a color feature and an image as a stereo camera can be simultaneously acquired by a pair of imaging units each including a single lens, and the distance to the recognized stereo ranging object can be estimated. The feature object can be recognized as an image. In this way, the vehicle forward monitoring device 100 including the image processing device 2 estimates the distance to the stereo distance measuring object located at a long distance in front of the vehicle, and uses it for a collision prevention warning and cruise control. Can do. Also, by recognizing the color feature object as an image, automatic signal recognition and automatic sign recognition are possible. Therefore, it is possible to provide a vehicle forward monitoring apparatus that can perform equivalent image processing without using two stereo cameras.

  For example, since a signal or a sign can be recognized, the warning means 71 can be used for driving assistance that issues a voice or warning. Further, when the driver does not recognize the red light or the regulation sign, the warning by the warning means 71 and the brake operation by the danger avoiding operation means 72 can be performed.

  Further, if the first image is a short-distance image with a wide visual field range, lane detection is possible, and meandering detection, warning, and control are also possible. Therefore, it is possible to simultaneously acquire an image of a wide visual field range and an image as a stereo camera by a pair of imaging means each having one lens, and estimate the distance to the stereo distance measurement object recognized as an image, Warning and control with a wide field of view image is possible.

  Further, once the feature point recognized as the target object deviates from the second image captured in the visual field range of the second imaging unit 20, the feature point is detected in the first image having a wide visual field range. Can continue to recognize. Therefore, the extracted object can be continuously tracked within the visual field range of the display image data sent to the display image output unit 44.

  As described above, the image processing apparatus 2 according to the present embodiment includes the first image in the visible light wavelength band and the second image in the infrared light wavelength band having a common area with the first image. get. Thereby, the stereo image of this 2nd image and the 3rd image cut out corresponding to the common area | region with the 2nd image among 1st images can be acquired. Furthermore, the objects included in the second image and the third image can be recognized as a stereo distance measurement object by extracting feature points, and the distance to the stereo distance measurement object can be estimated. In addition, the color feature of the object is detected from the first image, and the object is recognized as a color feature object associated with the color feature. Therefore, it can be set as the vehicle front monitoring apparatus which can perform equivalent image processing, without using two stereo cameras.

  Therefore, a pair of imaging means can simultaneously acquire an image including color features, a night vision image, and an image as a stereo camera to estimate the distance to the recognized stereo distance measurement object, and the color feature object. It is possible to provide an image processing device and a vehicle front monitoring device that can recognize the image of the vehicle.

  The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the scope of the invention. For example, the present invention can be modified as follows, and these embodiments also belong to the technical scope of the present invention.

  Wireless communication means may be provided in the image processing apparatus so as to perform wireless communication with a remote control device having the display means 60.

  Alternatively, the display unit 60 may be integrally provided in the image processing apparatus.

DESCRIPTION OF SYMBOLS 1, 2 Image processing apparatus 10 1st imaging means 11 Wide angle lens 12 1st imaging device 20 2nd imaging means 21 Telephoto lens 22 2nd imaging device 30 Camera unit 31 Image output part 40 Image recognition means (image recognition) unit)
DESCRIPTION OF SYMBOLS 41 Image input part 42 Image processing part 43 Distance estimation process part 44 Display image output part 45 Coordinate conversion table memory | storage part 50 Control means 60 Display means 70 Warning determination part 71 Warning means 72 Risk avoidance operation means 100 Vehicle forward monitoring apparatus

Claims (4)

A first imaging unit capable of imaging the front of the vehicle with visible light and near infrared light by receiving visible light and near infrared light collected through a lens having a relatively wide angle of view ;
By receiving far-infrared light collected through a lens having a relatively narrow angle of view, the field-of-view range including the field of view of the first imaging means and the common area is also far-infrared light in the front of the vehicle. A second imaging means capable of imaging;
Based on the second image information captured by the first image information and said second image pickup means which is captured by the first imaging means includes an image recognition means for recognizing an object, a,
The image recognition means corrects the first image information to obtain a first image, and corrects the second image information to obtain a second image. Further, from the first image, the image recognition means A black and white third image corresponding to a common area with the second image is cut out, a color feature of the object is detected from the first image, and a color feature object related to the color feature is detected. An image processing unit in which processing for recognizing the object is performed;
Extracting a feature point of the object included in the third image, recognizing the object as a stereo ranging object, searching for a corresponding point corresponding to the feature point from the second image, A distance estimation processing unit that performs a process of estimating the distance from the feature point of the third image and the corresponding point of the second image to the object,
Even after the extracted feature point is out of the range of the second and third images, the object continues to be recognized as the color feature object in the first image, and if necessary, a vehicle occupant An image processing apparatus characterized by notifying information of the object.   The image processing apparatus according to claim 1, wherein the first imaging unit and the second imaging unit are arranged on the same substrate. The image processing unit has a coordinate conversion table storage unit in which a coordinate conversion table is stored in advance,
2. The image according to claim 1, wherein a parallelization process using the coordinate conversion table is performed on the first image information and the second image information in the image processing unit. Processing equipment.   A vehicle forward monitoring device comprising the image processing device according to any one of claims 1 to 3 in a vehicle, wherein the distance to the object estimated by the image processing device or the image recognized A vehicle forward monitoring device that informs passengers of forward information with a color feature object.