JP7117876B2 - Vehicle image processing device, vehicle image processing method, and vehicle image processing program - Google Patents

Description

The present invention relates to a vehicle image processing device, a vehicle image processing method, and a vehicle image processing program for processing an image captured by an imaging device mounted on a vehicle.

2. Description of the Related Art Liquid such as raindrops sometimes adheres to the lens surface of an imaging device mounted on the rear or side of a vehicle for imaging the outside of the vehicle. In this case, the lens effect of the liquid causes distortion in the captured image, making it impossible to obtain an original distortion-free image. Therefore, the image processing apparatus according to Patent Document 1 estimates the adhesion position of the rain and snow on the vehicle window based on the feature amount of the rain and snow on the captured images obtained by continuously capturing and the measured falling speed characteristics. Then, the rain/snow adhering portion of the captured image is corrected.

JP 2004-212184 A

If it starts to rain and snow while the vehicle is parked, there is a high possibility that the rain and snow will have already adhered to the imaging device when the imaging device starts up and starts imaging. In that case, the image processing apparatus according to Patent Document 1 uses a captured image in which rain and snow are reflected, and there is a problem that a captured image without rain and snow and without distortion cannot be reproduced. .

SUMMARY OF THE INVENTION It is an object of the present invention to solve the above problems, and to reproduce a distortion-free captured image even when the captured image is distorted.

A vehicular image processing apparatus according to the present invention is an image pickup apparatus mounted on a vehicle, which picks up an image of the scene outside the vehicle including the light irradiated on the vehicle or the marker member made of the retroreflective material installed on the vehicle. A storage unit that stores an undistorted image captured by the device as a reference image, and an image processing unit that corrects distortion of the image captured by the imaging device using the reference image already stored in the storage unit. and a detection unit that detects adhesion of liquid to the lens of the imaging device, and when the detection unit detects adhesion of liquid, the image processing unit uses the reference image stored in the storage unit, It is characterized by correcting the distortion of the image captured by the imaging device .

According to the present invention, a non-distorted image captured by an imaging device mounted on a vehicle and capturing a scene outside the vehicle including a part of the vehicle is used as a reference image, and the distortion of the image captured by the imaging device is corrected. Since the correction is made, even if the captured image is distorted, the captured image without distortion can be reproduced by image processing.

1 is a block diagram showing a configuration example of a vehicle image processing device according to Embodiment 1; FIG. FIG. 2 is a top view of a vehicle showing an example of vehicle installation of the rear camera according to Embodiment 1; 3A is a top view of a vehicle showing an example of installation of a side camera in a vehicle according to Embodiment 1, and FIG. 3B is a side view of the vehicle. 4A and 4B are diagrams showing an example of a captured image without distortion and a captured image with distortion in Embodiment 1. FIG. 4 is a flow chart showing an operation example of the vehicle imaging device according to Embodiment 1; FIG. 11 is a block diagram showing a configuration example of a vehicle image processing apparatus according to Embodiment 2; 9 is a flowchart showing an operation example of the vehicle image processing device according to Embodiment 2; It is a figure which shows the hardware structural example of the image processing apparatus for vehicles which concerns on each embodiment.

Embodiment 1.
FIG. 1 is a block diagram showing a configuration example of a vehicle image processing apparatus 10 according to Embodiment 1. As shown in FIG. The vehicle image processing apparatus 10 includes an image acquisition unit 11, an image processing unit 12, and a storage unit 13, and is mounted on a vehicle. This vehicle image processing device 10 is connected to a rear camera 1, a side camera 2, and a display device 3 mounted on the same vehicle.

FIG. 2 is a top view of a vehicle showing an installation example of the rear camera 1 according to the first embodiment. The rear camera 1 is installed in the rear portion 21 of the vehicle and captures an imaging range 1a indicated by a dashed line. This imaging range 1a includes the scenery behind the vehicle and a part of the rear portion 21 of the vehicle such as the bumper. The rear camera 1 outputs an image obtained by imaging the imaging range 1a to the image acquisition section 11 of the vehicle image processing device 10 .

3A is a top view of a vehicle showing an installation example of the side camera 2 according to Embodiment 1, and FIG. 3B is a side view of the vehicle. The side camera 2 is installed at the position of the side mirror on the side portion 31 of the vehicle instead of the side mirror, and captures an imaging range 2a indicated by a broken line. The imaging range 2a includes the scenery obliquely behind the vehicle and a part of the vehicle side portion 31 such as side windows, doors, tires, and the vehicle body. The side camera 2 outputs an image obtained by imaging the imaging range 2a to the image acquisition section 11 of the vehicle image processing device 10 .
In Embodiment 1, the operation of the vehicle image processing apparatus 10 is described using the side camera 2 installed on one side of the vehicle as an example. The vehicle image processing apparatus 10 also operates in the same manner.

The image acquisition unit 11 shown in FIG. 1 acquires images from the rear camera 1 and the side camera 2 and outputs the images to the image processing unit 12 .

The image processing unit 12 corrects the distortion of the images captured by the rear camera 1 and the side camera 2 using the reference image already stored in the storage unit 13 , and outputs the corrected image to the display device 3 . Image distortion is caused by liquid such as raindrops adhering to the lenses of the rear camera 1 and the side camera 2 . Details of the correction method will be described later.

The storage unit 13 stores an undistorted image of the imaging range 1a captured in advance by the rear camera 1 and an undistorted image of the imaging range 2a captured in advance by the side camera 2 as reference images. there is The reference image is, for example, an image captured when the rear camera 1 and the side camera 2 are attached to the vehicle, and no liquid adheres to the lenses of the rear camera 1 and the side camera 2 .

The display device 3 receives the distortion-corrected images of the rear camera 1 and the side camera 2 from the image processing unit 12 and displays them on the screen. This display device 3 functions, for example, as an electronic mirror.

FIG. 4 is a diagram showing an example of a captured image 41 without distortion and a captured image 42 with distortion according to the first embodiment. In FIG. 4, an image in which a grid pattern is captured is illustrated so that the presence or absence of distortion can be easily recognized.
There is a high possibility that raindrops will adhere to the lenses of the rear camera 1 and the side camera 2 installed outside the vehicle when it rains. When raindrops adhere to the lens, the raindrops have an effect like a lens, and a captured image 41 without distortion should be captured under normal conditions, but the lens effect of the raindrops causes distortion like a captured image 42. - 特許庁Therefore, if raindrop countermeasures are not taken, a distorted captured image 42 captured by the rear camera 1 and the side camera 2 during rain will be displayed on the display device 3 .

Therefore, the vehicle image processing apparatus 10 utilizes the fact that the rear camera 1 and the side camera 2 capture an image of a part of the vehicle such as the bumper. The vehicular image processing apparatus 10 causes the storage unit 13 to store, as a reference image, the captured image 41 without distortion, which is captured in a normal state in which no liquid adheres to the lens. When the liquid adheres to the lens, distortion occurs as in the captured image 42, so the positions of the bumpers and the like reflected in the distorted captured image 42 deviate from the positions of the bumpers and the like reflected in the reference image.

The image processing unit 12 utilizes the difference between the position of the bumper or the like reflected in the reference image and the position of the bumper or the like reflected in the distorted captured image 42, and performs image processing calculation to bring the distorted captured image 42 closer to the reference image. Thereby, the distortion of the captured image 42 is corrected. In the example of FIG. 2, the image processing unit 12 acquires the absolute position of the bumper contour 22 in the reference image captured by the rear camera 1 during normal operation and stored in the storage unit 13, and the image captured by the rear camera 1 during rainfall. Based on the amount of change from the absolute position of the outline of the bumper in the imaged image acquired by the unit 11, the degree of distortion of the entire imaged image is calculated. Then, the image processing unit 12 distorts the entire captured image so as to cancel out the calculated distortion. 3A and 3B, the image processing unit 12 performs an absolute calculation of the contours 32 of the side windows, doors, tires, and vehicle body in the reference image captured by the side camera 2 during normal operation and stored in the storage unit 13. Distortion of the entire captured image based on the amount of change between the position and the absolute position of the outline of the side window, door, tires, and vehicle body in the captured image captured by the side camera 2 during rain and acquired by the image acquisition unit 11. Calculate additions and subtractions. Then, the image processing unit 12 distorts the entire captured image so as to cancel out the calculated distortion. Accordingly, the image processing unit 12 can relatively easily reproduce a captured image without distortion.

2, 3A and 3B, the image processing unit 12 is configured to use the positions of the contours 22 and 32 of the vehicle for distortion correction. good too. For example, the image processing unit 12 may use the position of the paint design of the vehicle body. Further, for example, when light having a light distribution such as a lattice pattern as shown in FIG. may If a light source that emits invisible light such as infrared rays is used, the design of the vehicle will not be affected. Further, for example, when an invisible marker such as a grid pattern made of retroreflective material is installed on the vehicle, the image processing unit 12 may use the position of this invisible marker.

FIG. 5 is a flow chart showing an operation example of the vehicle image processing apparatus 10 according to the first embodiment. The vehicle image processing device 10 repeatedly performs the operation shown in the flowchart of FIG. 5 while the display device 3 is activated, for example. The flowchart will be described below using an example in which the vehicle image processing device 10 corrects the distortion of the image captured by the rear camera 1 .

In step ST<b>11 , the image acquisition section 11 acquires a captured image from the rear camera 1 and outputs it to the image processing section 12 .

In step ST12, the image processing unit 12 detects the change in the position of the bumper or the like appearing in the captured image of the rear camera 1 received from the image acquiring unit 11 and the position of the bumper appearing in the reference image stored in the storage unit 13. Based on the amount, the distortion of the captured image of the rear camera 1 is corrected. Then, the image processing unit 12 outputs the distortion-corrected captured image of the rear camera 1 to the display device 3 for display.

As described above, the vehicle image processing apparatus 10 according to Embodiment 1 includes the storage unit 13 and the image processing unit 12 . The storage unit 13 stores, as a reference image, an image without distortion captured by the rear camera 1 and the side camera 2 mounted on the vehicle and capturing a scene outside the vehicle including a part of the vehicle. The image processing unit 12 uses the reference image stored in the storage unit 13 to correct the distortion of the images captured by the rear camera 1 and the side camera 2 . With this configuration, the vehicle image processing apparatus 10 can reproduce a distortion-free captured image by image correction even if the captured image is already distorted when the operation of the vehicle image processing apparatus 10 is started.
In addition, since the vehicle image processing apparatus 10 only performs image processing of the captured image, there is no need for a mechanical mechanism such as wiping or blowing off the liquid on the lens that causes distortion. costs and mechanical failure factors can be eliminated.
Further, it is conceivable to prevent the occurrence of distortion in captured images by applying a hydrophilic treatment to the lens surface, but the hydrophilic treatment deteriorates over time. In this case, by combining the hydrophilic treatment of the lens surface and the image processing by the vehicle image processing apparatus 10, distortion occurring in the captured image can be corrected even after the hydrophilic treatment has deteriorated over time.

Further, the image processing unit 12 of Embodiment 1 uses the position of the contour of a part of the vehicle reflected in the reference image and the position of the contour reflected in the image captured by the rear camera 1 and the side camera 2 to , corrects the distortion of the image captured by the rear camera 1 and the side camera 2 . With this configuration, the vehicular image processing apparatus 10 can correct image distortion by utilizing the fact that the rear camera 1 and the side camera 2 capture an image of a part of the vehicle such as a bumper. Therefore, there is no need for a dedicated structure or the like used as a reference in distortion correction.

Further, the reference image stored in the storage unit 13 of Embodiment 1 is captured with the rear camera 1 and the side camera 2 attached to the vehicle. The image processing unit 12 corrects the distortion of the image by using the image actually captured by the rear camera 1 and the side camera 2 as the reference image instead of correcting the distortion of the image by using the reference image prepared in advance. . As a result, even if the imaging ranges 1a and 2a of each vehicle deviate due to misalignment when the rear camera 1 and the side camera 2 are attached, the position of the part of the vehicle reflected in the reference image and the normal image can be detected. coincides with the position of a part of the vehicle reflected in the captured image at that time. Therefore, the vehicle image processing apparatus 10 can accurately correct the distortion.

Embodiment 2.
FIG. 6 is a block diagram showing a configuration example of the vehicle image processing device 10 according to the second embodiment. A vehicle image processing apparatus 10 according to the second embodiment has a configuration in which a detection unit 14 is added to the vehicle image processing apparatus 10 according to the first embodiment shown in FIG. In FIG. 6, parts identical or corresponding to those in FIG.

The detection unit 14 detects adhesion of liquid to the lens of the rear camera 1 and adhesion of liquid to the lens of the side camera 2, and notifies the image processing unit 12 when the adhesion of liquid is detected. The detection unit 14 detects the position of the part of the vehicle reflected in the reference image stored in the storage unit 13 and the position of the part of the vehicle reflected in the image captured by the rear camera 1 and the side camera 2. If there is a difference, it is determined that liquid such as raindrops adheres to the lens, and if there is no difference, it is determined that liquid does not adhere to the lens. Note that the detection unit 14 does not need to determine that the liquid does not adhere to the lens only when the position of the part of the vehicle in the reference image exactly matches the position of the part of the vehicle in the captured image. For example, the detection unit 14 may be provided with a threshold in advance in consideration of the amount of vibration of the rear camera 1 and the side camera 2 that vibrate due to vibration during vehicle travel. In this case, if the amount of change between the position of the part of the vehicle in the reference image and the position of the part of the vehicle in the captured image is less than the threshold value, the detection unit 14 determines that no liquid is attached to the lens. If the amount of change is equal to or greater than the threshold value, it is determined that liquid is attached to the lens.

FIG. 7 is a flow chart showing an operation example of the vehicle image processing apparatus 10 according to the second embodiment. The vehicle image processing apparatus 10 repeatedly performs the operation shown in the flowchart of FIG. 7 while the display device 3 is activated, for example. The flowchart will be described below using an example in which the vehicle image processing device 10 corrects the distortion of the image captured by the rear camera 1 .

In step ST<b>11 , the image acquisition section 11 acquires a captured image from the rear camera 1 and outputs it to the image processing section 12 and the detection section 14 .

In step ST21, the detection unit 14 determines that there is a difference between the position of the bumper or the like appearing in the captured image of the rear camera 1 received from the image acquisition unit 11 and the position of the bumper appearing in the reference image stored in the storage unit 13. If so, it is determined that liquid is attached to the lens of the rear camera 1 (step ST21 "YES"), and the process proceeds to step ST12. In step ST12, the image processing unit 12 detects the change in the position of the bumper or the like appearing in the captured image of the rear camera 1 received from the image acquiring unit 11 and the position of the bumper appearing in the reference image stored in the storage unit 13. Based on the amount, the distortion of the captured image of the rear camera 1 is corrected. Then, the image processing unit 12 outputs the distortion-corrected captured image of the rear camera 1 to the display device 3 for display.

On the other hand, when there is no difference between the position of the bumper or the like appearing in the captured image of the rear camera 1 received from the image acquiring unit 11 and the position of the bumper appearing in the reference image stored in the storage unit 13, the detecting unit 14 , it is determined that no liquid adheres to the lens of the rear camera 1 (step ST21 "NO"). In this case, the image processing unit 12 outputs the captured image of the rear camera 1 received from the image acquisition unit 11 to the display device 3 as it is without correcting the distortion, and causes the display device 3 to display the image.

As described above, the vehicular image processing apparatus 10 according to the second embodiment includes the detection unit 14 that detects adhesion of liquid to the rear camera 1 and the side camera 2 . When the detection unit 14 detects the adhesion of liquid, the image processing unit 12 uses the reference image stored in the storage unit 13 to correct the distortion of the images captured by the rear camera 1 and the side camera 2 . With this configuration, the vehicle image processing apparatus 10 needs to perform correction only when the captured image is distorted, so the processing load for distortion correction can be reduced.

Finally, the hardware configuration of the vehicle image processing apparatus 10 according to each embodiment will be described.
FIG. 8 is a diagram showing a hardware configuration example of the vehicle image processing apparatus 10 according to each embodiment. The image acquisition unit 11 in the vehicle image processing device 10 is a camera interface 100 . The camera interface 100 outputs image data of the rear camera 1 and the side camera 2 to the processor 101 . Functions of the image processing unit 12 and the detection unit 14 in the vehicle image processing apparatus 10 are realized by the processor 101 that executes programs stored in the memory 102 . The storage unit 13 in the vehicle image processing device 10 is the memory 102 .

The functions of the image processing unit 12 and the detection unit 14 in the vehicle image processing device 10 are realized by software, firmware, or a combination of software and firmware. Software or firmware is written as a program and stored in memory 102 . The processor 101 reads out and executes programs stored in the memory 102 to achieve the functions of each unit. That is, the vehicular image processing apparatus 10 includes a memory 102 for storing a program that, when executed by the processor 101, results in the execution of the steps shown in the flowchart of FIG. 5 and the like. It can also be said that this program causes a computer to execute the procedures or methods of the image processing unit 12 and the detection unit 14 .

Here, the processor 101 is a CPU (Central Processing Unit), a DSP (Digital Signal Processor), a processing device, an arithmetic device, a microprocessor, or the like.
The memory 102 may be a nonvolatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), or flash memory, or a hard disk or flexible disk. or an optical disc such as a CD (Compact Disc) or a DVD (Digital Versatile Disc).

It should be noted that, within the scope of the present invention, it is possible to freely combine each embodiment, modify any component of each embodiment, or omit any component of each embodiment.

In the above description, the rear camera 1 and the side camera 2 are exemplified as imaging devices, but the imaging device is not limited to these cameras, but is mounted on a vehicle and captures the scenery outside the vehicle including a part of the vehicle. If it is
Also, the imaging device may be installed outside the vehicle or inside the vehicle. For example, when an imaging device is installed inside a vehicle and images the scenery behind the vehicle including the bumper through the rear window, liquid such as raindrops adhering to the rear window causes distortion in the captured image. In this case, the vehicle image processing apparatus 10 corrects the distortion of the captured image caused by the liquid adhering to the rear window.
Further, in the above description, the display device 3 functioning as an electronic mirror was exemplified as an application of the captured image distortion-corrected by the vehicle image processing device 10, but the application of the captured image is not limited to the electronic mirror. It may be used for detecting obstacles around the vehicle.

1 rear camera (imaging device) 1a, 2a imaging range 2 side camera (imaging device) 3 display device 10 vehicle image processing device 11 image acquisition unit 12 image processing unit 13 storage unit 14 detection unit , 21 vehicle rear, 22, 32 contour, 31 vehicle side, 41, 42 captured image, 100 camera interface, 101 processor, 102 memory.

Claims (5)

An imaging device mounted on a vehicle, which is imaged by an imaging device that captures the scenery outside the vehicle including the light irradiated on the vehicle or the marker member made of the retroreflective material installed on the vehicle, without distortion. a storage unit that stores an image as a reference image;
an image processing unit that corrects distortion of an image captured by the imaging device using the reference image stored in the storage unit ;
A detection unit that detects adhesion of liquid to the lens of the imaging device,
The image processing unit corrects distortion of the image captured by the imaging device using the reference image stored in the storage unit when the detection unit detects the adhesion of the liquid. A vehicle image processing device. The image processing unit determines the position of the light irradiated to the vehicle reflected in the reference image or the position of the marker member made of a retroreflective material installed on the vehicle and the position of the marker member reflected in the image captured by the imaging device. 2. The distortion of the image picked up by the imaging device is corrected by using the light irradiated to the vehicle or the position of a marker member made of a retroreflective material installed on the vehicle. The vehicle image processing device described . 3. The vehicular image processing device according to claim 1, wherein the reference image stored in the storage unit is captured with the imaging device attached to the vehicle. . An imaging device mounted on a vehicle, which captures an image taken by an imaging device that captures a scene outside the vehicle including light irradiated on the vehicle or a marker member made of a retroreflective material installed on the vehicle. A vehicle image processing method comprising:
an image processing unit correcting the distortion of the image captured by the imaging device, using the undistorted image captured by the imaging device and stored in the storage unit as a reference image ;
a detection unit detecting adhesion of liquid to the lens of the imaging device;
The image processing unit corrects distortion of the image captured by the imaging device using the reference image stored in the storage unit when the detection unit detects the adhesion of the liquid. An image processing method for a vehicle. An imaging device mounted on a vehicle, which is imaged by an imaging device that captures the scenery outside the vehicle including the light irradiated on the vehicle or the marker member made of the retroreflective material installed on the vehicle, without distortion. a storage procedure for storing the image as a reference image;
an image processing procedure for correcting distortion of an image captured by the imaging device using the reference image stored by the storage procedure;
A vehicle image processing program that causes a computer to execute a detection procedure for detecting adhesion of liquid to the lens of the imaging device ,
The image processing procedure is characterized in that, when adhesion of liquid is detected by the detection procedure, distortion of the image captured by the imaging device is corrected using the reference image stored by the storage procedure. image processing program for vehicles.

Images