JP2000123151A - Image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent an erroneous recognition caused by the influence of the wiper of an automobile, to improve the safety and reliability of a system and to efficiently perform image processing by detecting the operation/position of the wiper from an image and excluding this wiper from the image. SOLUTION: An image processor fetches an image 110 to be processed and performs the difference processing of a present image 101 and an image 102 preceding to this present image 101 for one frame. As a result of this difference processing, a difference image 103 can be provided. Next, based on the difference image 103, the image processor discriminates whether the present image 101 is a wiper frame or not. When the present image 101 is discriminated as the wiper frame as a result of wiper frame discriminating processing, concerning the present image 101, real processing for wiper frame is executed. This real processing for wiper frame excludes the image 101 of the wiper frame since that image 101 exerts an adverse influence upon image processing.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing method for processing an image captured by an on-vehicle camera in order to perform road surface condition recognition, obstacle detection, tracking of a preceding vehicle, and the like in an automatic driving / running assist system for an automobile. It is about.

[0002]

2. Description of the Related Art Conventionally, in an automatic driving / running assist system of an automobile, a rearview mirror (rear mirror) of the automobile
An in-vehicle camera attached to the back of the vehicle captures an image of the front of the vehicle, and the captured image is processed by an image processing device to perform road surface condition recognition, obstacle detection, forward vehicle tracking, and the like.

[0003] Here, when it rains, the driver operates the wiper of the car, but when the wiper is imaged by the onboard camera, the image processing device erroneously detects the wiper as an obstacle. This may adversely affect image processing, such as image processing.

Therefore, conventionally, the image processing apparatus obtains the fact that the wiper is operating from mechanical and electrical information linked to the switch of the wiper, thereby stopping the image processing during the operation of the wiper. Alternatively, the position of a wiper that is mechanically operating is detected, and an image or a portion of the image on which the wiper is photographed is excluded by image processing.

[0005]

However, when image processing is simply stopped during the operation of the wiper as described above, road surface condition recognition, obstacle detection, forward vehicle tracking, and the like are not performed during that time. System security is reduced. Further, among the images captured by the on-board camera during the operation of the wiper, there are also images and image parts that do not affect image processing (the wiper is not shown). There has been a problem that it cannot be used efficiently for image processing.

Further, when the position of the wiper is mechanically detected, the wiper movement resistance exists in principle, and the wiper has bending or play to some extent, so that the accurate position of the wiper is detected. And it is difficult to reflect the position of the wiper in the image processing.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems, and prevents erroneous recognition due to the influence of a wiper, improves the safety and reliability of the system, and ensures accurate operation of the wiper. An object of the present invention is to obtain an image processing method capable of efficiently performing image processing by obtaining a position.

[0008]

According to the first aspect of the present invention, there is provided an image processing method comprising the steps of:
The position is detected, and the wiper is excluded from the image.

According to a second aspect of the present invention, there is provided an image processing method for capturing an image in front of a vehicle captured by an image capturing means, and a current image and a previous image captured by the image capturing process. Difference processing for differentiating pixel densities, wiper frame determination processing for determining whether or not the current image is a wiper frame based on the average density and variance of the difference image obtained as a result of the difference processing, and wiper frame determination As a result of the processing, when the current image is a wiper frame, the wiper portion of the current image is removed, and when the current image is a non-wiper frame, image processing such as automatic driving / driving assistance of a vehicle is performed based on the current image. Processing and an image storage process for storing a previous image one frame before the current image in order to perform a difference process.

According to a third aspect of the present invention, in the image storing process, the current image is stored only when the current image is a non-wiper frame.

In the image processing method according to the present invention, when the current image is determined to be a wiper frame in the wiper frame determination processing, several subsequent frames are unconditionally processed as wiper frames. Things.

An image processing method h according to the present invention.
In the wiper frame determination process, the determination criterion of the wiper frame is changed in accordance with the change in the external illuminance.

According to a sixth aspect of the present invention, there is provided an image processing method in which a shutter speed of an image pickup means is controlled in accordance with external illuminance in order to keep a density level between a current image and a previous image constant. It is.

According to a seventh aspect of the present invention, there is provided an image processing method wherein the shutter speed of the image pickup means is increased, and at the same time, the level of the reference value of the average density and the variance of the difference image in the wiper frame determination process is reduced. is there.

According to an eighth aspect of the present invention, in the wiper frame determination processing, it is determined whether or not the current image is a wiper frame based on an average density and a variance of a rectangular range set at a predetermined position in the difference image. It is something to do.

According to a ninth aspect of the present invention, in the image processing method, the difference processing is performed for each rectangular range of the difference image.

According to the image processing method of the present invention, a rectangular area having the largest number of wipers is determined, and only this rectangular area is excluded, and image processing is performed on other areas.

In the image processing method according to the present invention, in the difference processing, a density level equal to or higher than a saturation threshold of the difference image is cut, and noise is removed by filter processing before the wiper frame determination processing. It was made.

[0019]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS One embodiment of the present invention will be described below. Embodiment 1 FIG. FIG. 1 is a diagram showing an example of an image for explaining an image processing method according to the first embodiment of the present invention.
In FIG. 1, an image 101 is a vehicle-mounted camera (not shown).
Is the current image taken. In this image 101, a wiper portion 101A of the automobile is shown. The average value (average density) of the densities (luminances) of all the pixels of the image 101 is AN, and the variance value (variance) of the densities of all the pixels is VN.

An image 102 is an image one frame before the image 101 captured by the vehicle-mounted camera. Incidentally, the frame refers to an image at a certain moment captured by the vehicle-mounted camera. The image 102 does not show the wiper portion 101A of the automobile like the image 101. The average value (average density) of the densities of all the pixels of the image 102 is AB, and the variance value (variance) of the densities of all the pixels is VB.

The image 103 is a difference image between the image 101 and the image 102, that is, the density of each pixel of the image 101 and the image 102
5 is an image of the difference value of the density of each pixel. The density of each pixel of the difference image 103 may be either an absolute value of the difference between the densities of the images 101 and 102 or a case where a negative value is generated. As a result of the difference between the image 101 and the image 102, a wiper portion 103A remains in the difference image 103. The average value (average density) of the densities of all the pixels of the difference image 103 is a, and the variance value (variance) of the densities of all the pixels is v.

Next, the image processing operation will be described. FIG. 2 is a flowchart for explaining the image processing method according to the first embodiment of the present invention. As shown in FIG. 2, first, an image processing apparatus (not shown) fetches an image 101 to be subjected to image processing (step ST1), and
01 and the image 102 one frame before the current image 101 are subjected to a difference process (step ST2). As described above, the difference processing is for obtaining a difference value between the density of each pixel of the images 101 and 102, and as a result of the difference processing, a difference image 103 is obtained.

Next, the image processing apparatus determines whether or not the current image 101 is a wiper frame based on the difference image 103 (step ST3). Here, the wiper frame refers to an image in which the wiper is captured among images captured at a certain moment by the vehicle-mounted camera.

To determine whether or not the frame is a wiper frame, an average density a and a variance v are obtained from the difference image 103, and the average density a and the variance v are set to predetermined thresholds TA and TB of the average density a and a predetermined threshold TV of the variance v. On the other hand, when the following conditional expression is satisfied, the current image 101 is determined to be a wiper frame.

(1) a> TA or (2) a> TB and v <TV where TA> TB.

That is, the average density a of the entire difference image 103
Is higher (greater than the predetermined threshold TA), or the level of the average density a of the difference image 103 is somewhat higher (greater than the predetermined threshold TB), and the variance v is (greater than the predetermined threshold TV).
If smaller, it is determined as a wiper frame. Because, when the wiper portion 101A is shown in the image 101, the wiper portion 103A of the difference image 103
This is because the average density of the entire difference image 103 is increased, and the variance of pixels having a high density is also reduced.

Wiper frame determination processing (step ST)
As a result of 3), when it is determined that the current image 101 is a wiper frame, actual wiper frame processing is performed on the current image 101 (step ST4). This actual processing for the wiper frame is performed on the image 10 which is the wiper frame.
1 has an adverse effect on the image processing, so that the image 101 is excluded.

On the other hand, as a result of the wiper frame determination processing (step ST3), if it is determined that the current image 101 is not a wiper frame, actual processing for a non-wiper frame is performed on the current image 101 (step ST5). The actual process for a non-wiper frame performs image processing based on an image 101 that is not a wiper frame, and performs road surface state recognition, obstacle detection, and the like.

Actual processing for wiper frame (step ST)
4) or actual processing for a non-wiper frame (step ST
After 5), regardless of whether the image 101 is a wiper frame or a non-wiper frame, the current image 101 is stored for differential processing with the image (101) of the next frame (step ST6). That is, the image saved by the current frame saving process is the image (102) one frame before the next frame.

After the current frame storing process (step ST6), the process returns to the image capturing process (step ST1), and the process of the next frame is executed.

As described above, according to the first embodiment, the operation / position of the wiper is not obtained from the mechanical / electrical information interlocked with the wiper as in the related art.
Difference image 103 obtained from captured images 101 and 102
Therefore, the operation and position of the wiper can be accurately detected, and as a result, the safety and reliability of the system can be improved.

Further, since it is determined whether or not each image is a wiper frame, it is possible to exclude only a wiper frame which adversely affects image processing, and it is possible to perform image processing efficiently.

Embodiment 2 FIG. In the first embodiment, in the current frame saving process (step ST6), the image is saved for the next frame regardless of whether or not the image is a wiper frame. In the second embodiment, the image is saved. Only when the frame is a non-wiper frame, the image is stored for the next frame. That is, in the second embodiment, the difference processing is performed between the current image and the closest past non-wiper frame image.

FIG. 3 is a diagram showing an example of an image for explaining an image processing method according to the second embodiment of the present invention. The images 111 to 116 are arranged in chronological order from left to right. Images 111, 112, and 116 are non-wiper frames, and images 113 to 115 are wiper frames.

Next, the image processing operation will be described. FIG. 4 is a flowchart for explaining an image processing method according to Embodiment 2 of the present invention. (1) Processing from the image 112 in FIG. 3 will be described. The image processing apparatus captures the image 112 (step ST1), as in the first embodiment, and generates the current image 112 and the current image 11 (step ST1).
Image 111 of the closest past non-wiper frame 2
Difference processing is performed with respect to (an image one frame before the current image 112) (step ST2). Then, the image processing apparatus determines whether or not the current image 112 is a wiper frame based on the difference image (not shown) as in the first embodiment (step ST3).

Since the current image 112 is a non-wiper frame, the image processing apparatus stores the current image 112 as the closest past non-wiper frame used in image processing of the next frame and thereafter (update from the image 111). (Step ST11). Then, as in the first embodiment, the image processing apparatus executes actual processing for a non-wiper frame on the image 112 (step ST5), and shifts to processing of the next image 113 (step ST1).

(2) The processing of the image 113 in FIG. 3 will be described. As described above, the image processing apparatus fetches the image 113 (step ST1), and stores the current image 113 and the closest past non-wiper frame image 1 of the current image 113.
12 (step ST2), and it is determined whether or not the current image 113 is a wiper frame based on the difference image (step ST3).

Since the current image 113 is a wiper frame, the image processing apparatus executes actual wiper frame processing on the image 113 without storing the current image 112 in the same manner as in the first embodiment. In step ST4), the process proceeds to the next image 113 (step ST1). That is, the image 112 that is a non-wiper frame is used for image processing of the next frame and subsequent frames without being updated while being stored.

(3) In the processing of the images 114 and 115 in FIG. 3, since the images 114 and 115 are wiper frames, similar to the processing of the image 113 described above, the image capturing processing (step ST1) Difference processing with the image 112 that is a non-wiper frame (step ST2),
Wiper frame determination processing (step ST3) and actual processing for a wiper frame (step ST4) are performed.

(4) The processing of the image 116 in FIG.
Since 16 is a non-wiper frame, the above image 1
12, the image capture process (step ST1),
Difference processing with the image 112 that is the closest past non-wiper frame (step ST2), wiper frame determination processing (step ST3), and saving the image 116 as the closest past non-wiper frame (updated from the image 112)
The current frame saving process (step ST11) and the non-wiper frame actual process (step ST5) are performed. Therefore, the image 116 is used for the difference processing in the image processing of the next frame and thereafter.

The current frame preservation process (step ST11) when the current image is a non-wiper frame is performed before the non-wiper frame actual process (step ST5). Step ST
It may be performed after 5).

As described above, according to the second embodiment, the image of the closest past non-wiper frame is stored for the difference processing from the current image, so that the wiper frame determination processing (step ST3) ) Can be reduced.

That is, in the first embodiment, the current image is stored for the next frame regardless of whether the current image is a wiper frame. Therefore, when the image stored for the difference is a wiper frame, Since the average density of the wiper frame is high, the average density a of the difference image between the wiper frame and the wiper frame decreases, and the current image may be erroneously determined to be a non-wiper frame even though the current image is the wiper frame. However, in the second embodiment, since the difference processing is always performed with the non-wiper frame, the erroneous determination of the current image as the non-wiper frame is reduced.

Embodiment 3 FIG. In the first embodiment, it is determined whether or not each image is a wiper frame. However, in the third embodiment, if the current image is a wiper frame, the subsequent several frames are processed as wiper frames. Is what you do.

The image example according to the third embodiment can be applied to the same image examples as those shown in FIGS. 1 and 3, and therefore the description is omitted.

Next, the image processing operation will be described. FIG. 5 is a flowchart for explaining an image processing method according to Embodiment 3 of the present invention. The image processing device
As in the second embodiment, when an image to be processed is captured (step ST1), it is next determined whether or not the current image can be unconditionally determined as a wiper frame (step ST21).

Whether or not the determination as an unconditional wiper frame is made is determined by whether or not the value of “wip_cnt” is 0, that is, whether or not the value of “wip_cnt” is 1 or more. Here, “wip_cnt” is a value indicating how many subsequent frames after the wiper frame is detected by the image processing apparatus are unconditionally determined to be wiper frames. For example, if the value of “wip_cnt” of the current image captured by the image processing apparatus is 3, the image processing apparatus unconditionally determines up to three subsequent frames as wiper frames.

If the value of “wip_cnt” is 1 or more as a result of the unconditional wiper frame determination process (step ST 21), the image processing apparatus is in a period in which the current image can be unconditionally determined to be a wiper frame. The decrement process, that is, the value of “wip_cnt” is decremented by 1 (step ST22), and then the actual process for the wiper frame is executed (step ST4), and the process proceeds to the next image (step ST1).

On the other hand, if the value of "wip_cnt" is 0 as a result of the unconditional wiper frame determination processing (step ST21), it cannot be unconditionally determined that the current image is a wiper frame. There is a need to. Therefore, similarly to the second embodiment, a difference image is obtained by performing a difference process (step ST2) between the current image and an image of the closest past non-wiper frame, and the current image is wiped based on the difference image. It is determined whether the frame is a frame (step ST3).

Wiper frame determination processing (step ST)
As a result of 3), when it is determined that the current image is a wiper frame, the image processing apparatus determines the number of subsequent frames to be unconditionally determined to be a wiper frame.
The value of “p_cnt” is set to an arbitrary value n. That is, thereafter, the number of frames that use the current image as the wiper frame is unconditionally set to n (step ST23). Then, as in the second embodiment, the image processing apparatus executes the actual process for the wiper frame (step ST4), and shifts to the process of the next image (step ST1).

If the current image is determined to be a non-wiper frame as a result of the wiper frame determination process (step ST3), the image processing apparatus determines the current image as the nearest past image, as in the second embodiment. Is stored as the non-wiper frame (step ST11), and thereafter, the actual processing for the non-wiper frame is executed (step ST5), and the process proceeds to the next image (step ST1).

As described above, according to the third embodiment, when it is detected that the current image is a wiper frame, the subsequent several frames are unconditionally processed as wiper frames. (Or an image having a high possibility of being a wiper frame), it is possible to immediately execute the actual processing for the wiper frame (step ST4), and to improve the efficiency of the image processing and to perform the wiper frame determination processing. The possibility of erroneous determination in (Step 3) can be reduced.

Embodiment 4 In the first to third embodiments, the density of the current image and the density of the previous image for difference processing (the image one frame before the current image or the image of the past non-wiper frame closest to the current image) Is obtained, and it is determined whether or not the current image is a wiper frame from the average density a and the variance v of the differential image, so that the density levels of the current image and the previous image are kept constant. There is a need.

However, as the vehicle moves, the illuminance of the outside world fluctuates (the outside of the vehicle becomes brighter or darker), so that the density levels of the current image and the previous image are not constant. For example, if the car is approaching the shadow of a building, the external illuminance will be dark and the image will be dark. Thus, in the fourth embodiment, the present embodiment is adapted to cope with a change in the external illuminance.

FIG. 6 is a diagram showing an example of an image for explaining an image processing method according to the fourth embodiment of the present invention. In FIG. 6, an image 121 is a current image captured by a vehicle-mounted camera when approaching the shadow of a building. In this image 121, it is assumed that a wiper portion 121A of an automobile is shown and is a wiper frame. The average density AN ′ and the variance VN ′ of the image 121 are normally (image 1 in FIG. 1).
01 (average concentration AN and variance VN).

The image 122 is a previous image for difference processing taken by the vehicle-mounted camera when the vehicle is not approaching the shadow of the building. It is assumed that the average density AB and the variance VB of the image 122 are normal (similar to the average density AB and the variance VB of the image 102 in FIG. 1).

The image 123 is a difference image between the image 121 and the image 122. The difference image 123 includes an image 121
And the image 122, the wiper portion 123A remains. The average density a ′ and the variance v ′ of the difference image 123 are lower than normal (average density a and variance v of the difference image 103 in FIG. 1) because the image 121 is entirely dark behind the building. ing.

The image 124 is a difference image in which the fluctuation of the average density a 'and the variance v' due to the influence of the external illuminance of the difference image 123 (decrease in the density of the entire image due to shadow) is corrected in accordance with the external illuminance. The average density a and the variance v of the difference image 124 are the same as normal (the average density a and the variance v of the difference image 103 in FIG. 1).

The operation of correcting the difference image 123 by the image processing apparatus will be described. The image processing apparatus calculates the average density a ′ and the variance v ′ of the difference image 123, and then calculates the average density a ′ and the variance v ′. The variation of the average density a ′ and the variance v ′ due to the influence of the external illuminance is corrected. That is, the average density a ′ and the variance v ′ of the difference image 123 affected by the external illuminance are corrected to the average density a and the variance v of the difference image 124 not affected by the external illuminance according to the external illuminance.

As described above, by calculating the average density a and the variance v, as described in the first embodiment,
The average density a and the variance v of the difference image 124 are equal to a predetermined threshold T
Whether or not the current image 121 is a wiper frame can be determined based on whether or not a conditional expression based on A, TB, and the predetermined threshold TV is satisfied.

The average density a ′ and the variance v ′ of the difference image 123 are not corrected to the normal average density a and the variance v according to the external illuminance, but the wiper frame determination is performed according to the external illuminance. The predetermined threshold values TA and TB and the predetermined threshold value TV of the conditional expression may be changed. That is, the current image 121 is a wiper frame depending on whether the average density a ′ and the variance v ′ of the difference image 123 satisfy conditional expressions based on the predetermined threshold values TA ′ and TB ′ and the predetermined threshold value TV ′ according to the external illuminance. It can also be determined whether there is. At this time, the predetermined thresholds TA ′, TB ′ and the predetermined threshold TV ′ are values proportional to the external illuminance.

As described above, according to the fourth embodiment, the average density a ′ and the variance v ′ are changed according to the external illuminance so as to correspond to the change in the external illuminance, or
Since the predetermined thresholds TA and TB and the predetermined threshold TV are varied, the influence of the external illuminance can be eliminated, and the reliability of the wiper detection can be improved.

Embodiment 5 FIG. Generally, when the shutter speed (second / frame) of the camera is increased, the level of the density (luminance) of the entire captured image is reduced, and the entire image is darkened. Conversely, when the shutter speed of the camera is reduced, the density level of the entire captured image increases, and the entire image becomes bright. That is, the shutter speed of the camera and the image density are in inverse proportion.

In the fourth embodiment, in order to eliminate the influence of the external illuminance, the average density a ′ and the variance v ′ are changed according to the external illuminance, or the predetermined thresholds TA, TB and the predetermined threshold TV are changed. In the fifth embodiment, the relationship between the shutter speed of the camera and the density of the image is used to make the level of the density of the current image and the level of the density of the previous image constant. The effect of the external illuminance is eliminated.

When the external illuminance is high (when the outside is bright), the shutter speed is increased to lower the density level of the entire image. Conversely, when the external illuminance is low (when the outside is dark), the shutter speed is increased. To increase the density level of the entire image. For example, in FIG. 6, the shutter speed when capturing the image 121 with low external illuminance is lower than the shutter speed when capturing the normal image 122 with external illuminance.
Concentration level can be kept constant.

As described above, according to the fifth embodiment, by controlling the shutter speed of the vehicle-mounted camera in accordance with the illuminance of the external environment, the density level of the current image and the density level of the previous image become constant. The influence of the external illuminance can be eliminated, and as a result, the reliability of the wiper detection can be improved.

Embodiment 6 FIG. In the fifth embodiment, the shutter speed of the on-vehicle camera is controlled in accordance with the external illuminance in order to keep the density level of the current image and the density level of the previous image constant. In such a case, in case of rain,
The shutter speed is reduced due to low external illuminance (because the outside is dark). However, when the shutter speed is reduced, the time taken for the wiper becomes longer, and the number of images (non-wiper frames) available for image processing decreases. As a result, image processing cannot be performed efficiently.

Therefore, in the sixth embodiment, in the case of rain, the shutter speed is increased, and at the same time, the predetermined threshold values TA, TB and the predetermined threshold value TV used in the conditional expression for determining the wiper frame are reduced.

As described above, by simultaneously increasing the shutter speed and decreasing the predetermined threshold values TA and TB and the predetermined threshold value TV, the density level of the current image and the density level of the previous image become constant, and the wiper detection is performed. Can be improved, the time required for the wiper to be captured is shortened, and the number of images (non-wiper frames) that can be used for image processing increases, enabling efficient image processing.

Embodiment 7 FIG. In the first to sixth embodiments, whether or not the current image is a wiper frame is determined based on the average density a and the variance v of the entire difference image. In the seventh embodiment, however, the difference image The determination is made based on the average density and the variance of the rectangular range set at a predetermined position in the box.

FIG. 7 is a diagram showing an example of an image for explaining an image processing method according to the seventh embodiment of the present invention. In FIG. 7, an image 131 is a current image captured by the on-vehicle camera.
31A is shown, which is a wiper frame. Image 1
32 is a previous image of the image 131 captured by the onboard camera, and this image 132 is a non-wiper frame.

The image 133 is a difference image between the image 131 and the image 132. In the difference image 133, rectangular ranges 133K to 133P are set at predetermined positions. The setting positions of the rectangular ranges 133K to 133P are set to locations where the wipers can pass through in the difference image 133 (varies depending on the type of automobile). Also, the rectangular range 133 of the difference image 133
The average concentrations in K to 133P are respectively a1 to a6,
Also, the variances within the rectangular ranges 133K to 133P are denoted by v1 to v6, respectively.

Next, the image processing operation will be described. FIG. 8 is a flowchart for explaining an image processing method according to Embodiment 7 of the present invention. The image processing device
As in the first to sixth embodiments, the current image 13
1 (step ST1), and a difference process is performed on the current image 131 and the previous image 132 (step ST2) to obtain a difference image 133. At this time, the difference image 13
3. Average density a1 to a of a rectangular range 133K to 133P in 3
6 and the variances v1 to v6 are also determined.

Next, the image processing apparatus initializes the numerical value of the difference image 133 (step ST31). That is, the flag “wip_frm” for the wiper frame is set to 0, and the rectangle count number “i” is set to 1.

Next, the image processing apparatus determines whether or not the rectangular count number “i” of the difference image 133 is a value larger than “n” (step ST32), and thereby the rectangular range 133K to be processed is determined. The number of 133P is managed. Here, the value of “n” is the number of rectangular ranges 133K to 133P set in the difference image 133. Therefore, when “i” is smaller than “n”, there is a rectangular range 133K to 133P where the difference image 133 has not been processed, and when “i” is larger than “n”, the difference image 133 This indicates that all the rectangular ranges 133K to 133P have been processed.
In the case of the difference image 133, the rectangular range 133K to 13K
Since there are six 3Ps, “n” is five.

Processing rectangle number management processing (step ST32)
As a result, if “i” is smaller than “n”, there are rectangular ranges 133K to 133P that have not been processed yet, and the image processing apparatus next proceeds to “i” th rectangular range 133K to 133P.
It is determined whether 133P is a wiper rectangle (rectangular range where the wiper is imaged) (step ST33). Here, it is determined whether or not the wiper rectangle is, for example, a rectangular range 1
A case where the average densities a1 to a6 within 33K to 133P become maximum is determined as a wiper rectangle.

If the result of the wiper rectangle determination process is not a wiper rectangle but a non-wiper rectangle (a rectangular range in which the wiper is not imaged), the processing rectangle number increment process is performed.
That is, one is added to the value of “i” (step ST34), and thereafter, the process returns to the processing rectangle number management processing (step ST32), and it is determined whether or not there is any rectangle range 133K to 133P that has not been processed yet.

As a result of the wiper rectangle determination processing, if the wiper rectangle is determined, the image processing apparatus determines that the current image 131 is a wiper frame, and sets the wiper frame flag “wip_frm” to 1 (step ST3).
5). Then, in the wiper frame determination process (step ST36), the image processing apparatus determines whether the wiper frame flag “wip_frm” is 1 or not.
It is determined whether or not the current image 131 is a wiper frame (step ST36). If “wip_frm” is 1, the actual process for the wiper frame is executed (step S36).
T4), the process shifts to processing of the next image (step ST)
1).

On the other hand, the processing rectangle number management processing (step ST
32) As a result, when “i” is larger than “n”, all the rectangular ranges 133K to 133P have been processed, and the process shifts to wiper frame determination processing (step ST36).
At this time, since “wip_frm” is 0, it is determined that the current image 131 is a non-wiper frame, and
31 is stored for difference processing of the next image (step ST1).
1) The actual processing for the non-wiper frame is executed (step ST5), and the processing shifts to the processing of the next image (step ST5).
1).

As described above, according to the seventh embodiment, whether or not the current image 131 is a wiper frame is determined by the average densities a1 to a6 and the variances v1 to v6 of the rectangular ranges 133K to 133P in the difference image 133. Since it is determined whether or not the wiper frame is determined in a range narrower than the entire image, the reliability of the wiper frame determination can be further improved.

Embodiment 8 FIG. In the seventh embodiment, the difference processing is performed for each image. In the eighth embodiment, the difference processing is performed for each rectangular range.

FIG. 9 is a flowchart for explaining an image processing method according to the eighth embodiment of the present invention.
The example of the image is the same as that shown in FIG. The image processing device determines the current image 1
When 312 is loaded (step ST1), next, a numerical value initialization process (step ST31) is performed without performing the difference process (step ST2).

After the numerical value initialization processing, the image processing apparatus performs processing rectangle number management processing (step ST32). If “i” is smaller than “n” as a result of the processing rectangle number management processing (step ST32), there are rectangular areas 133K to 133P that have not yet been processed. Difference processing is performed in units of 133P (step ST41). Then, the image processing device determines whether or not the wiper rectangle is set in units of the rectangular ranges 133K to 133P (step ST42).

As a result of the wiper rectangle determination processing, if the current rectangular range is a wiper rectangle, the wiper frame flag "wip_frm" is set to 1, and one is added to the value of "i" in the processing rectangle number increment processing. (Step ST4
5) Then, the processing rectangle number management processing (step ST3)
Returning to 2), the rectangular range 133K-
It is determined whether there is 133P.

If the current rectangle range is a non-wiper rectangle as a result of the wiper rectangle determination process, the current rectangle range is stored for difference processing with the next rectangle range (step ST44). Returning to (Step ST32), it is determined whether or not there are any rectangular ranges 133K to 133P that have not been processed yet.

On the other hand, processing rectangle number management processing (step ST
32) As a result, when “i” is larger than “n”, all the rectangular ranges 133K to 133P have been processed, and the process shifts to wiper frame determination processing (step ST36).

As a result of the wiper frame determination processing, the image processing apparatus sets the wiper frame flag “wip_fr”.
If “m” is 1, it is determined that the current image 131 is a wiper frame, and actual processing for the wiper frame (step S
T4), and if the wiper frame flag “wip_frm” is 0, it is determined that the current image 131 is a non-wiper frame, and the non-wiper frame actual processing (step ST5) is performed. The process proceeds to processing of the next image (step ST1).

As described above, according to the eighth embodiment, since the difference processing is performed in units of the rectangular ranges 133K to 133P, it is determined whether or not all the rectangular ranges 133K to 133P are wiper rectangles. It is possible,
As a result, the reliability of the wiper frame determination can be further improved.

Embodiment 9 In Embodiments 7 and 8 described above, if it is determined that the rectangular range 133K to 133P is a wiper rectangle, the entire image 131 is excluded by the actual processing for the wiper frame. , Only the rectangular range of the wiper rectangle having the largest number of wiper portions is excluded.

FIG. 10 is a diagram showing an example of an image for explaining an image processing method according to the ninth embodiment of the present invention.
As shown in FIG. 10, a rectangular area 14
0K to 140N are set so as to overlap. The size (length of side) of the rectangular ranges 140K to 140N is the maximum movement amount of the wiper (= maximum wiper speed × shutter time) + α, and the interval between the rectangular ranges 140K to 140N (the rectangular range 140K to 140N). (Interval between sides) is (maximum speed of the wiper × imaging interval (撮 像 α)). That is, the sizes and intervals of the rectangular ranges 140K to 140N are
The wiper portion is set such that the wiper portion always fits (is imaged) without protruding into one of the rectangular ranges 140K to 140N.

The images 141 to 145 are arranged in chronological order from top to bottom. The image 141 is a non-wiper frame, and the images 142 to 145 are wiper frames. Wiper portions 142A-1 of images 142-145
45A is a rectangular area 14 of each of the images 142 to 145.
2K, 143L, 144M, 145N.

FIG. 11 shows one image 143 of FIG.
It is a figure which shows the position of the rectangular range 143K-143N.
As shown in FIG. 11, the rectangular area 143K in the image 143
143N is the rectangular area 14 of the image 140 in FIG.
It is set at the same position as 0K to 140N. That is, the subscript alphabet of the rectangular ranges 142K, 143L, 144M, and 145N is the rectangular range 140K of the image 140.
１４０140N. The wiper portion 143A of the image 143 fits without protruding within the rectangular range 143L.

Next, the image processing operation will be described. The image processing operation of the ninth embodiment differs from the image processing operation of the seventh or eighth embodiment in the following points. That is, in the image 143 of FIG.
If it is determined that the rectangular range 143K to 143M is a wiper rectangle among the 143N (the rectangular range 143N is determined to be a non-wiper rectangle), the wiper portion 143A is the largest among the rectangular ranges 143K to 143M of the wiper rectangle.
Is determined.

Here, a rectangular range 143K of the wiper rectangle is set.
The standard for determining the rectangular range 143L that includes the wiper portion 143A most out of 143M is the rectangular range 143L.
The one with the largest total value Ai of the absolute values of the differences in the K to 143M or the one with the smallest variance Vi in the rectangular ranges 143K to 143M is selected. Also, in some cases,
A conditional expression such as Ai-α · Vi or Ai-α√ (Vi) is used.

Then, the image processing apparatus determines that the wiper portion 1 in the rectangular range 143K to 143M of the wiper rectangle is the largest.
A process for excluding the rectangular range 143L including 43A is executed.

For the area other than the rectangular area 143L,
Since the image processing apparatus does not include the wiper portion 143A, the image processing apparatus executes actual processing for a non-wiper frame.

As described above, according to the ninth embodiment, since the rectangular range of the wiper rectangle having the largest number of wiper portions is determined and only this rectangular range is excluded, the image of the wiper frame including the wiper portion 143A is obtained. 143 can be used effectively, and as a result, the safety and reliability of the system can be further improved.

Embodiment 10 FIG. In the first to ninth embodiments, the noise of the difference image is not considered, but in the tenth embodiment, the image processing is performed in consideration of the noise of the difference image. .

FIG. 12 is a diagram showing an example of an image for explaining noise removal processing of a difference image according to the tenth embodiment of the present invention. In the figure, an image 150 is a difference image showing the wiper portion 150A.

FIG. 13 is a diagram showing a-
It is a figure which shows the density level in a 'line pixel. In FIG. 13, the line images (A), (B), and (C)
It shows the density level (vertical axis) at the aa ′ line pixel (horizontal axis) of the difference image 150. As shown in the line images (A) and (B) of FIG.
Because the car is moving while the on-board camera is taking an image, the current image and the previous image
Since a position where the density level is high or a position where the density level is low is shifted, a pixel having an extremely high density level is generated in the difference image 150 between the current image and the previous image, and this becomes noise. is there.

Next, the image processing operation will be described. FIG. 14 is a flowchart illustrating an image processing method according to the tenth embodiment of the present invention. The image processing apparatus captures the current image as in the first embodiment (step ST1).

Next, the image processing apparatus performs difference processing with saturation (step ST51). In the difference processing with saturation, in addition to the above-described difference processing, an upper limit (saturation threshold) of the difference between the density level of the current image and the previous image is set, and the level of the density equal to or higher than the saturation threshold is cut. Is included. As described above, performing the difference processing with saturation can be ignored in the wiper frame determination processing when the level of the noise density is low, but when the level of the noise density increases,
The average density a increases depending on the noise density level,
This is because, in the wiper frame determination processing, there is a possibility that an incorrect determination is made that the frame is a wiper frame even though the frame is not a wiper frame.

As a result of such differential processing with saturation, the waveform of the line image (A) is cut at a level higher than the saturation threshold, and becomes a waveform as shown in the line image (B).

Next, the image processing device performs a minimum filter process (step ST52). This minimum filter processing is processing for setting the minimum value of the density level in the number of pixels around the pixel of the processing target line to the density level of the pixel, thereby removing noise. As a result of such minimum filter processing, the noise of the line image (B) is removed and the waveform of the line image (B) becomes a waveform like the line image (C).

The subsequent processing is the same as that described with reference to FIG. 2 of the first embodiment, and a duplicate description will be omitted.

As described above, according to the tenth embodiment, while the density level equal to or higher than the saturation threshold is cut,
Since the wiper frame judgment is performed after removing the noise, erroneous judgments in the wiper frame judgment processing can be reduced, and as a result, the efficiency of image processing can be improved and the safety and reliability of the system can be improved. Can be improved.

[0107]

As described above, according to the first aspect of the present invention, the operation and position of the automobile wiper are detected from the image, and the wiper is excluded from the image.
This has the effect of preventing erroneous recognition of the system due to the influence of the wiper and improving the safety and reliability of the system.

According to the second aspect of the present invention, the image capturing process for capturing the image in front of the vehicle captured by the image capturing means, and the current image and the previous image captured in the image capturing process are used as the density of each pixel. , A wiper frame determination process for determining whether the current image is a wiper frame based on the average density and variance of the difference image obtained as a result of the difference process, and a result of the wiper frame determination process If the current image is a wiper frame, remove the wiper portion of the current image, and if the current image is a non-wiper frame, perform image processing such as automatic driving / running assistance of the vehicle based on the current image; In order to perform the difference processing, the image storage processing for storing the previous image one frame before the current image is provided, so that the operation / position of the wiper can be accurately detected, As a result, it is possible to improve the safety and reliability of the system, and to determine whether or not each image is a wiper frame. Therefore, it is possible to exclude only a wiper frame that adversely affects image processing, and efficiently perform image processing. There is an effect that can be performed.

According to the third aspect of the invention, in the image storing process, the current image is stored only when the current image is a non-wiper frame, so that erroneous determination in the wiper frame determining process is reduced. There is an effect that can be.

According to the fourth aspect of the present invention, when the current image is determined to be a wiper frame in the wiper frame determination processing, the subsequent several frames are unconditionally processed as wiper frames. In addition, the process for the wiper frame can be immediately executed for the image that is the wiper frame, and the effect of increasing the efficiency of the image processing can be achieved.

According to the fifth aspect of the present invention, in the wiper frame determination processing, the determination criterion of the wiper frame is changed in accordance with the fluctuation of the external illuminance. The effect can be eliminated, and the reliability of wiper detection can be improved.

According to the sixth aspect of the invention, the shutter speed of the image pickup means is controlled in accordance with the external illuminance in order to keep the density level of the current image and the previous image constant. As in the case of 5, the effect of the external illuminance can be eliminated, and the reliability of the wiper detection can be improved.

According to the seventh aspect of the present invention, the shutter speed of the image pickup means is increased, and at the same time, the level of the reference value of the average density and the variance of the difference image in the wiper frame determination process is reduced. In addition to maintaining the reliability, the time required for the wiper to be photographed is shortened, and the number of images that can be used for image processing increases, so that image processing can be performed efficiently.

According to the eighth aspect of the invention, in the wiper frame determination processing, it is determined whether or not the current image is a wiper frame based on the average density and variance of a rectangular range set at a predetermined position in the difference image. Therefore, the wiper frame determination can be performed in a range narrower than the entire image, and as a result, there is an effect that the reliability of the wiper frame determination can be further improved.

According to the ninth aspect of the present invention, the difference processing is performed for each rectangular range of the differential image, so that it is possible to determine whether or not all the rectangular ranges are wiper rectangles. There is an effect that the reliability of the wiper frame determination can be further improved.

According to the tenth aspect of the present invention, a rectangular range having the largest number of wipers is determined, only this rectangular range is excluded, and image processing is performed on the other regions.
The image of the wiper frame including the wiper portion can be effectively used, and as a result, the image processing efficiency can be improved, and the safety and reliability of the system can be further improved. .

According to the eleventh aspect of the present invention, in the difference processing, the density level equal to or higher than the saturation threshold of the difference image is cut, and noise is removed by filter processing before the wiper frame determination processing. In addition, erroneous determination in the wiper frame determination process can be reduced, and as a result, the efficiency of image processing can be further improved, and the safety and reliability of the system can be improved.

[Brief description of the drawings]

FIG. 1 is a diagram showing an example of an image for explaining an image processing method according to a first embodiment of the present invention;

FIG. 2 is a flowchart illustrating an image processing method according to the first embodiment of the present invention.

FIG. 3 is a diagram showing an example of an image for explaining an image processing method according to a second embodiment of the present invention;

FIG. 4 is a flowchart illustrating an image processing method according to a second embodiment of the present invention;

FIG. 5 is a flowchart illustrating an image processing method according to a third embodiment of the present invention.

FIG. 6 is a diagram showing an example of an image for explaining an image processing method according to a fourth embodiment of the present invention.

FIG. 7 is a diagram showing an example of an image for explaining an image processing method according to a seventh embodiment of the present invention;

FIG. 8 is a flowchart illustrating an image processing method according to a seventh embodiment of the present invention.

FIG. 9 is a flowchart illustrating an image processing method according to an eighth embodiment of the present invention.

FIG. 10 is a diagram showing an example of an image for explaining an image processing method according to a ninth embodiment of the present invention;

FIG. 11 is a diagram showing positions of a rectangular range of one image of FIG. 10;

FIG. 12 is a diagram showing an example of an image for explaining noise removal processing of a difference image according to the tenth embodiment of the present invention.

13 is a diagram illustrating density levels of aa ′ line pixels of the difference image of FIG. 12;

FIG. 14 is a flowchart illustrating an image processing method according to a tenth embodiment of the present invention.

[Explanation of symbols]

101, 102, 111 to 116, 121, 122, 1
31, 132 images 103, 123, 124, 133, 140, 141-1
45, 150 difference images 101A, 103A, 121A, 123A, 124A,
131A, 133A, 141A to 145A, 150A
Wiper part 133K-133P, 140K-140N, 142K,
143K-N, 144M, 145N Rectangular range

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Yuji Katsuno 1-1-1, Shinurashima-cho, Kanagawa-ku, Yokohama-shi, Kanagawa Prefecture F-term in NEC Robot Engineering Co., Ltd. 5B057 CA08 CA12 CA16 CA16 CB08 CB12 CB16 CC03 CE20 CH08 DA02 DA17 DB02 DC22 5H180 AA01 CC04 EE13 LL01 LL04

Claims (11)

[Claims] 1. An image processing method for taking an image of the front of an automobile by an imaging means, and processing the captured image to perform automatic driving / assistance of the automobile, and the like. An image processing method, comprising detecting a position and excluding the wiper from the image. 2. An image capturing process for capturing an image in front of a vehicle captured by an image capturing means, a difference process for differentiating a current image and a previous image captured by the image capturing process with respect to a density of each pixel, A wiper frame determination process for determining whether or not the current image is a wiper frame based on an average density or a variance of the difference image obtained as a result of the difference process; In the case of, the wiper portion of the current image is removed, and if the current image is a non-wiper frame, an image actual process of performing image processing such as automatic driving / running assistance of the vehicle based on the current image; An image storing method for storing a previous image one frame before the current image in order to perform the processing. 3. The image processing method according to claim 2, wherein in the image storing process, the current image is stored only when the current image is a non-wiper frame. 4. The method according to claim 2, wherein when the current image is determined to be a wiper frame in the wiper frame determination processing, several subsequent frames are unconditionally processed as wiper frames. Image processing method. 5. The wiper frame determination process according to claim 2, wherein a determination criterion for the wiper frame is changed in accordance with a change in external illuminance.
The image processing method according to claim 1. 6. A shutter speed of an image pickup means is controlled in accordance with external illuminance in order to keep a density level between a current image and a previous image constant.
The image processing method according to claim 1. 7. The image processing method according to claim 6, wherein the shutter speed of the imaging unit is increased, and at the same time, the level of the reference value of the average density and the variance of the difference image in the wiper frame determination process is reduced. 8. The wiper frame determination processing, wherein it is determined whether or not the current image is a wiper frame based on an average density and a variance of a rectangular range set at a predetermined position in the difference image. Item 8. The image processing method according to any one of Items 7. 9. The image processing method according to claim 8, wherein the difference processing is performed for each rectangular range of the difference image. 10. The image processing method according to claim 8, wherein a rectangular area having the largest number of wipers is determined, and only this rectangular area is excluded, and image processing is performed on other areas. 11. The difference processing according to claim 2, wherein a density level equal to or higher than a saturation threshold of the difference image is cut, and noise is removed by filter processing before the wiper frame determination processing. The image processing method described in the above.