JP2006174377A - Image processing apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To detect a disappearance line of a picked-up image and to correct the image position so that the disappearance line is in the center of a monitor. <P>SOLUTION: A controller 103 calculates the image speed of an image of a scene in front of a vehicle picked up by a camera 101 and calculates a variance speed of a speed in each detection region set on the image. One detection region where the absolute value of the calculated variance value of the speed is below a designated threshold and detects the extracted detection region as a disappearance line. Then the display position is so corrected that the detected disappearance line meets the vertical center of the monitor 104. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus and a method for processing an image captured by a camera mounted on a vehicle.

  The following position detection apparatus is known from Patent Document 1. In this position detection apparatus, the amount of vertical movement between frames of an object detected on the imaged screen is measured, and the amount of change in pitching angle is calculated based on the amount of movement. Then, the pitching correction amount is calculated by adding the calculated pitching change amount of the current frame to the pitching change amount in the past frame, and the image is corrected so as to cancel the influence of the pitching.

JP 2001-84497 A

  However, in the conventional apparatus, since the pitching correction amount is calculated by adding the amount of change in the pitching angle calculated in each frame, the measurement error is accumulated and the pitching correction amount is shifted. There has been a problem that the influence of may not be completely cancelled.

  The present invention performs image processing on an image in front of the host vehicle imaged by the imaging means, calculates speed information of the image of each pixel, and horizontally in the image based on the calculated speed information of the image of each pixel. The velocity dispersion value in the plurality of divided detection areas is calculated, one detection area where the absolute value of the calculated velocity dispersion value is equal to or less than a predetermined threshold is detected as an erasure line, and the position of the detected erasure line Is characterized in that the display position of the image is corrected so as to coincide with the vertical center of the display means.

  According to the present invention, the variance value of the velocity in the detection region of the image is calculated, one detection region in which the absolute value of the calculated velocity variance value is equal to or less than a predetermined threshold is extracted, and the extracted detection region is It was detected as a vanishing line, and the display position of the image was corrected so that the position of the detected vanishing line coincided with the vertical center of the display means. As a result, the object present on the vanishing line can detect the vanishing line with high accuracy in consideration of the fact that the absolute value of the velocity dispersion value on the image becomes relatively small even when pitching occurs. Further, by correcting the display position of the image so that the position of the vanishing line coincides with the center in the vertical direction of the display means, it is possible to accurately correct the vertical position change of the image due to the pitching of the vehicle. .

-First embodiment-
FIG. 1 is a block diagram illustrating a configuration example of an embodiment of an image processing apparatus according to the first embodiment. The image processing apparatus 100 is mounted on a vehicle, and includes a camera 101 that captures the front of the vehicle, an image memory 102 that stores an image captured by the camera 101, a control device 103 that will be described later, and a front of the host vehicle captured by the camera 101. And a monitor 104 for displaying an image.

  The camera 101 is a high-speed camera having an image sensor such as a CCD or a CMOS, for example, and continuously at a very small fixed time Δt interval, for example, 2 ms interval (frame rate = 500 fps (frame per second)) while the host vehicle is traveling. The vehicle front is imaged and output to the image memory 102 for each frame. As shown in FIG. 2, the camera 101 is installed in front of the interior of the vehicle, its visual axis direction Z is directed to the front front direction of the vehicle, and the horizontal axis X of the imaging surface is parallel to the ground surface. The vertical axis Y of the imaging surface is set to be perpendicular to the ground surface.

  As will be described later, the control device 103 calculates speed information (image speed) of the captured image 3a captured by the camera 101 shown in FIG. 3, and extends in the horizontal direction in the captured image based on the calculated image speed. A position corresponding to the installation height of the camera 101, that is, the vanishing line 3b is detected. And the pitching angular velocity of a vehicle is calculated based on the detected image velocity in the vanishing line 3b.

  First, a detection area for detecting the vanishing line 3b is set on the captured image 3a. Specifically, as shown in FIG. 4, the captured image 3a has a height corresponding to a predetermined pixel in the vertical direction (vertical direction), for example, a height corresponding to one pixel, and a plurality of regions 4a extending in the horizontal direction. Is set as the detection region 4a. That is, the captured image 3a is divided in the horizontal direction in the detection region 4a. Then, edge extraction processing is performed on the captured image 3a, an optical flow is calculated using a known gradient method or block matching method, and an image speed for each pixel in the captured image 3a is calculated. Since the edge extraction process and the optical flow calculation process are well-known techniques, the description thereof is omitted.

  For example, as shown in FIG. 5, there is an object 5a that exists above the vanishing line 3b in the real space, an object 5b that exists above the vanishing line 3b, and an object 5c that exists below the vanishing line 3b. The image speed calculated from each object is as shown in FIG. That is, the object 5a existing above the vanishing line 3b approaches the host vehicle as time passes, and the upward image speed 6a increases as the distance from the host vehicle decreases. Further, the object 5c existing below the vanishing line 3b has a downward image speed 6c that increases with time. On the other hand, the image speed 6b calculated from the object 5b existing on the vanishing line 3b does not take into account the image speed detected when the visual axis direction of the camera 101 changes due to the effect of pitching. Regardless of the distance to the vehicle, it is always 0.

  Thereby, in the object located above or below the vanishing line 3b, even if the height from the existing ground is the same in the real space, if the distance from the own vehicle is different, The calculated image speed is also different. For example, as shown in FIG. 7, when the object 7a exists at a position farther from the host vehicle than the above-described object 5c and the height from the ground is the same, the image speed is as shown in FIG. . That is, the image speed 8b of the object 5c present at a position closer to the own vehicle is calculated to be larger than the image speed 8a of the object 7a located far from the own vehicle (in the negative direction in FIG. 8).

  In this way, different image velocities are calculated from objects located above or below the vanishing line 3b depending on the distance from the host vehicle to each object, whereas objects existing on the vanishing line 3b In, the vanishing line 3b in the captured image 3a is detected in consideration of the fact that the image speed is normally 0 regardless of the distance to the host vehicle. That is, for each detection area 4a, the variance value of the image speed calculated in each detection area 4a is calculated. As described above, since the object existing at a position other than the vanishing line 3b has different image speeds depending on the distance from the own vehicle, the absolute value of the variance value becomes large.

  However, since the image speed calculated from the object existing on the vanishing line 3b is normally 0, even if the image speed is detected by the change of the visual axis direction of the camera 101 due to the effect of pitching, the vanishing line is detected. The absolute value of the dispersion value of the image speed is relatively small as compared with positions other than 3b. From this, the detection area 4a in which the absolute value of the calculated variance value of the image speed is equal to or less than a predetermined threshold is extracted as the disappearance line candidate area. If there is one vanishing line candidate region extracted here, it is determined that the vanishing line 3b exists in the extracted vanishing line candidate region, and the vanishing line 3b is detected. That is, the vanishing line candidate region at this time is determined as the vanishing line 3b, and the position where the vanishing line 3b exists in the captured image 3a is detected.

  On the other hand, if there are two or more extracted disappearance line candidate regions, the following processing is performed in order to narrow it down to one. That is, an image is picked up by reducing the frame rate of the camera 101 by a predetermined rate, and a variance value of the image speed is calculated by the above-described processing based on this image. As described above, a plurality of objects existing in the vicinity of the vanishing line 4b position are extracted by reducing the frame rate of the camera 101 because the change in the image speed is small even if the distance to the host vehicle changes. The difference in image speed between vanishing line candidate regions can be clarified.

  For this reason, if the detection area 4a in which the absolute value of the variance value of the image speed calculated again is equal to or less than a predetermined threshold is extracted, the number of erasure line candidate areas to be extracted becomes smaller than before the frame rate is reduced. Therefore, if there is one vanishing line candidate region extracted at this time, it is determined that the vanishing line 3b exists in the extracted vanishing line candidate region, and the position of the vanishing line 3b in the captured image 3a is detected. To do. If there are still two or more vanishing line candidate regions extracted, the frame rate of the camera 101 is gradually decreased at a predetermined rate interval until the number of vanishing line candidate regions is limited to one.

  When only one vanishing line candidate region is extracted by the above processing, it is determined that the vanishing line 3b in the captured image 3a exists in the vanishing line candidate region, and the position of the vanishing line 3b in the captured image 3a is determined. Is detected. Since the detection area 4a is set to have a predetermined pixel width in the vertical direction on the captured image 3a, a width is generated in a range where the vanishing line 3b exists. By setting the region 4a to have a minute height in the vertical direction, for example, one pixel height, the range of the vanishing line 3b position is limited, and the detection region 4a is detected as the vanishing line 3b as it is. In addition, the vanishing line can be detected with high accuracy.

  Then, by correcting the captured image 3a and displaying it on the monitor 104 so that the detected disappearance line 3b is always in the center in the vertical direction of the monitor 104, the visual axis direction of the camera 101 changes due to the influence of pitching. It is possible to correct the captured image 3a captured in a state shifted vertically and present it to the user.

  Next, the angular velocity of pitching occurring in the vehicle is calculated based on the detected image velocity of each pixel present on the vanishing line 3b. First, an average value (average image speed) of image speeds of the respective pixels existing on the vanishing line 3b is calculated. As described above, since the image speed calculated from the object existing on the vanishing line 3b is normally 0, the image speed calculated for each pixel existing on the vanishing line 3b is affected by the pitching. It can be determined that the visual axis direction has been detected. That is, by calculating the above-described average image speed, the image speed due to pitching can be calculated.

  Based on the calculated average image speed, the movement amount of the vanishing line 3b per unit time on the captured image 3a is calculated, and the view of the camera 101 in the real space is calculated using the movement amount and the parameters of the camera 101. By calculating the axial displacement amount, the change amount of the camera visual axis angle per unit time, that is, the pitching angular velocity is calculated. Thereby, the situation of pitching occurring in the vehicle can be detected without using a special measuring device.

  By executing the above-described processing for each frame that is continuously captured by the camera 101, the captured image 3a can be displayed on the monitor 104 with the influence of pitching always removed, and the pitching angular velocity is measured. Thus, it is possible to detect a pitching situation that sequentially changes in accordance with the road surface situation.

  In the first embodiment, the above-described disappearance line 3b is detected in each frame, but the position of the disappearance line 3b does not change greatly between consecutive frames that are imaged at very small time intervals Δt. Therefore, based on the calculated average image speed, the existence range of the vanishing line 3b of the next frame is predicted, and only the detection area 4a existing within the predicted existence range is used as the vanishing line candidate area. Calculate the value. Thereby, the range for detecting the vanishing line 3b can be limited, and the processing speed can be improved.

  FIG. 9 is a flowchart showing the processing of the image processing apparatus 100 according to the first embodiment. The processing shown in FIG. 9 is executed by the control device 103 when the power of the image processing device 100 is turned on by turning on the ignition switch of the vehicle on which the image processing device 100 is mounted. In step S10, the captured image 3a from the image memory 102 is captured, and the process proceeds to step S20. In step S20, the vanishing line detection area 4a is set on the captured image 3a, and the process proceeds to step S30.

In step S30, edge extraction processing is performed on the captured image 3a, an optical flow is calculated using a known gradient method or block matching method, and an image speed for each pixel in the captured image 3a is calculated. Thereafter, the process proceeds to step S40, and it is determined whether or not the existence range of the vanishing line 3b has been predicted in the process for the previous frame. If it is determined that the vanishing line existence range has been predicted, the process proceeds to step S50. In step S50, a variance value of the image speed is calculated in the detection area 4a existing within the predicted range, and the process proceeds to step S70. on the other hand,
If it is determined that the vanishing line existence range has not been predicted, the process proceeds to step S60. In step S60, the variance value of the image speed is calculated in all the set detection areas 4a, and the process proceeds to step S70.

  In step S70, the detection area 4a in which the absolute value of the calculated variance value is less than a predetermined threshold is extracted as a vanishing line candidate area. Then, it progresses to step S80 and it is judged whether the extracted vanishing line candidate area | region is one. If it is determined that the number of extracted disappearance line candidate regions is not one, the process proceeds to step S90. In step S90, the frame rate of the camera 101 is decreased by a predetermined rate, the process returns to step S10, and the process is repeated until the number of vanishing line candidate regions is limited to one. On the other hand, if it is determined that there is one extracted vanishing line candidate region, the process proceeds to step S100.

  In step S100, one extracted vanishing line candidate region is detected as vanishing line 3b, and the process proceeds to step S110. In step S110, the captured image 3a is corrected so that the position of the detected disappearance line 3b is at the center of the monitor 104, and the captured image 3a from which the influence of pitching is eliminated is displayed on the monitor 104. Thereafter, the process proceeds to step S120, where the average image speed on the vanishing line 3b is calculated as described above, and the pitching angular speed is calculated in consideration of the parameters of the camera 101 based on the calculated average image speed. Thereafter, the process proceeds to step S130.

  In step S130, based on the calculated average image speed, the existence range of the disappearance line 3b of the next frame is predicted, and the process proceeds to step S140. In step S140, it is determined whether or not the ignition switch of the vehicle on which the image processing apparatus 100 is mounted is turned off. If it is determined that the ignition switch is not turned off, the process returns to step S10, and the above-described processing is performed on the captured image 3a of the next frame. On the other hand, if it is determined that the ignition switch is turned off, the process is terminated.

According to the first embodiment described above, the following operational effects can be obtained.
(1) The variance value of the image speed is calculated in each detection area 4b set on the captured image 3a, and the detection area 4a in which the absolute value of the variance value of the image speed is equal to or less than a predetermined threshold is extracted as an erasure line candidate area. When there is one vanishing line candidate region, the vanishing line candidate region is detected as the position where the vanishing line 4b exists. As a result, the image speed calculated from the object existing on the vanishing line 3b is normally 0, and the image speed is detected by changing the visual axis direction of the camera 101 due to the effect of pitching. However, the vanishing line 4b can be detected with high accuracy in consideration of the fact that the absolute value of the dispersion value of the image speed is relatively small as compared with a position other than the position on the vanishing line 3b.

(2) When there are two or more extracted vanishing line candidate regions, the frame rate of the camera 101 is gradually decreased at a predetermined rate interval until the number of vanishing line candidate regions is limited to one. As a result, a plurality of objects existing in the vicinity of the vanishing line 4b position are extracted by reducing the frame rate of the camera 101, considering that the change in the image speed is small even if the distance to the host vehicle changes. A difference in image speed between vanishing line candidate regions can be clarified, and vanishing line candidate regions can be narrowed down.

(3) The captured image 3 a is corrected and displayed on the monitor 104 so that the position of the detected disappearance line 3 b is always at the center in the vertical direction of the monitor 104. This makes it possible to correct the captured image captured in a state where the viewing axis direction of the camera 101 changes due to the effect of pitching and shifts up and down, and present it to the user.

(4) Based on the calculated average image speed, the existence range of the disappearance line 3b of the next frame is predicted, and the dispersion value of the image speed is calculated using only the detection area 4a existing within the predicted existence range as the disappearance line candidate area. I tried to do it. As a result, it is possible to limit the range for detecting the vanishing line 3b in consideration of the fact that the position of the vanishing line 3b does not change greatly between consecutive frames imaged at very small time intervals Δt. , Processing speed can be improved.

(5) Based on the calculated average image speed, the movement amount of the vanishing line 3b per unit time on the captured image 3a is calculated, and the movement amount of the camera 101 in the real space is calculated using the movement amount and the parameters of the camera 101. The pitching angular velocity is calculated by calculating the visual axis direction displacement amount. Thereby, the pitching angular velocity can be detected without using a special measuring device.

-Second embodiment-
In the second embodiment, as described above in the first embodiment, the erasure line candidate region is obtained from a captured image captured in a state slower than a captured image captured in a state where the frame rate is fast. The vanishing line is detected in consideration of narrowing down. 1 to 8 are the same as those in the first embodiment, and a description thereof will be omitted.

  After setting the detection region 4b on the captured image 3a, the control device 103 extracts the vanishing line candidate region in a state where the frame rate of the camera 101 is high, for example, in a state where the frame rate = 500 fps, and the position of the vanishing line 4b is determined to some extent. Narrow down to the range. Thereafter, a predetermined ratio, for example, a 10% frame rate is reduced, and the erasure line candidate region is extracted within the previously narrowed range to further narrow the range. This process is repeated, and the vanishing line candidate region extracted when the frame rate drops to a preset threshold value is detected as the vanishing line 4b. Note that a sufficient frame rate for narrowing the vanishing line candidate region to one is set as the threshold here, and when the frame rate is reduced to a preset threshold, the vanishing line candidate region is always It is assumed that it is narrowed down to one.

  In this way, by initially narrowing down and extracting the erasure line candidate area in an approximate range at a fast frame rate, and then gradually narrowing down the erasure line candidate area while slowing down the frame rate, It is possible to detect the vanishing line 4b with high accuracy.

  FIG. 10 is a flowchart illustrating processing of the image processing apparatus 100 according to the second embodiment. The processing shown in FIG. 10 is executed by the control device 103 when the power of the image processing device 100 is turned on by turning on the ignition switch of the vehicle on which the image processing device 100 is mounted. Note that the same step numbers are assigned to the processes common to those in FIG. 9 in the first embodiment, and differences will be mainly described.

  In step S41, it is determined whether or not the vanishing line candidate region has been extracted. If it is determined that the vanishing line candidate region has been extracted, the process proceeds to step S51. In step S51, a variance value of the image speed is calculated from the extracted vanishing line candidate region, and the process proceeds to step S70. On the other hand, when it is determined that the vanishing line candidate region has not been extracted, the process proceeds to step S60, and the variance value of the image speed is calculated in all the detection regions 4a set on the captured image 3a.

  In step S81, it is determined whether the frame rate of the camera 101 has decreased to a predetermined threshold. If it is determined that the frame rate of the camera 101 has not decreased to the predetermined threshold, the process proceeds to step S90, and the frame rate is decreased at a predetermined rate. On the other hand, when it is determined that the frame rate of the camera 101 has decreased to a predetermined threshold value, the process proceeds to step S100, and the extracted vanishing line candidate region is detected as the vanishing line 3b.

According to the second embodiment described above, the following operational effects can be obtained in addition to the effects of the first embodiment.
(1) The frame rate of the camera 101 is decreased at a predetermined rate, and the range of the disappearance line candidate region is narrowed down. Then, the vanishing line candidate region extracted when the frame rate drops to a preset threshold value is detected as the vanishing line 4b. As a result, the vanishing line candidate region can be first narrowed down and extracted in an approximate range, and then the vanishing line candidate region can be narrowed down gradually, so that the vanishing line 4b can be detected with high accuracy. Become.
(2) When detecting the vanishing line candidate region extracted when the frame rate is lowered to a preset threshold value as the vanishing line 4b, a frame rate sufficient to narrow the vanishing line candidate region to one at this threshold value Was set. As a result, when the frame rate is lowered to a preset threshold value, the erasure line candidate region can always be narrowed down to one.

-Modification-
The above-described image processing apparatus 100 can be modified as follows.
(1) In the first and second embodiments described above, the example in which the image processing apparatus 100 according to the present invention is mounted on a vehicle has been described. However, the present invention is not limited to this, and the image processing apparatus 100 may be mounted on other moving bodies. Good.

(2) In the first and second embodiments described above, edge extraction processing is performed on a captured image, and an optical flow is calculated using a known gradient method or block matching method. An example of calculating the image speed for each pixel has been described. However, the present invention is not limited to this, and the optical flow may be calculated by other methods to calculate the image speed for each pixel in the captured image.

(3) In the first and second embodiments described above, the example in which the frame rate of the camera 101 is reduced at a predetermined rate in order to narrow the vanishing line candidate region to one has been described. However, the present invention is not limited to this, and a vehicle speed sensor that detects the vehicle speed of the host vehicle may be mounted, and the frame rate may be changed according to the vehicle speed detected by the vehicle speed sensor. For example, the frame rate is increased as the host vehicle accelerates, and the frame rate is decreased as the host vehicle decelerates. As a result, the vanishing line 4b can be detected with higher accuracy.

  Note that the present invention is not limited to the configurations in the above-described embodiments as long as the characteristic functions of the present invention are not impaired.

  The correspondence between the constituent elements of the claims and the embodiment will be described. The camera 101 corresponds to an imaging unit, and the monitor 104 corresponds to a display unit. The control device 103 corresponds to speed information calculation means, variance value calculation means, vanishing line detection means, display position correction means, pitching angular velocity calculation means, prediction means, and frame rate change means. This correspondence is an example, and the correspondence is different depending on the configuration of the embodiment.

It is a block diagram which shows the structural example of one Embodiment of an image processing apparatus. It is a figure which shows the example of installation to the vehicle of the camera. It is a figure which shows the specific example of the image imaged with the camera. It is a figure which shows the specific example at the time of setting the detection area 4a to the captured image. It is a 1st figure which shows the positional relationship of a vanishing line and an object. It is a figure which shows the specific example of the image speed which changes according to the positional relationship of a vanishing line and an object. It is a 2nd figure which shows the positional relationship of a vanishing line and an object. It is a figure which shows the specific example of the image speed which changes according to the distance with the own vehicle. It is a flowchart figure which shows the process of the image processing apparatus 100 in 1st Embodiment. It is a flowchart figure which shows the process of the image processing apparatus 100 in 2nd Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 101 Camera 102 Image memory 103 Control apparatus 104 Monitor

Claims (8)

An image processing apparatus mounted on a vehicle,
Imaging means for capturing an image in front of the host vehicle;
Speed information calculating means for processing the image captured by the imaging means and calculating speed information of the image of each pixel;
Based on the velocity information of the image of each pixel calculated by the velocity information calculating unit, a variance value calculating unit that calculates a variance value of the velocity in a plurality of detection areas divided in the horizontal direction on the image;
Vanishing line detection means for detecting one detection area where the absolute value of the velocity dispersion value calculated by the dispersion value calculation means is equal to or less than a predetermined threshold as an disappearance line;
An image processing apparatus comprising: a display position correction unit that corrects a display position of an image so that a position of the vanishing line detected by the vanishing line detection unit coincides with a vertical center of the display unit. The image processing apparatus according to claim 1.
An image processing apparatus, further comprising: a pitching angular velocity calculating unit that calculates a pitching angular velocity of the vehicle based on speed information of an image of each pixel existing on the vanishing line on the image detected by the vanishing line detecting unit. The image processing apparatus according to claim 1 or 2,
Based on the velocity information of the image of each pixel present on the vanishing line on the image detected by the vanishing line detecting unit, the prediction unit further predicts a range where the vanishing line exists in the next frame,
For the next frame image,
The variance value calculator calculates the variance value of the velocity in the detection area of the range predicted by the prediction means,
The vanishing line detection unit detects the vanishing line from a detection region within a range predicted by the prediction unit. In the image processing device according to any one of claims 1 to 3,
Frame rate changing means for changing the frame rate of the imaging means,
The vanishing line detection unit controls the frame rate changing unit to detect the vanishing line while changing the frame rate of the imaging unit. The image processing apparatus according to claim 4.
The vanishing line detecting means controls the frame rate changing means when a plurality of detection areas in which the absolute value of the velocity variance value calculated by the variance value calculating means is not more than a predetermined threshold are extracted. The image processing apparatus is characterized in that the frame rate of the imaging means is reduced to limit the detection area where the absolute value of the velocity variance value is equal to or less than a predetermined threshold to one. The image processing apparatus according to claim 4.
The vanishing line detection means controls the frame rate changing means to gradually reduce the frame rate of the imaging means, and the absolute value of the velocity variance value calculated by the variance value calculation means is a predetermined threshold value. An image processing apparatus that narrows down the detection area to be one. In the image processing device according to any one of claims 4 to 6,
The image processing apparatus, wherein the frame rate changing means changes a frame rate of the imaging means in accordance with a vehicle speed of the host vehicle. Image processing is performed on the image ahead of the host vehicle captured by the imaging unit, and the speed information of the image of each pixel is calculated.
Based on the calculated velocity information of the image of each pixel, the variance value of the velocity in a plurality of detection areas divided in the horizontal direction on the image is calculated,
Detecting the one detection area where the absolute value of the calculated variance value of the velocity is equal to or less than a predetermined threshold as an erasure line;
An image processing method comprising correcting an image display position so that a position of a detected disappearance line coincides with a center in a vertical direction of a display unit.

Images