JP2006170961A - Image processor and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To calculate the amount of displacement in a captured image, caused by the vibration of a vehicle. <P>SOLUTION: A control unit 103 specifies traveling components that travel in the vertical directions in an image accompanying vibration of the vehicle in the image captured by a camera 101; calculates the strength of the vibration frequency in the traveling components, based on the travel history of a specified traveling component; extracts the range of the vibration frequency where the strength of the vibration frequency is the largest; extracts a balanced image captured at the balanced point of vibration within a prescribed extracted frequency range; detects the vanishing point at the center position of the displacement in the image displaced by the vibration of the vehicle, based on the extracted balanced image; monitors the detected vanishing point; and calculates the amount of displacement in the image, caused by the vibration of the vehicle in each captured image. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus and method for processing an image captured by a camera mounted on a vehicle.

  The following position detection apparatus is known from Patent Document 1. In this position detection device, an average value of y-coordinate displacement amounts of each extracted object is calculated from a plurality of acquired images, and this is used as the displacement amount of the image due to vehicle vibration.

JP 2001-84497 A

  However, in the conventional apparatus, since the average value of the y-coordinate displacement amount of the object is set as the image displacement amount due to the vibration of the vehicle, even if the position of the y-coordinate changes due to the movement of the object. The displacement amount is averaged as the displacement amount of the image due to the vibration of the vehicle, and there is a problem that an error occurs in the displacement amount of the image due to the vibration of the vehicle.

  The present invention is an image processing method that performs image processing on an image that is mounted on a vehicle and picked up by an image pickup means, and in the picked-up image picked up by the image pickup means, a moving component that moves up and down in the image with the vibration of the vehicle And the frequency analysis is performed on the movement history of the specified moving component, the vibration frequency intensity of the moving component is calculated, and the vibration frequency within a predetermined range is extracted from the calculated vibration frequency intensity of the moving component. Then, based on the extracted frequency intensity within the predetermined frequency range, the captured image captured at the equilibrium point of vibration within the predetermined frequency range is extracted as an equilibrium image, the image speed in the equilibrium image is calculated, and the calculated equilibrium Using the image speed in the image, the image speed generated by steady vehicle vibration is excluded from the captured image, and the image speed generated by steady vehicle vibration is excluded. Based on the image, the center position of the displacement in the image displaced by the vibration of the vehicle is detected, the center position of the detected displacement is monitored, and the amount of image displacement caused by the vibration of the vehicle in the image captured by the imaging means Is calculated.

  According to the present invention, the center position of the displacement in the image displaced by the vibration of the vehicle is detected, the detected center position of the displacement is monitored, and the displacement amount of the image due to the vibration of the vehicle in the captured image is calculated. I tried to do it. As a result, the center position of the displacement in the image can always be grasped and the displacement amount from the center position can be calculated, so that the position change of the object present in the image is caused by the vibration of the vehicle. It is possible to prevent erroneous detection, and it is possible to accurately calculate the displacement amount of the image due to the vibration of the vehicle.

  FIG. 1 is a block diagram illustrating a configuration example of an embodiment of an image processing apparatus according to the present embodiment. The image processing apparatus 100 is mounted on a vehicle and is captured by a camera 101 that captures the front of the vehicle, an image memory 102 that stores an image captured by the camera 101, a control device 103 and a history memory 104, which will be described later, and the camera 101. And a monitor 105 for displaying an image in front of the host vehicle.

  The camera 101 is a high-speed camera having an image sensor such as a CCD or a CMOS, for example, and continuously images the front of the vehicle at a very small constant time Δt interval, for example, 1 ms interval while the host vehicle is running, for each frame. To the image memory 102. As shown in FIG. 2, the camera 101 is installed in front of the interior of the vehicle, its visual axis direction Z is directed to the front front direction of the vehicle, and the horizontal axis X of the imaging surface is parallel to the ground surface. The vertical axis Y of the imaging surface is set to be perpendicular to the ground surface.

  In general, while the vehicle is running, the vehicle is vibrated due to the occurrence of pitching. That is, since vibration does not generally occur when the vehicle is stopped, the frequency of vibration is 0 Hz as shown in FIG. In the example shown in FIG. 3 (a), for example, the vehicle body is sinking due to the weight of the occupant, or a nose dive is generated due to the effect of deceleration before stopping. The state where the vehicle is stopped while maintaining is shown.

  On the other hand, when the vehicle starts running, various vibrations are generated as follows. That is, when the vehicle gets over the undulations and unevenness of the road surface, the vertical vibration of 1 to 2 Hz as shown in FIG. Further, when traveling on a continuous uneven road such as a gravel road, the vibration in the vertical direction of 2 to 15 Hz as shown in FIG. In addition to these steady vibrations, sudden vibrations may occur depending on road conditions. In this way, when vibrations of various frequencies occur depending on the road surface conditions, the camera 101 attached to the vehicle also vibrates at the same time as shown in FIG. 2, and the visual axis direction of the camera 101 changes up and down. The object reflected in the captured image under the influence moves up and down with the time change.

  For this reason, when the image picked up by the camera 101 is directly output to the monitor 105 and displayed, the image in front of the host vehicle is displayed while moving up and down, which is difficult for the user to see. In order to avoid this, in the present embodiment, the control device 103 detects the vibration of the vehicle from the captured images continuously captured by the camera 101, and calculates the vertical displacement amount of the captured image caused by the vibration. . Then, the horizontal center of the image displacement is calculated, and the display image position is corrected so that the center is always displayed at the horizontal center of the monitor 105. As a result, the center of the image displacement is always displayed at the center of the monitor 105 regardless of the vibration of the vehicle, and the display can be easily viewed by the user. Also, in image processing for detecting road surface markers such as white lines and obstacles around the vehicle, image vibration caused by camera vibration can be eliminated, so that an easy-to-process image can be acquired.

  The control device 103 executes the correction process described above as follows. First, in order to detect vertical vibration from a captured image captured by the camera 101, a moving component having a density gradient in the vertical direction is specified in the captured image. For example, since a horizontal edge (horizontal edge) extending in the horizontal direction in the captured image has a density gradient in the vertical direction, in the present embodiment, for example, a lateral edge detection Sobel filter is applied to the captured image. A horizontal edge image is calculated by applying (vertical direction Sobel filter), and a horizontal edge in the captured image is detected. Then, a predetermined range including an arbitrary horizontal edge in the calculated horizontal edge image, for example, a range of 2 pixels × 3 pixels is set as a detection region.

  The horizontal edge image is represented by an xy coordinate system in which the upper left of the image is the origin, the horizontal direction is the x axis, and the vertical direction is the y axis, and the coordinate value located at the center point of the set detection area is the horizontal edge image. The coordinate value of the detection area in the inside. Then, in the horizontal edge image of each frame captured continuously, the detection area is moved along with the vertical change of the horizontal edge position in the set detection area, and the horizontal edge is tracked in the detection area. The y coordinate value of the detection area is stored in the history memory 104 over time.

  After that, when the history stored in the history memory 104 is accumulated for a predetermined time, for example, 1 second, frequency analysis is performed by performing Fourier transformation on the history of the y coordinate value of the detection region, that is, the time series y coordinate. Then, as shown in FIG. 4A, a graph showing the relationship between the vibration frequency in the detection region and the power spectrum intensity indicating the intensity of each frequency is created. In the graph of FIG. 4A, a predetermined range 4b including the frequency 4a having the highest power spectrum intensity is detected, and the band pass filter shown in FIG. 4B is applied to extract the range. Thereby, as shown in FIG. 4C, a predetermined range 4b including a frequency having the highest power spectrum intensity is extracted. The extracted frequency range 4b corresponds to a range of vibration frequencies that occur most frequently, that is, regularly in the vehicle within a predetermined time when data is read from the history memory 104.

  In FIG. 4A, when detecting the predetermined range 4b including the frequency 4a having the highest power spectrum intensity, as described above with reference to FIG. Considering that vibration with a frequency of 2 to 15 Hz is likely to occur constantly, a range in which the frequency 4a with the highest power spectrum intensity exists is detected from these two ranges. Therefore, the band-pass filter shown in FIG. 4B is prepared with the band-pass filter for extracting the two types of frequencies described above, and an appropriate band-pass filter is obtained according to the detection result in FIG. Select and apply a filter.

  Next, an inverse Fourier transform is performed on the extraction result in the range 4b shown in FIG. 4C, and the time change of the displacement amount in the vibration frequency range 4b in the detection region is calculated as shown in FIG. Then, for example, a plurality of points with a displacement amount of 0 as indicated by reference numeral 5 a are extracted in succession, and a captured image captured at a point with the extracted displacement amount of 0 is read from the image memory 102. As a result, among the vibrations generated during traveling of the host vehicle, the image captured at the vibration equilibrium point in the frequency range 4b having the highest frequency of occurrence, that is, the influence of the vibrations that the camera 101 is constantly generating. It is possible to extract an equilibrium image that has been captured in a state where it has not been received.

  A horizontal edge extraction process is performed on each balanced image read from the image memory 102, and the horizontal edges of each balanced image are vertically extracted using a known gradient method or block matching method. The movement speed in the direction and the optical flow in which the movement direction, that is, the velocity component is represented by a vector are calculated. Accordingly, the vertical image speed and the speed direction (balanced speed direction) in the balanced image can be calculated. Next, in each balanced image, the average value of the calculated balanced image velocities of all the horizontal edges is calculated, and the balanced image speed Vcnv in each balanced image is calculated. Note that the equilibrium image speed Vcnv can also be calculated from the time change amount of the displacement amount at the point where the displacement amount is 0, for example, as indicated by reference numeral 5b, that is, the longitudinal change velocity.

  Since the balanced image is an image captured when the camera 101 is not affected by vibration within the vibration frequency range 4b as described above, the image speed in the vertical direction at this time is normally 0. Become. Therefore, the calculated equilibrium image speed Vcnv corresponds to the image speed generated due to the vibration that is constantly generated in the vehicle. In consideration of this, the vibration component contained in the image is removed by the vibration that is constantly generated in the vehicle using the balanced image speed Vcnv. That is, first, the image speed of each image captured by the camera 101 is calculated. Then, by subtracting the balanced image speed Vcnv from the image speed of each image, it is possible to correct so as to exclude the image speed generated by the steady vibration of the host vehicle.

  For example, as shown in FIG. 6A, when the velocity component is calculated from each horizontal edge extracted from the captured image, the image is corrected by subtracting the equilibrium image velocity Vcnv from each calculated velocity component and correcting it. As shown in FIG. 6B, the influence of vibration is eliminated. That is, since the captured image shown in FIG. 6A is captured when the vehicle is moving from the top to the bottom due to the vibration that is constantly generated, on the captured image, within the vibration frequency 4b. The velocity component is detected in a state in which the upward image velocity is added under the influence of the vibration of. On the other hand, by subtracting the equilibrium image speed Vcnv from each speed component, the upward image speed added under the influence of vibration in the vibration frequency 4b is subtracted as shown in FIG. The influence of vibration that occurs constantly is eliminated.

As shown in FIG. 7, with respect to this corrected captured image (corrected image), a y-axis 7a is set at the center in the horizontal direction of the image, and an x-axis 7b orthogonal to the y-axis 7a is set. Then, the distribution of the moving speed in the y-axis 7a direction in the corrected image is measured as shown in FIG. As a result, the coordinate value y _{0 of the} point where the moving speed in the y-axis 7a direction becomes ₀ is calculated, and this y ₀ is set as the vanishing point 7c of the vibration in the corrected image. The vanishing point 7c set here corresponds to the center position of the displacement in the image displaced by the vibration of the vehicle.

Then, a tracking area having a predetermined size, for example, 2 pixels × 3 pixels, including a pixel having a density gradient in the vertical direction, for example, a pixel in which a horizontal edge is detected, is set around the set vanishing point 7c. For example, as shown in FIG. 8 (a), two points whose y coordinate value is y ₀ ± α centered on the y coordinate value y ₀ of the vanishing point 7c are respectively set as an upper limit and a lower limit, and predetermined in the x-axis direction. A tracking area 8a having a width is set.

The tracking area is set to each captured image continuously captured by the camera 101, and pixels having a density gradient in the vertical direction included in the tracking area 8a are tracked and monitored. That is, the tracking area 8a is moved together as shown in FIG. 8B while tracking the pixels having a density gradient in the vertical direction included in the tracking area 8a as they move in the image due to the vibration of the vehicle. Moving. A y-coordinate value y ₁ of the center of the tracking region 8a in this case, by taking the difference between the y-coordinate value y ₀ of the vanishing point 7c described above, the amount of the moved track region 8a with the vibration of the vehicle, i.e., imaging The amount of displacement 8b from the equilibrium point due to image vibration can be calculated.

Then, the image position is corrected and displayed on the monitor 105 so that the displacement amount becomes 0, that is, the y coordinate value of the tracking area 8a moved with the vibration of the vehicle becomes y ₀ , so that it is always displayed. The center of the image displacement is displayed so as to be the center of the monitor 105, and the display can be easily viewed by the user. At the same time, by storing the y-coordinate value positioned at the center point of the tracking area 8a set here in the history memory 104, the tracking area 8a can be made the above-described detection area, and the road surface that changes sequentially. It is possible to correct the image position corresponding to the vibration generated based on the situation.

  FIG. 9 is a flowchart showing processing of the image processing apparatus 100 in the present embodiment. The processing shown in FIG. 9 is executed as a program started by the control device 103 when the power of the image processing device 100 is turned on by turning on the ignition switch of the vehicle on which the image processing device 100 is mounted. In step S10, capturing of a captured image from the image memory 102 is started, and the process proceeds to step S20. In step S20, a detection region for detecting vehicle vibration from the vertical displacement of the object in the captured image is set. Thereafter, the process proceeds to step S30, where the history of the position change of the set detection area, that is, the y coordinate value history of the detection area is accumulated in the history memory 104, and the process proceeds to step S40.

  In step S40, it is determined whether or not the history of position change of the detection area has been accumulated in the history memory 104 by a predetermined amount or more, that is, by a predetermined time or more. If it is determined that a predetermined amount or more of the history of position change of the detection area has been accumulated, the process proceeds to step S50. In step S50, the vibration frequency is analyzed by performing Fourier transform on the time-series change of the displacement amount of the detection area based on the accumulated history of the position change of the detection area. Thereafter, the process proceeds to step S60, where the vibration frequency having the highest power spectrum intensity is identified based on the analysis result of the vibration frequency, and the vibration having the highest power spectrum intensity is performed using a bandpass filter corresponding to the identified frequency range. A predetermined range including the frequency is extracted. Thereafter, the process proceeds to step S70.

  In step S70, inverse Fourier transform is performed on the power spectrum intensity of the vibration frequency within a predetermined range including the vibration frequency having the highest power spectrum intensity, and the time series change of the displacement amount within the vibration frequency range is performed. Is calculated. Thereafter, the process proceeds to step S80, where the above-described equilibrium image is extracted based on the time series change of the displacement amount within a predetermined range including the calculated vibration frequency having the highest power spectrum intensity, and the process proceeds to step S90. In step S90, the optical flow of the extracted equilibrium image is calculated to calculate the equilibrium image velocity Vcnv, and the process proceeds to step S100. In step S100, the image speed is corrected by subtracting the equilibrium image speed Vcnv from each speed component in the calculated captured image. Then, it progresses to step S110.

  In step S110, the distribution of the moving speed in the y-axis direction in the corrected image obtained by correcting the image speed is measured, and as a result, the point at which the moving speed in the y-axis direction becomes 0 is set as the vanishing point. Thereafter, the process proceeds to step S120, a tracking area including pixels having a density gradient in the vertical direction around the vanishing point is set, and the process proceeds to step S130. In step S130, pixels having a density gradient in the vertical direction included in the tracking region are tracked, and the difference between the y coordinate value of the tracking region at that time and the y coordinate value of the vanishing point is obtained, and After calculating the amount of displacement from the equilibrium point due to vibration, the process proceeds to step S140.

  In step S140, based on the calculated displacement amount, the image position is corrected so that the center of the image displacement always becomes the center of the monitor 105 and displayed on the monitor 105. Thereafter, the process proceeds to step S150, and it is determined whether or not the ignition switch of the vehicle on which the image processing apparatus 100 is mounted is turned off. If it is determined that the ignition switch is not turned off, the process proceeds to step S160, the tracking area set in the above-described process is set as a detection area, and the process returns to step S30. On the other hand, if it is determined that the ignition switch has been turned off, the process proceeds to step S170, the capture of the image from the image memory 102 is terminated, and the process is terminated.

According to the present embodiment described above, the following operational effects can be obtained.
(1) The center position of the displacement in the image displaced by the vibration of the vehicle is detected as the vanishing point, the vanishing point position in the image continuously captured by the camera 101 is monitored, and the displacement amount of the image is calculated. I did it. Thereby, the center position of the displacement in the image can always be grasped, and the displacement amount from the center position can be calculated, so that the displacement amount of the image caused by the vibration of the vehicle can be accurately calculated.
(2) In a tracking region set in a predetermined range centering on the vanishing point, a pixel having a density gradient in the vertical direction included in the tracking region is tracked, and the y coordinate value of the tracking region is the y coordinate value of the vanishing point. Thus, the image position is corrected and displayed on the monitor 105. As a result, the center of the image displacement is always displayed so as to be the center of the monitor 105, and the display can be easily viewed by the user.
(3) The band-pass filter shown in FIG. 4B is a vehicle that has a frequency of 1 to 2 Hz or a frequency of 2 to 15 Hz in a vehicle. A bandpass filter for extracting types of frequencies is prepared in advance, and which bandpass filter is used is selected according to the frequency having the highest power spectrum intensity detected from FIG. I made it. Accordingly, it is possible to apply an appropriate bandpass filter corresponding to the type of vibration that changes according to the road surface condition.

The image processing apparatus according to the above-described embodiment can be modified as follows.
(1) In the above-described embodiment, in order to detect the vertical vibration from the captured image captured by the camera 101, as a moving component having a density gradient in the vertical direction in the captured image, The example in which the edge in the horizontal direction is detected and used has been described. However, the present invention is not limited to this, and a moving component having a density gradient in the vertical direction in the captured image may be specified based on other attention points.

(2) In the above-described embodiment, the example in which the frequency analysis is performed by performing the Fourier transform on the history of the y coordinate value of the detection area stored in the history memory 104 has been described. However, the present invention is not limited to this, and frequency analysis may be performed by other methods such as DCT.

(3) In the above-described embodiment, the bandpass filter shown in FIG. 4B is considered in consideration that the vehicle is likely to generate vibration with a frequency of 1 to 2 Hz or vibration with a frequency of 2 to 15 Hz. Thus, an example of preparing a band pass filter for extracting these two types of frequencies in advance and selecting which band pass filter to use according to the frequency with the highest power spectrum intensity has been described. . However, the present invention is not limited to this, and the frequency of vibration generated in the vehicle may be further finely classified so that three or more types of bandpass filters can be selected.

(4) Instead of classifying the vibration frequencies that are likely to occur, a band-pass filter that extracts a different frequency band for each vehicle type on which the image processing apparatus is mounted may be used. This makes it possible to select a more appropriate band-pass filter in consideration of the fact that the frequency of vibration generated generally differs depending on the vehicle type.

(5) In the above-described embodiment, the example in which the camera 101 is installed so as to capture the front of the vehicle as shown in FIG. 2 and the pitching amount of the vehicle is measured has been described. However, the present invention is not limited to this, and the pitching amount of the vehicle may be measured by installing the camera 101 around the vehicle, for example, so as to image the rear of the vehicle, or the camera 101 may image the side of the vehicle. It may be installed to measure the rolling amount of the vehicle.

(6) In the above-described embodiment, the example in which the image processing apparatus 100 according to the present invention is mounted on a vehicle has been described. However, the present invention is not limited to this, and may be mounted on another moving body.

  Note that the present invention is not limited to the configurations in the above-described embodiments as long as the characteristic functions of the present invention are not impaired.

  The correspondence between the constituent elements of the claims and the embodiment will be described. The camera 101 corresponds to an imaging unit, the history memory 104 corresponds to a storage unit, and the monitor 105 corresponds to a display unit. The control device 103 corresponds to specifying means, frequency intensity calculating means, frequency extracting means, image speed calculating means, exclusion means, center position detecting means, displacement amount calculating means, and display position correcting means. This correspondence is an example, and the correspondence is different depending on the configuration of the embodiment.

It is a block diagram which shows the structural example of one Embodiment of an image processing apparatus. It is a figure which shows the example of installation to the vehicle of the camera. It is a figure which shows the kind of typical vibration which generate | occur | produces in a vehicle. It is a figure which shows the specific example in the case of performing a Fourier-transform with respect to the log | history of the y coordinate value of the detection area memorize | stored in the log | history memory 104, and extracting a specific frequency range. It is a figure which shows the time change of the displacement amount in a specific frequency range. It is a figure which shows the specific example in the case of calculating the image after correction | amendment based on an equilibrium image speed. It is a figure which shows the specific example which sets a vanishing point. It is a figure which shows the specific example which sets the tracking area | region on a captured image and calculates the displacement amount of the image resulting from the vibration of a vehicle. It is a flowchart figure which shows the process of the image processing apparatus 100 in this Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 101 Camera 102 Image memory 103 Control apparatus 104 History memory 105 Monitor

Claims (5)

An image processing apparatus that is mounted on a vehicle and performs image processing on an image captured by an imaging unit,
In the captured image captured by the imaging unit, a moving component specifying unit that specifies a moving component that moves up and down in the image with the vibration of the vehicle,
Storage means for storing a movement history of the moving component specified by the moving component specifying means;
Frequency intensity calculating means for performing frequency analysis on the movement history of the moving component stored in the storage means, and calculating the intensity of the vibration frequency of the moving component;
A frequency extraction means for extracting a predetermined range of vibration frequencies that are constantly generated in the vehicle from the vibration frequency intensity of the moving component calculated by the frequency intensity calculation means;
Based on the frequency intensity within the predetermined frequency range extracted by the frequency extraction means, the captured image captured at the equilibrium point of vibration within the predetermined frequency range is extracted as an equilibrium image, and the image speed in the equilibrium image is calculated. Image speed calculating means;
Excluding means for excluding the image speed generated by steady vehicle vibration from the captured image using the image speed in the equilibrium image calculated by the image speed calculating means;
Center position detecting means for detecting a center position of displacement in the image displaced by the vibration of the vehicle, based on the captured image in which the image speed generated by the steady vibration of the vehicle is eliminated by the exclusion means;
Displacement amount calculating means for monitoring the center position of the displacement detected by the center position detecting means and calculating the displacement amount of the image caused by the vibration of the vehicle in the image captured by the image capturing means. Image processing device. The image processing apparatus according to claim 1.
Display means for displaying an image captured by the imaging means;
An image processing apparatus comprising: a display position correcting unit that corrects a display position of the image on the display unit based on a displacement amount of the image caused by the vibration of the vehicle calculated by the displacement amount calculating unit. The image processing apparatus according to claim 1 or 2,
The image processing apparatus, wherein the frequency extraction unit changes a range of frequencies to be extracted based on the intensity of the vibration frequency of the moving component calculated by the frequency intensity calculation unit. The image processing apparatus according to claim 1 or 2,
The image processing apparatus according to claim 1, wherein the frequency extracting unit changes a range of frequencies to be extracted according to a vehicle type of a vehicle on which the image processing apparatus is mounted. An image processing method for performing image processing on an image mounted on a vehicle and captured by an imaging means,
In the captured image captured by the imaging means, identify a moving component that moves up and down in the image with the vibration of the vehicle,
Perform frequency analysis on the movement history of the identified moving component, calculate the intensity of the vibration frequency of the moving component,
From the calculated vibration frequency intensity of the moving component, extract a predetermined range of vibration frequencies that occur constantly in the vehicle,
Based on the extracted frequency intensity within the predetermined frequency range, the captured image captured at the equilibrium point of vibration within the predetermined frequency range is extracted as an equilibrium image, and the image speed in the equilibrium image is calculated,
Using the calculated image speed in the equilibrium image, the image speed generated by steady vehicle vibration is excluded from the captured image,
Based on the captured image excluding the image speed generated by the steady vehicle vibration, the center position of the displacement in the image displaced by the vehicle vibration is detected,
An image processing method characterized by monitoring a center position of a detected displacement and calculating a displacement amount of an image caused by vibration of a vehicle in an image captured by the imaging means.

Images