JP2007199932A - Image processor and its method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor for detecting a dangerous obstacle jumping out ahead the traveling information of one's own vehicle by using one TV camera. <P>SOLUTION: An image input part 10 acquires a time series image from one TV camera mounted on a traveling object, and a featured point extraction part 20 extracts ground featured points belonging to the ground position of an obstacle from an original image, and a moving orbit extraction part 30 extracts a moving orbit on the image of the ground featured points, and an obstacle detection part 40 judges that the ground featured points are belonging to the obstacle when the terminal point of the extracted moving orbit is present at a position closer to a direction where the traveling object exists on the image than a straight line connecting a dissipating point and the start point of the moving orbit. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and method for detecting a dangerous obstacle that pops out ahead of host vehicle travel information using an image of a camera attached to a moving body typified by a vehicle such as an automobile.

  As a method for detecting an obstacle that jumps forward, a method using a radar and a method using an image have been proposed.

  As a method using a radar, Patent Document 1 proposes a detection method using a laser radar. The method using laser radar has a problem that it is vulnerable to bad weather such as rain. Also, when detecting obstacles diagonally forward, such as jumping out, the laser radar is illuminated diagonally, so it is difficult to receive reflected waves, and because the range that can be irradiated is narrow, obstacles diagonally forward such as jumping out are detected. Is difficult.

  The image obtained from the camera has a wider field of view than the radar, and it is possible to shoot an obstacle in front of the diagonal as well as an obstacle in front. In addition, since the travel lane of the host vehicle can be detected, there is an advantage that a region that is dangerous for the travel of the host vehicle can be specified. As a method using an image, there are a method using a single TV camera and a method using a plurality of TV cameras.

  As a method using a plurality of TV cameras, in Patent Document 2, the distance to a vehicle is measured by stereo vision, a movement vector of another vehicle in a four-dimensional scene is obtained, and when an extension line and a driving lane cross each other, A detection method has been proposed. This stereo method cannot detect until the interrupting vehicle enters the common visual field region, so that the detection is slow. Another problem is that the amount of calculation increases because parallax is obtained by matching points between the left and right images.

  As a method using a single TV camera, Patent Document 3 proposes a method of detecting an obstacle by paying attention to a change in the number of edge points on a road surface. In this method, it is assumed that an obstacle exists when the preset maximum value of the number of edge points in a certain area is equal to or greater than a certain threshold value. In this method, since detection is performed from the number of edge points, if there is a pattern on the road surface, the number of edge points increases even if there are no obstacles, so there is a possibility of erroneous detection.

In an apparatus that detects an obstacle by analyzing a motion vector (optical flow) of each point of an image (see, for example, Patent Document 4, Patent Document 5, and Non-Patent Document 1), the optical flow changes depending on the movement of the host vehicle. Therefore, stable detection is difficult. Although the movement on the image caused by the shaking of the host vehicle can be canceled using a method such as Patent Document 6, there are cases where various moving bodies are present in the image, so that the movement cannot be canceled well.
JP-A-7-89367 JP 11-213295 A Japanese Patent Laid-Open No. 8-249597 JP 2000-285393 A JP-A-7-239998 JP 2002-112252 A N. Takeda, et.al., "Moving Obstacle Detection Using Residual Error of FOE Estimation", Proc. Of IROS, pp. 1857 to 1865, 1993.

  It is an object of the present invention to reliably detect a dangerous obstacle that pops out ahead of travel information of a moving body using a time-series image obtained from an image input means attached to the moving body such as a vehicle. .

  The present invention provides an image input means for acquiring a time-series image attached to a moving body that moves on a plane in real space, and an original image that is an image at a certain time in the time-series image, A feature point extracting means for extracting a feature point representing a point or region that may belong to a ground contact position on a plane as a ground feature point; a vanishing point estimating means for estimating a vanishing point on the time-series image; The movement trajectory extracting means for extracting the movement trajectory of the ground feature point from each image of the time-series image, and the end point of the movement trajectory is bounded by a straight line connecting the vanishing point and the start point of the movement trajectory. An image processing apparatus comprising: obstacle detection means for determining that an object including the ground feature point is an obstacle when the body is in an existing region.

  The present invention provides an imaging unit that acquires a time-series image of a moving body in a traveling direction, and an obstacle candidate extraction unit that extracts an obstacle candidate and its estimated ground contact position from an image at a first time of the time-series image. A vanishing point estimating means for estimating a vanishing point on the time-series image, and tracking for obtaining a destination position of the estimated ground position from an image at a second time after the first time of the time-series image. The obstacle candidate having the estimated ground contact position when the moving object is in a region on the side where the moving object exists with a straight line connecting the vanishing point and the estimated ground contact position as a boundary. And an obstacle detection means for detecting an obstacle as an obstacle.

  The present invention provides an imaging unit that acquires a time-series image of a moving body in a traveling direction, and an obstacle candidate extraction unit that extracts an obstacle candidate and its estimated ground contact position from an image at a first time of the time-series image A vanishing point estimating means for estimating a vanishing point on the time-series image, and tracking for obtaining a destination position of the estimated ground position from an image at a second time before the first time of the time-series image. And the movement destination position is in a region opposite to the side where the moving object exists with a straight line connecting the vanishing point and the estimated grounding position as a boundary, the estimated grounding position having the estimated grounding position. An image processing apparatus comprising: obstacle detection means for detecting an obstacle candidate as an obstacle.

  ADVANTAGE OF THE INVENTION According to this invention, the dangerous obstacle which jumps out ahead of the progress information of a moving body can be detected reliably.

  Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings.

(First embodiment)
(1) Configuration of Image Processing Device FIG. 1 shows a configuration of an image processing device according to the first embodiment.

  The image input unit 10 acquires a time series image (xy image) from one TV camera attached to a moving body such as a vehicle.

  In the feature point extraction unit 20, a point or a region feature point (hereinafter referred to as a grounding feature point) that may belong to the grounding position of the obstacle from an image at a certain time in the time series image (hereinafter referred to as an original image). ).

  The movement trajectory extraction unit 30 extracts a movement trajectory on the time-series image of the ground feature points extracted by the feature point extraction unit 20.

  In the obstacle detection unit 40, the end point of the movement track extracted by the movement track extraction unit 30 is in a direction where the moving object exists on the image rather than the straight line connecting the vanishing point on the time-series image and the start point of the movement track. When it is in a close position, it is determined and detected that the point or region from which the movement trajectory is extracted belongs to an obstacle.

  The obstacle information output unit 50 calculates information about the obstacle, such as the distance to the detected obstacle and the time until the collision, to the vehicle control device and the vehicle driver on which this device is mounted. Output.

  Hereinafter, the feature point extraction unit 20, the movement trajectory extraction unit 30, the obstacle detection unit 40, and the obstacle information output unit 50 will be described. In addition, each function of each part 20-50 is realizable by the program memorize | stored in the computer.

  For the sake of simplicity, it is assumed that the TV camera is mounted with the road surface facing down on the image and the optical axis parallel to the road surface and parallel to the traveling direction. In this camera arrangement, the horizon is horizontal on the image and is a straight line passing through the center of the image. The same method can be applied to other installation methods. Further, as shown in FIG. 4, the xy image has an x axis in the horizontal direction and a y axis in the vertical direction with the upper left corner of the image as the origin.

(2) Feature point extraction unit 20
The feature point extraction unit 20 extracts a feature point of a point or a region that may belong to an obstacle grounding position from the original image, that is, a grounding feature point. The grounding feature point is a point where the luminance distribution in the vicinity of the point is different from the surrounding luminance distribution and may belong to the grounding position of the obstacle.

  A feature point extraction method will be described with reference to FIG.

  First, an edge image is generated by detecting an edge in the vertical direction y by applying differentiation in the horizontal direction x to the original image (xy image).

  Next, a curve (edge curve) indicating the edge of the edge image is extracted.

  Edge curves are extracted as follows. When there is an edge (pixel B) in a surrounding pixel (pixel existing in the surrounding eight directions) with respect to a certain edge (pixel A), it is determined that these pixels belong to the same edge curve. Whether or not there is an edge around the pixel (pixel B) having the newly added edge information is similarly examined. What is generated by repeating these procedures is called an edge curve. The point at the lower end of this edge curve or the feature point of the region is set as the ground feature point.

(3) Movement locus extraction unit 30
FIG. 2 shows a detailed configuration example of the movement trajectory extraction unit 30. The movement trajectory extraction unit 30 includes a pitching motion correction unit 31, a yawing motion correction unit 32, a vertical direction end point setting unit 33, and a movement trajectory calculation unit 34.

(3-1) Pitching motion correction unit 31
The pitching motion correction unit 31 removes the influence of the vehicle pitching motion caused by road surface unevenness on the motion of each point on the image.

  Since the pitching motion of the vehicle is substantially a rotational motion around the horizontal axis of the camera, this rotational amount is estimated from the xy image, and the image may be converted so that the rotational amount becomes zero. An existing method is used as a method for estimating the rotational motion around the horizontal axis of the camera from the xy image. For example, Patent Document 6 uses the fact that the vertical motion appearing around the horizon is caused by the rotational motion around the horizontal axis of the camera, and the rotational motion around the horizontal axis is changed from the vertical motion appearing around the horizon. Estimated. At this time, since the approximate position of the horizon in the image is often known when the camera is attached to the moving body, the position of the horizon can be determined when the camera is attached.

  When a wide-angle lens with a small focal length is used in the image input unit 10, the influence of the pitching motion on the image is small and can be ignored. In such a case, the system may be configured without the pitching motion correction unit 31.

(3-2) Yawing motion correction unit 32
The yawing motion correction unit 32 removes the influence that the yawing motion of the vehicle caused by the vehicle traveling on the curve has on the motion of each point on the image. The yawing motion correction unit 32 also plays a role of estimating a vanishing point on the xy image.

  Since the yawing motion of the vehicle is substantially a rotational motion around the vertical axis of the camera, the rotation amount is estimated from the xy image, and the image may be converted so that the rotation amount becomes zero. There are various methods for estimating the rotational motion around the vertical axis of the camera from the xy image. For example, in Patent Document 6, the lateral movement that appears around the vanishing point is caused by the rotational movement around the vertical axis of the camera, and the lateral movement that appears around the vanishing point is rotated from the lateral movement that appears around the vanishing point. By estimating the motion, the influence of the yawing motion on the image change can be removed.

  At this time, since the approximate position of the vanishing point is often known when the camera is attached to the moving body, the position of the vanishing point can be determined when the camera is attached.

(3-3) Longitudinal End Point Setting Unit 33
The vertical direction end point setting unit 33 sets the end point position y1 of the vertical direction component y on the image of the movement trajectory to be extracted.

The end point position y1 is relative to the start point position y0 on the image of the movement trajectory of the ground contact point of interest.

y1 = y0 + Δy

It becomes. However, Δy is a positive number. The reason why Δy is a positive number is that feature points that may collide always move downward on the xy image (see FIG. 5).

(3-4) Movement locus calculation unit 34
The movement trajectory calculation unit 34 calculates the movement trajectory of the feature points until the noticed grounding feature point reaches the vertical end point set by the vertical end point setting unit 33. The movement trajectory calculation unit 34 includes an existence position search unit 341 and an end point determination unit 342.

  The existence position search unit 341 searches for where a certain ground feature point in the original image at a certain time t0 has moved in the subsequent image at the time t1. In the search, the difference in luminance distribution between the neighboring pixel (original region) including the ground feature point at time t0 and the region in the image at time t1 is obtained, and the region where the difference is the smallest is the position where the ground feature point has moved. To do. Differences in luminance distribution are evaluated using the sum of absolute values of differences between corresponding pixels or the sum of values obtained by squaring the differences between pixels.

  The end point determination unit 342 determines whether the moving position of the ground feature point searched by the presence position search unit 341 has reached the end point position y1 set by the vertical direction end point setting unit 34. If the movement position is lower than the end point position y1, the calculation ends with that position as the end point of the movement track.

(4) Obstacle detection unit 40
The obstacle detection unit 40 determines and detects whether the movement trajectory extracted by the movement trajectory extraction unit 30 belongs to an obstacle.

  The x coordinate of the end point of the movement locus extracted by the movement locus extraction unit 30 is compared with the x coordinate at the same y coordinate as the end point of the straight movement locus connecting the vanishing point on the image and the starting point of the movement locus. When the x coordinate of is located in a position close to the direction in which the moving object exists on the image, it is determined and detected that the point or region from which the moving locus is extracted belongs to the obstacle.

  The case of FIG. 5 will be described as an example.

  In the original image at time t = To (upper left figure), there is a building (not an obstacle) on the left side of the screen, and a human (obstacle) popping out is shown on the right side of the screen. From this situation, it is assumed that both the building and the human feature point have reached the end point position at time t = T1 (To <T1: lower left figure) and the movement trajectory is obtained. The movement trajectories of the building and the human are respectively as shown in (a) and (b) in the determination diagram (lower right diagram). At this time, the x-coordinate of the end point of (a) and the x-coordinate on the same y-coordinate (y1) as the end point of (a) of the straight line connecting the start point and the vanishing point of (a) match and do not belong to the obstacle. Determined. At this time, the x coordinate of the end point of (b) and the same y coordinate (y1) as the end point of (b) of the straight line connecting the start point and the vanishing point of (b) are (c) and moved. It is determined that the object is located near the body and belongs to the obstacle.

(5) Obstacle information output unit 50
The obstacle information output unit 50 calculates information about the obstacle, such as the distance to the detected obstacle and the time until the collision, to the vehicle control device and the vehicle driver on which this device is mounted. Output.

  If the movement locus on the xy image of the obstacle detected by the obstacle detection unit 40 is used, the relative speed of the obstacle with respect to the host vehicle in the real space can be obtained.

  The correspondence between the position (xo, yo) on the image and the relative coordinates (Xo, Yo, Zo) with the camera in the real space as the origin can be easily calculated from the known geometric positional relationship between the camera and the road surface. Yes, but one of X, Y and Z will be assumed. Here, the movement trajectory extracted by the movement trajectory extraction unit 30 indicates the movement of the ground contact position of the obstacle on the image. That is, it can be assumed that (Yo = 0). Therefore, if the camera is assumed to be at a height h from the road surface in the assumed camera arrangement, the road surface is a plane (y = −h) perpendicular to the image plane, and therefore,

(Xo, Yo, Zo) = (− (xo · h) / yo, 0, −f · h / yo)

Using this equation, the relative movement of the obstacle relative to the host vehicle in the real space can be obtained from the movement locus of the obstacle contact position on the xy image. The relative speed Vo = (Vox, 0, Voz) is obtained from the relative movement information of the obstacle with respect to the own vehicle in the real space. The relative velocity Vo = (Vox, 0, Voz) can be calculated by various methods. For example, there are the following methods: It is obtained using the time and relative coordinates of the start point and end point of the movement trajectory.

  2. It is obtained using the relative coordinates of any two times of the movement trajectory.

  The predicted course (Xc, Yc, Tc) of the obstacle can be obtained from the relative speed thus obtained.

  The predicted course is obtained by assuming a constant velocity motion or a constant acceleration motion of the obstacle. When this predicted path passes through the origin, the obstacle and the host vehicle collide. By obtaining Tc that the predicted course satisfies the coordinate value (0, 0, Tc), the result of collision at time Tc is output. Actually, since the predicted course of the obstacle rarely passes through the origin, it is assumed that a collision occurs at time Tc when the predicted course of the obstacle passes near the origin.

  The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of the present invention.

1 is a block diagram of an image processing apparatus according to an embodiment of the present invention. It is a block diagram which shows the structure of one Embodiment of the movement locus | trajectory extraction part of FIG. It is explanatory drawing about the coordinate system of an xy image. It is explanatory drawing about the method of extracting a feature point. It is explanatory drawing about the method of determining the movement locus | trajectory which belongs to an obstruction.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image input part 20 Feature point extraction part 30 Movement locus extraction part 40 Obstacle detection part 50 Obstacle information output part

Claims (13)

Imaging means for acquiring a time-series image of the moving direction of the moving body;
Obstacle candidate extraction means for extracting an obstacle candidate and its estimated ground contact position from the image at the first time of the time series image;
Vanishing point estimating means for estimating vanishing points on the time-series image;
Tracking means for obtaining a destination position of the estimated ground contact position from an image at a second time after the first time of the time-series image;
When the destination position is in a region on the side where the moving object exists with a straight line connecting the vanishing point and the estimated ground contact position, the obstacle candidate having the estimated ground contact position is defined as an obstacle. Obstacle detection means to detect as,
An image processing apparatus comprising: An imaging means for acquiring a time-series image of the moving direction of the moving body;
Obstacle candidate extraction means for extracting an obstacle candidate and its estimated ground contact position from the image at the first time of the time series image;
Vanishing point estimating means for estimating vanishing points on the time-series images;
Tracking means for obtaining a destination position of the estimated ground contact position from an image at a second time before the first time of the time-series image;
The obstacle candidate having the estimated ground contact position when the destination position is in a region on the opposite side to the side where the mobile object exists with a straight line connecting the vanishing point and the estimated ground contact position as a boundary. Obstacle detection means for detecting the obstacle as an obstacle,
An image processing apparatus comprising: Image input means for acquiring a time-series image attached to a moving body that moves on a plane in real space;
Feature point extraction that extracts a feature point representing a point or region that may belong to the ground contact position on the plane from the original image that is an image at a certain time in the time series image as a ground feature point Means,
Vanishing point estimating means for estimating vanishing points on the time-series image;
Movement trajectory extracting means for extracting the movement trajectory of the ground feature point from each image of the time-series image;
When the end point of the movement trajectory is in a region where the moving object exists with a straight line connecting the vanishing point and the start point of the movement trajectory, the object including the ground feature point is determined to be an obstacle. Obstacle detection means to
An image processing apparatus comprising: The movement trajectory extraction means includes
Vertical end point setting means for setting an end point position of a vertical component on the original image in the movement locus;
A movement trajectory calculating means for calculating a movement trajectory of the grounding feature point until the end point position is reached;
The image processing apparatus according to claim 3, further comprising: The movement trajectory calculating means includes
Presence position search means for searching in which position in the image at a later time the ground feature point in the original image is located;
End point determination means for determining whether the search position has reached the end point position;
The image processing apparatus according to claim 4, further comprising: The image processing apparatus according to claim 1, further comprising a pitching motion correction unit that corrects an influence due to the pitching motion of the moving body. The image processing apparatus according to claim 3, further comprising a yawing motion correction unit that corrects an influence caused by a yawing motion of the moving body. Possible to belong to the ground contact position on the plane from the original image that is the image at a certain time in the time series image acquired from the image input device attached to the moving body moving on the plane in the real space Extract feature points that represent characteristic points or areas as ground feature points,
Vanishing point estimating means for estimating vanishing points on the time-series image;
Extracting the movement trajectory of the ground feature point from each image of the time series image,
When the end point of the movement trajectory is in a region where the moving object exists with a straight line connecting the vanishing point and the start point of the movement trajectory, the object including the ground feature point is determined to be an obstacle. An image processing method characterized by: When extracting the movement trajectory,
Set the end position of the vertical component on the original image in the movement trajectory,
The image processing method according to claim 8, further comprising: calculating a movement trajectory of the ground feature point until the end point position is reached. When calculating the movement trajectory,
Searching in which position in the image at the subsequent time the ground feature point in the original image is located;
The image processing method according to claim 9, wherein it is determined whether or not the search position has reached the end point position. Possible to belong to the ground contact position on the plane from the original image which is the image at a certain time in the time series image acquired from the image input device attached to the moving body moving on the plane in the real space A feature point extraction function that extracts a feature point representing a characteristic point or region as a ground feature point;
A vanishing point estimating function for estimating a vanishing point on the time-series image;
A movement trajectory extraction function for extracting the movement trajectory of the ground feature point from each image of the time-series image;
When the end point of the movement trajectory is in a region where the moving object exists with a straight line connecting the vanishing point and the start point of the movement trajectory, the object including the ground feature point is determined to be an obstacle. Obstacle detection function to
Is realized by a computer. The movement trajectory extraction function is:
A vertical direction end point setting function for setting an end point position of a vertical direction component on the original image in the movement locus;
A movement trajectory calculation function for calculating a movement trajectory of the grounding feature point until the end point position is reached;
The image processing program according to claim 11, wherein: The movement trajectory calculation function is:
An existing position search function for searching for a position in the image at a later time of the ground feature point in the original image;
An end point determination function for determining whether or not the search position has reached the end point position;
The image processing program according to claim 12, wherein the image processing program is realized.

Images