JP2007129560A - Object detector - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately detect an object with small movements such as a walker without being affected by the noise of a sensor or the like. <P>SOLUTION: On the basis of the moving amount of a feature region, a change amount between a first image and a second image, that is the change amount of an image due to the movement of a moving object, is estimated. By applying the estimated change amount of the image to the entire first image, a converted image is generated. The converted image indicates a position where each object probably exists at the time when the second image is imaged in the case of assuming that all the objects included in the image exist on a road plane. A region with the presence of a difference in a differential image indicates an object which actually does not exist on the road plane, and the object is detected on the basis of the region. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an apparatus for detecting an object included in an image based on an image captured by an imaging apparatus such as a camera.

  A method has been proposed in which an image in the traveling direction of a vehicle is taken with an infrared camera or a stereo camera mounted on the vehicle, and an object such as a preceding vehicle or a pedestrian is detected based on the obtained image. However, the method using the infrared camera is effective only at night, and the infrared camera itself is expensive. In addition, stereo cameras have many restrictions when mounted, and are expensive.

  On the other hand, Patent Document 1 describes a method of capturing a plurality of images with a monocular camera at time intervals and detecting a moving object from these images. In the method of Patent Document 1, when a moving body is detected based on the difference between two images before and after a certain time, the movement of the image during the certain time is estimated based on information from the in-vehicle sensor. Then, one image is converted using the motion information of the image obtained by the estimation, and a moving object is detected by obtaining a difference from the other image.

Japanese Patent No. 3463858

  However, in the method of Patent Document 1, since the motion of the image in a certain time is estimated based on the output of the in-vehicle sensor, it is easily affected by sensor measurement noise. For this reason, a lot of noise occurs in the difference image, and there is a problem that it is difficult to detect a moving body with a small movement such as a pedestrian.

  The present invention has been made to solve such problems, and the object of the present invention is to accurately detect an object having a small movement such as a pedestrian without being affected by sensor noise or the like. It is an object of the present invention to provide an object detection apparatus capable of performing the above.

  In one aspect of the present invention, an object detection device that detects an object based on a plurality of images captured at time intervals includes an imaging unit that captures a first image and a second image at time intervals; An estimation means for estimating an image change amount between the first image and the second image based on a movement amount of a feature region on the road surface common to the first image and the second image, and a subject included in the image The first image is converted based on the amount of change in the image assuming that the image is on the road plane, and a conversion image is generated, and a difference image between the converted image and the second image is generated Difference image generating means for detecting the object based on a region where the difference exists in the difference image.

  Said object detection apparatus is mounted in moving bodies, such as a vehicle, for example, and detects the object located in front of the said moving body. The object detection apparatus includes an imaging unit such as a CCD camera, for example, and captures an image of a road plane or the like at time intervals. Images taken at different time intervals are referred to as a first image and a second image. Next, the amount of change between the first image and the second image is estimated based on the amount of movement of the feature region on the road plane that is included in common with the first image and the second image. The feature region includes diagonal lines, signs, and the like fixedly arranged on the road plane. Since the movement amount of the feature region indicates the relative movement amount of the moving body, that is, the imaging unit with respect to the road plane, the change amount between the first image and the second image based on the movement amount of the feature region, that is, The amount of change in the image due to the movement of the moving body is estimated.

  Next, assuming that the subject included in the image is on the road plane, the converted image is generated by applying the estimated change amount of the image to the entire first image. This converted image indicates the position where each subject would have existed at the time when the second image was captured, assuming that all the subjects included in the image were on the road plane. Therefore, a difference image between the converted image and the actually captured second image is generated. When an individual subject actually exists on the road plane, the converted image and the difference image should match for the subject. On the other hand, when the subject does not actually exist on the road plane, the converted image and the difference image do not coincide with each other, and a difference occurs. That is, the area where the difference exists in the difference image indicates an object that did not actually exist on the road plane, and the object is detected based on the area. Specifically, the detected object includes a three-dimensional object present at a position higher than the road plane, a moving object moving with respect to the road plane, and the like.

  In this method, since the amount of image conversion is determined on the assumption that all subjects are on the road plane, the difference between still objects that are on the road plane but have a height of zero or low is brought close to zero accurately. Therefore, it is possible to reliably detect pedestrians with a small amount of movement.

  One aspect of the object detection apparatus includes an object identification unit that identifies the object based on a shape of a region where a difference exists in the difference image. In a preferred example, the shape of the region is a shape of a closed region formed by a region where a difference exists in the difference image or a shape of a region where a difference exists in the difference image. Since the shape of the region where the difference exists corresponds to the contour shape of a stationary solid object or a moving object, the object can be identified by extracting the shape and performing pattern recognition or the like. For example, a vehicle, a pedestrian, or the like can be identified by pattern matching the extracted shape with the shape of a vehicle, the shape of a part of a human body, or the like.

  Another aspect of the object detection apparatus includes an object identification unit that identifies a detected object based on a change in a region where a difference of the difference image exists between difference images at a plurality of time intervals. When the difference image is examined by changing the time interval, the manner in which the region where the difference exists is different depending on whether the object is a moving object or a stationary solid object. Therefore, based on this deformation method, it is possible to identify whether the detected object is a moving object or a stationary solid object. In a preferred example, the object identification unit is configured to increase a detected object with respect to a road plane when a shape of a region where a difference exists in the difference image is deformed toward a main axis in an imaging direction of the imaging unit. It can be identified as a stationary object having a thickness.

  Another aspect of the object detection apparatus includes an object motion detection unit that detects a motion of an object detected based on a region where a difference exists in the difference image. In this aspect, the motion of the object can be calculated by analyzing the shape of the region where the difference exists at time intervals. Further, in this case, it is possible to identify whether the object is a stationary object or a moving object based on the amount of movement of the object detected by the object motion detection means.

  Preferred embodiments of the present invention will be described below with reference to the drawings.

[Object detection device]
FIG. 1 shows a schematic configuration of an object detection apparatus according to the present invention. FIG. 1A is a block diagram of the object detection device 1, and FIG. 1B is a diagram showing a vehicle equipped with the object detection device. In the present embodiment, the object detection device 1 is mounted on a vehicle. As shown in FIG. 1A, the object detection device 1 includes a CCD camera 3 (hereinafter also simply referred to as “camera”), an image recognition ECU 4, a display device 5, a speaker 6, and an actuator 7. As shown in FIG. 1B, the camera 3 is attached to a position in front of the top of the vehicle 10 or the like, and mainly captures an image of a road plane (hereinafter also referred to as “road surface”) 15 in front of the vehicle 10. Therefore, when the object 50 exists on the road surface 15, an image of the road surface 15 including the object 50 is captured.

  The image recognition ECU 4 performs image processing based on a plurality of images captured by the camera 3 at time intervals, and detects an object. For example, when an object such as a forward vehicle or a pedestrian is detected, the image recognition ECU 4 sends an image including the object to the display device 5 and causes the display device 5 to display the image. Further, a warning or the like is displayed on the display device 5 as necessary. The display device 5 can be a display panel positioned in front of the driver's seat of the vehicle, for example. Further, when an object is detected, the image recognition ECU 4 sends a signal to the speaker 6 as necessary, and causes the speaker 6 to output a voice message or the like telling the driver to that effect. Furthermore, when an object is detected, the image recognition ECU 4 supplies signals to various actuators mounted on the vehicle to control the outputs of the engine, brakes, steering, etc. as necessary to avoid danger. To do.

[Object detection principle]
First, the basic principle of object detection according to the present invention will be described. The present invention has one feature in that it can detect a moving object that moves relative to the road surface and an object that has a height relative to the road surface. When the first image is corrected in consideration of the vehicle motion during this time for two images taken at different times, the points on the road surface and at a lower position from the road surface are the two images. The same coordinates and the same brightness, and the difference is zero. On the other hand, since the coordinate position in the two images changes between a point moved with respect to the road surface and a point having a large height from the road surface, a luminance difference occurs. Therefore, an object can be detected based on this difference.

  FIG. 2 schematically shows how the camera 3 moves relative to the road surface as the vehicle moves. For convenience of illustration, the vehicle 10 is not shown. FIG. 2 shows the movement from a certain time t to a time (t + Δt) after the lapse of Δt. The camera image at time t has the main axis (camera optical axis) in the imaging direction of the camera 3 as the Z-axis direction, the lower part of the camera perpendicular to the Z-axis as the Y-axis direction, and the horizontal direction perpendicular to the Z-axis as the X-axis. The direction is indicated by an XYZ coordinate system (also referred to as a “camera coordinate system”). On the other hand, the image of the camera at time (t + Δt) is indicated by the X′Y′Z ′ coordinate system as shown.

Now, the object 50 on the road surface 15 is located at the point M (x, y, z) in the XYZ coordinate system at time t, and the point M _{t + Δt} (in the X′Y′Z ′ coordinate system at time (t + Δt). x ′, y ′, z ′). In this case, the mapping from the point M to the point M _{t + Δt} corresponds to the relative movement of the camera 3 with respect to the road surface 15.

  FIG. 3A shows an example of an image taken at time t, and FIG. 3B shows an example of an image taken at time (t + Δt). When feature areas (for example, white lines, signs, etc. on the road surface) M1 to M4 exist on the road surface 15, the positions of the feature areas M1 to M4 on the captured image are changed according to the movement of the camera 3. Move as shown in 3 (a) and 3 (b). Therefore, in the present invention, the relative movement of the camera with respect to the road surface is estimated by image processing based on the movement amount of the feature region existing on the road surface. Theoretically, by setting four feature regions on the road surface, the relative movement of the camera with respect to the road surface can be calculated. As the feature area, a lane, a sign, or the like on the road surface can be used. Alternatively, a characteristic pattern or the like existing on the road surface may be used as a characteristic region, and estimation may be performed by emphasizing light and darkness of the pattern by image processing.

  Based on the estimated camera motion, a converted image corresponding to the captured image at time t (hereinafter referred to as “converted image”) is generated from the captured image at time (t + Δt). This converted image assumes that the subjects included in the captured image at time (t + Δt) (all elements included in the captured image) are all present on the road surface. Indicates the position that would have been present at t. Therefore, when the converted image is compared with the captured image at the actual time t, all the stationary objects existing on the road surface should be present at the same position. In other words, it can be said that the region where the difference is generated between the converted image and the captured image at the actual time t is a stationary object or a moving object that does not exist on the road surface. Therefore, in the present invention, an object is detected based on this difference. In this method, the relative movement of the camera with respect to the road surface is not estimated from the movement of the vehicle by the in-vehicle sensor, but is estimated from the captured image by image processing, so that the estimation accuracy may be reduced due to the measurement noise of the sensor. It is possible to detect an object with high accuracy.

[Object detection processing]
FIG. 4 shows a flowchart of object detection processing according to the present embodiment. This object detection process is executed by the image recognition ECU 4 shown in FIG.

  First, while the vehicle 10 is traveling, the camera 3 captures an image in front of the vehicle and generates a captured image (step S1). Since the camera 3 captures images at a predetermined time interval, the camera 3 generates a plurality of images captured at the predetermined time interval.

  Next, the image recognition ECU 4 calculates flow parameters based on the obtained captured image, and further calculates vehicle motion parameters and plane parameters based on the flow parameters (step S2). The flow parameter is a parameter indicating a planar optical flow on the road surface. The vehicle motion parameter is a parameter that defines the motion of the vehicle in a predetermined time (for example, Δt seconds), and the plane parameter is a parameter with respect to the camera coordinates of the plane defined on the road surface. Details of this processing will be described later.

  Next, the image recognition ECU 4 generates a plane image conversion formula based on the obtained vehicle motion parameters and plane parameters (step S3). As described above, the planar image conversion formula is generated by image processing based on the amount of movement of the feature region existing on the road surface, and indicates the relative movement amount of the camera with respect to the road surface. Details of this processing will be described later.

  Next, the image recognition ECU 4 performs image conversion using the obtained planar image conversion formula (step S4). Specifically, in step S1, a captured image at a certain time t (hereinafter referred to as a “first image”) and a captured image at a time (t + Δt) after Δt (seconds) has elapsed (hereinafter referred to as a “second image”). Is obtained). The image recognition ECU 4 converts the second image that is new in time using a planar image conversion formula, and generates a converted image (that is, a converted image). As described above, since the planar image conversion equation is an equation indicating the relative movement of the camera with respect to the road surface, the converted image is obtained when it is assumed that all subjects included in the second image are on the road surface. It becomes an image showing the position where those subjects would have existed at t.

  Then, the image recognition ECU 4 calculates the difference between the converted image and the actual captured image at time t, that is, the first image (step S5). In the case of a stationary object on the road surface, the movement from the position at time t to the position at time (t + Δt) is due to the relative movement of the camera with respect to the road surface. Therefore, for a stationary object on the road surface (hereinafter, “still object on the road surface”), the position of the object in the converted image obtained in step S4 and the position of the object in the first image captured in step S1 Match.

  On the other hand, for a stationary object that exists on the road surface but has a height (hereinafter also referred to as a “stationary solid object”), since the object has a height component, from the position at time t to time (t + Δt). The movement to the position cannot be expressed only by the relative movement of the camera with respect to the road surface. Therefore, the position of the object in the converted image obtained in step S4 does not match the position of the object in the first image imaged in step S1. In addition, an object that exists on the road surface but is moving (hereinafter also referred to as a “moving object”) has a moving component of the moving object itself, so that the position at time t changes to the position at time (t + Δt). This movement cannot be expressed only by the relative movement of the camera with respect to the road surface. Therefore, the position of the object in the converted image obtained in step S4 does not match the position of the object in the first image imaged in step S1.

  Therefore, by calculating the difference between the converted image and the first image, it is possible to detect a stationary solid object and a moving object included in the captured image. Specifically, the image recognition ECU 4 compares the difference obtained in step S5 with a predetermined threshold for each pixel included in the image. The position of a pixel where a difference equal to or greater than the threshold value can be determined as a stationary solid object or a part of a moving object. Therefore, the shape formed by the set of pixels in which the difference exceeding the threshold is detected can be detected as the contour shape of the object. Thus, in this embodiment, it is possible to detect an object including a stationary solid object and a moving object from a plurality of images having a predetermined time interval.

[Derivation of planar image conversion formula]
Next, the method for deriving the planar image conversion formula in steps S2 to S3 will be described in detail. The flow parameter is calculated using an optical flow model. This model is expressed using eight flow parameters in a known second-order flow model. These parameters are associated with individual positions in the image. In this example, on the premise that the road surface is a plane, a vector connecting the same points in the two images is identified.

  In the three-dimensional space, the movement of the vehicle is expressed using nine parameters as shown in FIGS. 5 (a) and 5 (b). Here, the nine physical parameters are vehicle speeds “a”, “b”, “c” in three directions, angular speeds “w1”, “w2”, “w3” with respect to three axes, and a camera pitch “ θ, roll “φ”, and height “H”.

  For example, if one of the nine physical parameters such as the camera height H is obtained, the other eight physical parameters are calculated using the optical flow parameter to determine the movement of the camera. it can. For example, the movement of the camera can be determined using an extended Kalman filter.

  Specifically, the movement and direction of the vehicle with respect to the road surface are determined by integrating movement parameters (change rates) in three-dimensional coordinates. The motion and direction in the three-dimensional space are estimated based on the optical flow vector. An optical flow vector is a vector connecting the same points in a plurality of images.

  FIG. 6 shows an example of a parameter calculation method. A plurality of images (for example, time t and time (t + Δt) are captured by the camera 3. On the other hand, the vehicle speed, the yaw rate, and the pitch rate are detected by the vehicle-mounted sensor. Based on the captured plurality of images, image processing is performed. Eight parameters of the optical flow and a two-dimensional flow model are calculated, from which the optical flow is processed using a Kalman filter to calculate the movement of the vehicle and the direction of the camera. Is used in image processing to improve the estimation accuracy, and motion detection will be described in detail below.

(1) Definition of Coordinate System and Parameters FIGS. 7 and 8 show three coordinate systems used in this embodiment, that is, an image coordinate system, a camera coordinate system, and a vehicle coordinate system. The image coordinate system is set on the image plane of the camera. The origin of the camera coordinate system is the point where the main axis (camera optical axis) in the imaging direction of the camera intersects the image plane. The vehicle coordinate system is obtained by rotating the camera coordinate system. These coordinate systems move as the camera moves. It is assumed that the road surface appearing in the image is basically a plane.

(2) Planar flow The movement of the camera mounted on the vehicle traveling on the road has six degrees of freedom: vertical speed a _camera , lateral speed b _camera , longitudinal speed c _camera , yaw rate w 1, pitch rate w 2. , Roll rate w3. These motion parameters are measured with respect to the camera coordinate system.

  Assuming that the road surface is flat, the plane equation holds for the camera coordinate system.

ここで、ｐ、ｑ、ｒは平面パラメータである。

Here, p, q, and r are plane parameters.

  The optical flow vector generated by the planar image in the image plane is as follows.

ここで、ｕ及びｖは、それぞれ画像座標系（x,y）におけるフローベクトルの水平及び垂直成分である。ｆはカメラの焦点距離である。Ｕ、Ｖ，Ａ、Ｂ、Ｃ、Ｄ、Ｅ、Ｆは、平面に対するカメラの動き及び方向について決定されたフローパラメータである。フローパラメータは以下の式で与えられる。なお、ｔｄは画像のサンプリング時間である。

Here, u and v are the horizontal and vertical components of the flow vector in the image coordinate system (x, y), respectively. f is the focal length of the camera. U, V, A, B, C, D, E, and F are flow parameters determined for the movement and direction of the camera relative to the plane. The flow parameter is given by: Note that td is an image sampling time.

平面パラメータは、路面に対するカメラの方向各及び位置、即ち、ピッチ各θ、ロール各φ及びカメラ高さＨにより示すことができる。一般的に、ほとんどの車両において、θ、φ及びΔＨ（Ｈの変化）はゼロに等しい。よって、

The plane parameter can be indicated by each direction and position of the camera with respect to the road surface, that is, each pitch θ, each roll φ, and camera height H. In general, in most vehicles, θ, φ and ΔH (changes in H) are equal to zero. Therefore,

となる。上記の近似を用いると、カメラと路面との間の地理的関係から、以下の式が得られる。

It becomes. Using the above approximation, the following equation is obtained from the geographical relationship between the camera and the road surface.

図８に示すように、カメラ座標系は、車両座標系に対してθ及びφだけ回転している。よって、カメラの並進速度ａ_camera、ｂ_camera、ｃ_cameraは、車両の並進速度ａ、ｂ、ｃにより以下のように示される。

As shown in FIG. 8, the camera coordinate system is rotated by θ and φ with respect to the vehicle coordinate system. Therefore, the translation speeds a _camera , b _camera , and c _camera of the _camera are expressed as follows by the translation speeds a, b, and c of the vehicle.

カメラを搭載する際、カメラの軸を、ステアリングをニュートラル位置にした状態における車両の走行方向と正確に一致させることは困難である。よって、カメラと車両の水平方向差を考慮して、パラメータγを導入する。水平方向差はカメラ及び車両の縦方向速度を測定するために使用される。

When mounting the camera, it is difficult to accurately match the camera axis with the traveling direction of the vehicle when the steering is in the neutral position. Therefore, the parameter γ is introduced in consideration of the horizontal difference between the camera and the vehicle. The horizontal difference is used to measure the vertical speed of the camera and vehicle.

  From the above, the flow vector obtained for the vehicle coordinate system is as follows.

こうして得られた車両運動パラメータ（ａ、ｂ、ｃ、ｗ１、ｗ２、ｗ３）及び平面パラメータ（ｐ、ｑ、ｒ）に基づいて、以下の平面画像変換式が得られる。

Based on the vehicle motion parameters (a, b, c, w1, w2, w3) and the plane parameters (p, q, r) thus obtained, the following planar image conversion formula is obtained.

座標（ｘ’、ｙ’）は第２画像の座標であり、座標（ｘ、ｙ）は変換画像の座標である。また、各変数Ｔ１１〜Ｔ３３は以下の式で与えられる。

The coordinates (x ′, y ′) are the coordinates of the second image, and the coordinates (x, y) are the coordinates of the converted image. Each variable T11 to T33 is given by the following equation.

ここで、ｋは未定の比例定数であり、最終的に上記の式（１）に代入して数値が得られる時点で分母・分子で相殺される。

Here, k is an undetermined proportionality constant, and is finally offset into the denominator / numerator when a numerical value is obtained by substituting into the above equation (1).

R11 to R33 are nine elements of a rotation matrix representing rotation of two planes. When the imaging time difference t is small, w1 to w3 in FIG.
R11 ≒ w1
R22 ≒ w2
R33 ≒ w3
R12 = R13 = R21 = R23 = R31 = R32 = 0
And can be approximated. “A”, “b”, and “c” are vehicle translational speed components in three axial directions as shown in FIG. Using the planar image conversion formula thus obtained, the image is converted in step S4.

[Moving object identification processing]
Next, the moving object identification process will be described. As described above, in the object detection process, an object including a stationary solid object and a moving object is detected. In the moving object identification process, the stationary solid object detected by the object detection process and the moving object are identified, that is, distinguished.

  In FIG. 9, the example of the difference image of a pedestrian, a vehicle, and a stationary solid object is shown. The moving object identification process of the present invention is characterized in that an object is identified using a plurality of difference images having different time differences. For example, as shown in FIG. 9, a difference image having a time difference Δt and a difference image having a time difference (Δt × 10) are generated as the difference images. Since the difference image of the time difference Δt has a small time difference, a direct contour line of the object appears as shown in FIGS. 9A and 9C. Therefore, it is possible to identify an object having a known specific contour such as a pedestrian or a vehicle by extracting the shape of a region where a difference exists such as a contour line using a difference image with a small time difference and recognizing the pattern. . In particular, in the case of a pedestrian, a vehicle, or the like, an area where a difference exists is a closed area. Therefore, if a closed area where a difference exists is extracted and pattern recognition is performed, an object can be identified efficiently. For example, a pedestrian can perform pattern recognition based on a circular outline of a human head, a vehicle can perform horizontal / vertical line segments, and the like.

  In the difference image of the time difference (Δt × 10), since the time difference is long, the outlines of two separate moving objects appear as shown in FIGS. 9B and 9D for the moving object. Therefore, pattern recognition can be performed on the distribution pattern itself of the two contour lines. That is, the contour line pattern recognition is performed using the difference image having a small time difference Δt, and the object is more accurately identified by performing the pattern recognition of the contour distribution pattern itself using the difference image having a large time difference Δt. It becomes possible.

  On the other hand, since a stationary solid object is not a moving object, a plurality of contour lines are not obtained. In the case of a stationary solid object, for example, as shown in FIGS. 9 (e) and 9 (f), a difference does not occur or is very small in a portion 90 having a low height (that is, a portion close to the road surface) of the stationary solid object. But. A difference occurs in the high portion 91. That is, when a stationary solid object observes a difference image with a large time difference, a difference does not occur or is very small in a part of the object, and a difference occurs in another part. Further, this difference increases as the height of the stationary solid object increases. Furthermore, as shown in FIGS. 9E and 9F, the shape of the area where the difference exists is deformed toward the imaging direction (optical axis direction) of the camera. Therefore, it is possible to identify a stationary solid object by examining a plurality of difference images having different time differences.

  Even when the feature of the difference image of the stationary solid object as described above cannot be used, the stationary solid object and the moving object can be distinguished by predicting the movement of the object. As described in the object detection process described above, the relative movement of the camera with respect to the road surface can be obtained as a planar image conversion formula. Therefore, for example, using the captured image and the planar image conversion formula at time t, a converted image at the previous time (t−Δt × 10) and time (t−Δt × 20) is generated. Then, the difference between the time (t−Δt × 10) is calculated from the converted image at the time (t−Δt × 10) and the actual captured image at the time (t−Δt × 10). Similarly, the difference at time (t−Δt × 20) is calculated from the converted image at time (t−Δt × 20) and the actual captured image at time (t−Δt × 20).

  When the object is a stationary three-dimensional object, a difference occurs due to the time difference (Δt × 10) as shown in FIG. 9F, but the time difference is the same and the object itself has not moved, so the time (t− The difference of [Delta] t * 10) matches the difference of time (t- [Delta] t * 20). On the other hand, if the object is a moving object, even if the time difference is the same, the object itself is moving, so the difference between the time (t−Δt × 10) and the time (t−Δt × 20) Does not match the difference. Therefore, it is possible to identify whether the detected object is a stationary solid object or a moving object by calculating differences at a plurality of times using a planar image conversion formula and comparing them.

  FIG. 10 shows an example of the moving object identification process. This process is also executed by the image recognition ECU 4 shown in FIG. First, the image recognition ECU 4 predicts the motion of the object based on the above-described planar image conversion formula (step S21), and generates a plurality of motion predicted images at different times based on the prediction. Then, the difference between the motion prediction image at those times and the captured image actually taken at those times is calculated (step S22). And the difference is evaluated and a moving object is identified (step S23). For example, as described above, if two differences are the same or close, it is determined that the object is a stationary solid object, and if the difference is greater than or equal to a predetermined difference, the object is determined to be a moving object.

  Further, the position, height, etc. of the stationary solid object identified as described above can be detected. For a moving object, the distance between the moving object and the vehicle, the moving speed of the moving object, and the like can be detected.

  Specifically, after identifying the stationary solid object and the moving object as described above, each is tracked (tracked) on the image. For a stationary three-dimensional object, the position and height of the object are determined from the combination of the movement of the host vehicle with respect to the road surface and the motion information on the image taking into account the movement of the host vehicle.

  Tracking processing is also performed for moving objects. Since the relative positional relationship between the stationary three-dimensional object and the host vehicle is known, the relative position of the moving object with respect to the vehicle is estimated from the existing position in the image. Regarding the height of the moving object, after tracking the highest part of the moving object parts that have been tracked together, the relative position of the object with the host vehicle and the size in the height direction on the image. Estimate the height of the moving object. This makes it possible to create a map relating to the general situation ahead of the vehicle.

It is a figure which shows schematic structure of the object detection apparatus which concerns on this invention. It is a figure which shows a mode that a camera moves with the movement of a vehicle typically. The example of the image imaged during the movement of a vehicle is shown. It is a flowchart of an object detection process. It is explanatory drawing of an optical flow model. It is a figure explaining the derivation method of a vehicle motion parameter. It is a figure which shows the relationship of a coordinate system. It is a figure which shows a vehicle coordinate system and a camera coordinate system. The example of the difference image used by an object identification process is shown. It is a flowchart of an object identification process.

Explanation of symbols

1 Object detection device 3 CCD camera 4 Image recognition ECU
5 Display device 6 Speaker 7 Actuator 10 Vehicle 15 Road surface (road plane)

Claims (7)

An object detection device for detecting an object based on a plurality of images taken at time intervals,
Imaging means for imaging the first image and the second image at time intervals;
Estimating means for estimating an image change amount between the first image and the second image based on a movement amount of a feature region on the road surface common to the first image and the second image;
An image conversion means for converting the first image based on the amount of change of the image on the assumption that the subject included in the image is on a road plane, and generating a converted image;
Difference image generation means for generating a difference image between the converted image and the second image;
An object detection device comprising: an object detection unit configured to detect an object based on a region where a difference exists in the difference image.   The object detection apparatus according to claim 1, further comprising an object identification unit that identifies the object based on a shape of a region where a difference exists in the difference image.   3. The object detection according to claim 2, wherein the shape of the region is a shape of a closed region formed by a region where a difference exists in the difference image or a shape of a region where a difference exists in the difference image. apparatus.   The object detection device according to claim 1, further comprising an object identification unit that identifies a detected object based on a change in a region where the difference of the difference image exists between the difference images at a plurality of time intervals. .   The object identification means is a stationary object having a height relative to a road plane when the shape of a region where a difference exists in the difference image is deformed toward a main axis in an imaging direction of the imaging means. The object detection device according to claim 4, wherein the object detection device is identified.   The object detection apparatus according to claim 1, further comprising an object motion detection unit configured to detect a motion of an object detected based on a region where a difference exists in the difference image.   7. The object detection according to claim 6, further comprising means for identifying whether the object is a stationary object or a moving object based on the amount of movement of the object detected by the object movement detection means. apparatus.

Images