JP4040657B2 - Moving object detection device - Google Patents

Description

  The present invention relates to a moving object detection device that detects a moving direction of an object based on image information of the object obtained by two cameras arranged apart from each other.

Two CCD cameras are mounted on the vehicle, and the distance between the object and the vehicle is detected based on the image shift obtained from the two cameras, that is, the parallax, and a pedestrian 30 to 60 meters ahead of the vehicle is detected. Such a crossing object detection device is conventionally known (Patent Document 1).
JP 9-226490 A

  However, since the conventional apparatus determines the possibility of collision using the optical flow direction of the object detected based on the image obtained by the camera as it is, the relative distance between the host vehicle and the object is determined. Depending on the vehicle speed, there is a problem that the determination accuracy is lowered. That is, for example, when the vehicle speed is higher than the moving speed of the object, the optical flow on the image corresponding to the object moving toward the center of the road is actually a vector that goes to the outside of the road. Misunderstandings may occur.

  Therefore, in order to accurately detect the movement vector of the object, the area centroid of the object image included in the image obtained by the camera is used as a determination point representing the position of the object image, and the real space of the determination point If the time-series data indicating the upper position is obtained based on the images obtained by the two cameras, the following problem occurs when the object image touches the edge of the image obtained by the cameras. To do. The area centroid means the centroid when it is assumed that the object image is a thin plate having a uniform thickness in the three-dimensional space.

  That is, as shown in FIG. 23A, when the object 101 is stationary or moving toward the traveling vehicle 102, time t = obtained by the camera 103 mounted on the vehicle 102. The time-series object images 101a, 101b, and 101c at k-2, k-1, and k change as shown by hatching in FIG. 5B, and the area centroid G is leftward on the image. Move to. Determination point of the object image included in this image (image obtained by the right camera 103), that is, determination of the position of the area centroid and the object image (not shown) included in the image obtained by the left camera 104 When the parallax is calculated from the position of the point, the distance between the object 101 and the vehicle 102 is obtained from the parallax, and the relative movement vector of the object 101 is obtained, the object 101 is actually stationary as shown by the solid line in FIG. Alternatively, a relative movement vector as indicated by a broken line can be obtained in spite of the movement in parallel with the traveling direction of the vehicle 102. As a result, although there is no possibility of colliding with the vehicle 102, there is a possibility of erroneous determination that there is a possibility.

  The present invention has been made paying attention to this point, and in the case where the object image obtained by the camera is in contact with the edge of the image, the relative movement direction of the object is erroneously detected. It is an object of the present invention to provide a moving object detection apparatus that can prevent the above-described problem.

  In order to achieve the above object, the invention according to claim 1 is a moving object detection method for detecting a moving direction of the object based on image information of the object obtained by two cameras arranged apart from each other. In the apparatus, an image edge determination means for determining that an object image obtained by at least one of the two cameras is in contact with an edge of the image, and the image edge determination means For an object image determined to be in contact with the edge of the object image, a determination point that moves the determination point representing the position of the object image from the center of gravity of the area of the object image to a predetermined point close to the center of the image It is provided with a moving means and a moving direction detecting means for detecting the moving direction of the object based on the determination point after the movement.

The “predetermined point close to the center of the image” is an end of the object image and the corresponding object image close to the center of the image, or the object image and the corresponding object image close to the center of the image. The end of the circumscribed rectangle.
The determination point moving means uses one of the images obtained by the two cameras as a reference image, and moves the determination point with respect to an object image in the reference image. In addition, it is preferable that the determination point moving means changes the position data acquired and stored in the past to the position data using the moved determination point for the object to which the determination point has been moved.

  According to a second aspect of the present invention, in the moving target object detection device according to the first aspect, among the image information of the target object obtained by the two cameras, the image edge determining means applies an image edge to the edge of the image. The moving direction detecting means further comprises distance calculating means for calculating the distance of the object by performing correlation calculation using the image information of the object determined to be in contact as a reference image and the other as a search image. The moving direction of the object is detected based on the determination point after movement by the determination point moving unit and the distance calculated by the distance calculating unit.

  According to a third aspect of the present invention, in the moving object detection device according to the second aspect, the image edge determination unit determines that both of the object images obtained by the two cameras are in contact with the edge of the image. In this case, the distance calculation unit uses an image having a smaller display area of the object image as a reference image.

  In the case of configuring a distance detection device that detects the distance to the object based on the image information of the object obtained by the two cameras arranged apart from each other, of the two cameras, Based on the object image (OBJR2) included in the image obtained by one of the cameras, a correlation operation is performed on the image obtained by the other camera, and the first corresponding object image obtained as a result is obtained. When (COBJL2) touches the edge of the image, a correlation operation is performed on the image obtained by the one camera based on the first corresponding object image (COBJL2), and the result is obtained. Parallax is calculated based on the second corresponding target object image (COBJR2) and the first corresponding target object image (COBJL2), and the distance from the calculated parallax to the target object is calculated. Masui.

  According to the first aspect of the present invention, for the object image determined to be in contact with the edge of the image, the determination point representing the position of the object image is determined from the area centroid of the object image. Since it is moved to a predetermined point close to the center and the moving direction of the object is detected based on the determination point after the movement, there is a problem such as when the moving direction is detected from the time series data of the area center of gravity of the object image Therefore, an accurate moving direction can be detected.

Embodiments of the present invention will be described below with reference to the drawings.
(First embodiment)
FIG. 1 is a diagram showing a configuration of a moving object detection device mounted on a vehicle according to a first embodiment of the present invention. This device includes two infrared cameras 1R and 1L capable of detecting far infrared rays. The yaw rate sensor 5 for detecting the yaw rate of the vehicle, the vehicle speed sensor 6 for detecting the travel speed (vehicle speed) VCAR of the vehicle, the brake sensor 7 for detecting the operation amount of the brake, and these cameras 1R, 1L An image processing unit 2 that detects an object such as an animal in front of the vehicle based on the image data obtained by the above and issues a warning when there is a high possibility of a collision, a speaker 3 that issues a warning by sound, and a camera 1R Alternatively, a head-up display (hereinafter referred to as “HUD”) for displaying an image obtained by 1L and for allowing the driver to recognize an object having a high possibility of collision. ) And a 4.

  As shown in FIG. 2, the cameras 1R and 1L are arranged in front of the vehicle 10 at almost target positions with respect to the central axis in the lateral direction of the vehicle 10. The optical axes of the two cameras 1R and 1L are They are parallel to each other and fixed so that their height from the road surface is equal. The infrared cameras 1R and 1L have a characteristic that the output signal level increases (the luminance increases) as the temperature of the object increases.

The image processing unit 2 includes an A / D conversion circuit that converts an input analog signal into a digital signal, an image memory that stores a digitized image signal, a CPU (Central Processing Unit) that performs various arithmetic processes, and data that is being calculated by the CPU Output circuit for outputting RAM (Random Access Memory) used for storing CPU, ROM (Read Only Memory) for storing programs and tables executed by CPU, map, etc., driving signal for speaker 3, display signal for HUD4, etc. The output signals of the cameras 1R and 1L and the sensors 5 to 7 are converted into digital signals and input to the CPU.
As shown in FIG. 2, the HUD 4 is provided so that the screen 4 a is displayed at the front position of the driver on the front window of the vehicle 10.

  FIG. 3 is a flowchart showing a processing procedure in the image processing unit 2. First, the output signals of the cameras 1R and 1L are A / D converted and stored in the image memory (steps S11, S12, and S13). The image data stored in the image memory is grayscale image data including luminance information. FIGS. 5A and 5B are diagrams for explaining gray scale images obtained by the cameras 1R and 1L (a right image is obtained by the camera 1R and a left image is obtained by the camera 1L), respectively. The hatched area is an intermediate gradation (gray) area, and the area surrounded by a thick solid line has a high luminance level (at a high temperature) and is an area of an object displayed as white on the screen (hereinafter, “ It is referred to as a “high luminance region”. In the right image and the left image, the horizontal positions of the same target object on the screen are shifted and displayed, and the distance to the target object can be calculated from this shift (parallax).

  In step S14 of FIG. 3, the right image is used as a reference image, and binarization of the digital image data, that is, an area brighter than an experimentally determined luminance threshold value ITH is set to “1” (white), and a dark area is set to “ A process of “0” (black) is performed. FIG. 6 shows an image obtained by binarizing the image of FIG. This figure shows that the hatched area is black and the high luminance area surrounded by the thick solid line is white. In this specification, an image (left image) paired with a reference image (right image) is referred to as a “search image”.

  In the subsequent step S15, the binarized image data is converted into run-length data. FIG. 7A is a diagram for explaining this, and in this figure, whitened areas by binarization are shown as lines L1 to L8 at the pixel level. Each of the lines L1 to L8 has a width of one pixel in the y direction and is actually arranged with no gap in the y direction, but is shown separated for the sake of explanation. The lines L1 to L8 have lengths of 2 pixels, 2 pixels, 3 pixels, 8 pixels, 7 pixels, 8 pixels, 8 pixels, and 8 pixels, respectively, in the x direction. The run length data includes the coordinates of the start point of each line (the leftmost point of each line) and the length (number of pixels) from the start point to the end point (the rightmost point of each line). It is shown. For example, since the line L3 includes three pixels (x3, y5), (x4, y5), and (x5, y5), the run length data is (x3, y5, 3).

  In steps S16 and S17, processing for extracting an object is performed by labeling the object as shown in FIG. 7B. That is, of the lines L1 to L8 converted to run length data, the lines L1 to L3 having portions overlapping in the y direction are regarded as one object 1, the lines L4 to L8 are regarded as one object 2, and the run length data The object labels 1 and 2 are added to. By this processing, for example, the high luminance regions shown in FIG. 6 are grasped as the objects 1 to 4, respectively.

ここで、Ｓは面積、ｘ（ｉ），ｙ（ｉ）は、対象物像を構成する画素の座標である。なお、面積重心Ｇの位置は、外接四角形の重心位置で代用してもよい。 In step S18, as shown in FIG. 7C, the area centroid G, the area S, and the aspect ratio ASPECT of the circumscribed rectangle indicated by the broken line are calculated. The area S is calculated by integrating the lengths of run length data for the same object, and the aspect ratio APECT is calculated as a ratio Dy / Dx between Dy and Dx shown in FIG. The coordinates (xg, yg) are calculated by the following equation (1).

Here, S is the area, and x (i) and y (i) are the coordinates of the pixels constituting the object image. The position of the area centroid G may be substituted by the centroid position of the circumscribed rectangle.

  In step S19, tracking of the object is performed between times, that is, the same object is recognized for each sampling period. The object extracted at time (k + 1) when the time t as an analog quantity is discretized at the sampling period is k and the object images 1 and 2 are extracted at time k as shown in FIG. The identity determination between the object images 3 and 4 and the object images 1 and 2 is performed. Specifically, when the following identity determination conditions 1) to 3) are satisfied, it is determined that the object images 1 and 2 and the object images 3 and 4 are the same, and the object images 3 and 4 are By changing the labels to 1 and 2 respectively, tracking between times is performed.

1) The area centroid position coordinates of the object image i (= 1, 2) at time k are (xi (k), yi (k)), respectively, and the object image j (= 3,4) at time (k + 1). ) Where the area centroid position coordinates are (xj (k + 1), yj (k + 1)),
| Xj (k + 1) −xi (k) | <Δx
| Yj (k + 1) −yi (k) | <Δy
Be. However, Δx and Δy are allowable values of the movement amount on the image in the x direction and the y direction, respectively.

2) When the area of the object image i (= 1, 2) at time k is Si (k) and the area of the object image j (= 3,4) at time (k + 1) is Sj (k + 1),
Sj (k + 1) / Si (k) <1 ± ΔS
Be. However, ΔS is an allowable value of area change.

3) The aspect ratio of the circumscribed rectangle of the object image i (= 1, 2) at time k is ASPECTTi (k), and the aspect ratio of the circumscribed rectangle of the object image j (= 3,4) at time (k + 1) is When ASPECTj (k + 1),
ASPECTj (k + 1) / ASPECTTi (k) <1 ± ΔASPECT
Be. However, ΔASPECT is an allowable value of the aspect ratio change.

8A and 8B, the size of each object image is large, but the object images 1 and 3 satisfy the above-described identity determination condition, and the object image 2 and 4 is recognized to correspond to the object images 1 and 2, respectively, because 4 and 4 satisfy the above-described identity determination condition. The position coordinates of the area centroid of each object image recognized in this way are stored in the memory as time-series position data and used for later calculation processing.
Note that the processing in steps S14 to S19 described above is executed for a binarized reference image (right image in the present embodiment).

  In step S20 of FIG. 3, the vehicle speed VCAR detected by the vehicle speed sensor 6 and the yaw rate YR detected by the yaw rate sensor 5 are read, and the yaw rate YR is integrated over time, whereby the turning angle θr of the host vehicle 10 (see FIG. 18). Is calculated.

  On the other hand, in step S31, in parallel with the processes in steps S19 and S20, a process (FIG. 4) for calculating the object, the vehicle 10, and the distance z is executed. Since this calculation requires a longer time than steps S19 and S20, it is executed in a longer cycle than steps S19 and S20 (for example, a cycle about three times the execution cycle of steps S11 to S20).

  In step S41 of FIG. 4, 1) by selecting one of the object images tracked by the binarized image of the reference image (right image), the right image is displayed as shown in FIG. 9A. The object image OBJR1 (here, the entire region surrounded by the circumscribed rectangle is taken as the object image) is extracted. 2) An image corresponding to the object image OBJR1 from the search image (left image) (hereinafter, “corresponding object” A search area for searching for “image” is set, and correlation calculation is executed.

ここで、ＩＲ（ｍ，ｎ）は、図１０に示す対象物像ＯＢＪＲ１内の座標（ｍ，ｎ）の位置の輝度値であり、ＩＬ（ａ＋ｍ−Ｍ，ｂ＋ｎ−Ｎ）は、探索領域内の座標（ａ，ｂ）を基点とした、対象物像ＯＢＪＲ１と同一形状の局所領域ＬＲ内の座標（ｍ，ｎ）の位置の輝度値である。
基点の座標（ａ，ｂ）を変化させて相関度パラメータＣ（ａ，ｂ）が最小となる位置及びそのときの相関度パラメータの値（以下「最小値」という）ＣＭＩＮを求める。 Specifically, as shown in FIG. 9B, the search area SL1 is set in the search image in accordance with the vertex coordinates of the object image OBJR1, and the degree of correlation with the object image OBJR1 in the search area SL1. Correlation degree parameter C (a, b) indicating the height of is calculated by the following equation (2). The correlation parameter C indicates that the smaller the value, the higher the correlation. This correlation calculation is performed using a gray scale image instead of a binary image. When there is past position data for the same object, an area SL1a (indicated by a broken line in FIG. 9B) narrower than the search area SL1 is set as a search area based on the position data.

Here, IR (m, n) is a luminance value at the position of coordinates (m, n) in the object image OBJR1 shown in FIG. 10, and IL (a + m−M, b + n−N) is within the search region. The luminance value at the position of the coordinates (m, n) in the local region LR having the same shape as the object image OBJR1 with the coordinates (a, b) of
By changing the coordinates (a, b) of the base point, the position where the correlation parameter C (a, b) is minimized and the value of the correlation parameter at that time (hereinafter referred to as “minimum value”) CMIN are obtained.

  In the subsequent step S42, it is determined whether or not the degree of correlation is high, that is, whether or not the minimum value CMIN calculated in step S41 is equal to or less than a predetermined threshold CMINTH. When CMIN> CMINTH and the degree of correlation is low, it is determined that no corresponding object image exists in the search image. In this case, since the parallax cannot be calculated, the distance is not determined (indefinite) (step S49), and this process ends.

  On the other hand, if CMIN ≦ CMINTH and the degree of correlation is high, the process proceeds to step S43. In steps S43 to S45, cases C0 to C4 are determined depending on whether or not the object image in the reference image and / or the corresponding object image in the search image is in contact with (applies to) the edge of each image. The distance is calculated according to the case.

  FIG. 11 shows a state in which the object images OBJ1 to OBJ3 in the reference image are in contact with the end portion (hereinafter referred to as “image end”) WR of the reference image, and the object image OBJ4 is not in contact with the image end WR. ing. Since each object image is grasped by the run-length data as described above, the object image in which the coordinates of the start point indicated by the black circle in the same figure are present at the image end WR is in contact with the image end WR. Determined. Similarly, an object image in which an end point coordinate obtained by adding a length (run length) to the start point coordinate is present at the image end WR is determined to be in contact with the image end WR.

  FIG. 12 shows a case where the object image in the reference image and the corresponding object image in the search image are not in contact with the image end (case C0), and the object image in the reference image is in contact with the image end. Although there is no corresponding object image in the search image, the case (case C2) is shown. In the case C0 shown in the figure, the object image OBJR1 in the reference image and the corresponding object image COBJL1 extracted by the correlation calculation are not in contact with the image edge, so that the parallax is obtained using the area centroid G of both. And the distance z to the object can be calculated. Since the correlation calculation is actually performed on a region slightly wider than the object image OBJR1 so as to include a little background portion, the corresponding object image COBJL1 is a true corresponding object image having the same shape as the object image OBJR1. It is extracted as an area slightly wider than OBJL1.

On the other hand, in the case C2, the object image OBJR2 is not in contact with the image end, but the true corresponding object image having the same shape as the object image OBJR2 is positioned as indicated by OBJL2 by a solid line and a broken line ( The broken line portion indicates a portion that is not actually obtained as an image), and the corresponding target object image COBJL2 extracted by the correlation calculation is extracted in a state of being in contact with the image end of the search image.
Note that areas SL1 and SL2 shown in FIG. 12 are search areas.

  FIG. 13A shows a case where the object image OBJR1 in the reference image is in contact with the image end and the true corresponding object image OBJL1 in the search image is not in contact with the image end (case C1). FIG. 6B shows the case C2 described above, that is, the case where the object image OBJR2 in the reference image is not in contact with the image end, but the true corresponding object image OBJL2 in the search image is in contact with the image end. FIG. 8C shows that the object image OBJR3 in the reference image is in contact with the image end, and the true corresponding object image OBJL3 in the search image is also in contact with the image end, and in the reference image The case where the object image OBJR3 is smaller than the true corresponding object image OBJL3 (case C3) is shown, and FIG. 4D shows that the object image OBJR4 in the reference image is in contact with the image end and the search image. The corresponding target object image COBJL4 In contact with the image end and when toward the object image OBJR4 in the reference image is larger than the true corresponding object image OBJL4 shows the (Case C4). In FIG. 13, areas SL1 to SL4 and SR1 and SR2 are search areas.

  In case C1, since the true corresponding object image OBJL1 included in the search image is not in contact with the image end, correct matching is performed and the corresponding object image COBJL1 is extracted. Accordingly, the parallax is calculated using the area centroid of the object image OBJR1 and the corresponding object image COBJL1, and the distance to the object is calculated.

  In Case C2, the corresponding target object image COBJL2 extracted by the correlation calculation from the reference image to the search image and the target object image OBJR2 are not in a correct matching relationship. Therefore, in this case, a correlation calculation from the search image to the reference image, that is, a correlation calculation for searching the target image corresponding to the corresponding target image COBJL2 from the reference image is executed. Since the corresponding target object image COBJR2 in the reference image obtained as a result and the corresponding target object image COBJL2 in the search image are in a correct matching relationship, the parallax is calculated using the area centroids of these images. Calculate the distance to the object.

In the case C3, correct matching is performed as in the case C1, and therefore, the parallax is calculated using the area centroids of the object image OBJR3 and the corresponding object image COBJL3, and the distance to the object is calculated.
In the case C4, as in the case C2, the corresponding target object image COBJL4 extracted by the correlation calculation from the reference image to the search image and the target object image OBJR4 do not have a correct matching relationship. Therefore, also in this case, the correlation calculation from the search image to the reference image, that is, the correlation calculation for searching the target image corresponding to the corresponding target image COBJL4 from the reference image is executed. Since the corresponding target object image COBJR4 in the reference image obtained as a result and the corresponding target object image COBJL4 in the search image are in a correct matching relationship, the parallax is calculated using the area centroids of these images. Calculate the distance to the object.

  Returning to FIG. 4, in step S43, whether the object image in the reference image is not in contact with the image end and the corresponding object image in the search image is not in contact with the image end, that is, corresponds to case C0. If the result of the correlation calculation corresponds to case C0, the process immediately proceeds to step S48, the distance is calculated from the parallax, and this process is terminated.

  When the result of the correlation calculation does not correspond to the case C0, whether or not the object image in the reference image is in contact with the image edge and the corresponding object image in the search image is in contact with the image edge, that is, in case C4. It is discriminated whether or not it corresponds, and when the result of the correlation calculation corresponds to case C4, the correlation calculation from the search image to the reference image is executed as described above (step S46). Next, it is determined whether or not the degree of correlation is high, that is, whether or not the minimum value CMIN of the degree of correlation parameter is equal to or smaller than a predetermined threshold CMINTH (step S47). If CMIN> CMINTH, the process proceeds to step S49. If CMIN ≦ CMINTH and the degree of correlation is high, the process proceeds to step S48 to calculate the distance.

  When the correlation calculation result does not correspond to case C4 in step S44, that is, corresponds to cases C1 to C3, it is determined whether or not the corresponding object image is in contact with the image end, that is, corresponds to case C2. (Step S45) When the case corresponds to the case C2, the process proceeds to Step S46. If the case corresponds to case C1 or C3, the process immediately proceeds from step S45 to step S48.

ここで、Ｂは基線長、すなわち図１５に示すようにカメラ１Ｒの撮像素子１１Ｒの中心位置と、カメラ１Ｌの撮像素子１１Ｌの中心位置との水平方向（ｘ方向）の距離（両カメラの光軸の間隔）、Ｆはレンズ１２Ｒ、１２Ｌの焦点距離、ｐは、撮像素子１１Ｒ、１１Ｌ内の画素間隔であり、Δｄ（＝ｄＬ−ｄＲ）が視差である。
以上のようにして距離ｚの算出が行われる。 The process for calculating the distance from the parallax in step S48 is performed as follows.
As shown in FIG. 14, the distance dR (number of pixels) between the area centroid position of the object image OJBR1 and the left end of the image and the distance dL (number of pixels) between the area centroid position of the corresponding object image COBJL1 and the left end of the image are obtained. Applying the following equation (3), the distance z between the host vehicle 10 and the object is calculated.

Here, B is the base line length, that is, the distance in the horizontal direction (x direction) between the center position of the image sensor 11R of the camera 1R and the center position of the image sensor 11L of the camera 1L as shown in FIG. (Axis interval), F is the focal length of the lenses 12R and 12L, p is the pixel interval in the image sensors 11R and 11L, and Δd (= dL−dR) is parallax.
The distance z is calculated as described above.

Returning to FIG. 3, in step S21, the coordinates (x, y) in the image and the distance z calculated by the equation (3) are applied to the following equation (4) to convert them into real space coordinates (X, Y, Z). To do. Here, as shown in FIG. 16A, the real space coordinates (X, Y, Z) are obtained by using the position of the middle point (the position fixed to the host vehicle 10) of the camera 1R, 1L as the origin O. The coordinates in the image are defined as x in the horizontal direction and y in the vertical direction with the center of the image as the origin, as shown in FIG.

  Here, (xc, yc) is the coordinate (x, y) on the right image based on the relative position relationship between the mounting position of the camera 1R and the real space origin O and the center of the image. Are converted into coordinates in a virtual image. F is a ratio between the focal length F and the pixel interval p.

  When the real space coordinates are converted, the object image (example shown in FIG. 11) determined to be in contact with the image edge among the object images in the reference image by the process of step S31 described above. Then, for the object images OBJ1 to 3), as shown in FIG. 17A, the determination point representing the position of the object image is the end of the circumscribed square that is closer to the center of the reference image from the area centroid G. To the point GE, and the coordinates after the movement are converted into real space coordinates. That is, for the object image OBJ1 that is in contact with the left end of the reference image, the determination point is moved to a point GE1 that has the same y coordinate as the y coordinate of the area centroid G1 and is located at the right end of the circumscribed rectangle. For the object image OBJ2 that is in contact with the upper end of the reference image, the determination point is moved to a point GE2 that has the same x coordinate as the x coordinate of the area centroid G2, and is located at the lower end of the circumscribed rectangle, and is in contact with the right end of the reference image. For the object image OBJ3, the determination point is moved to a point GE3 that has the same y coordinate as the y coordinate of the area centroid G3 and is located at the left end of the circumscribed rectangle.

For the object whose determination point has been moved in this way, the past position data also corresponds to the point GE at the end of the circumscribed rectangle that is closer to the center of the reference image from the data corresponding to the area centroid G. Process to change to data. By doing in this way, the time series data used for calculating the relative movement vector, which will be described later, becomes time series data unified at the determination points after the movement, and an accurate relative movement vector can be obtained.
Note that, as shown in FIG. 17B, the determination points may be moved to points GE1a, GE2a, and GE3a located at the ends of the object image, instead of the ends of the circumscribed rectangle.

In step S22, the turning angle correction for correcting the positional deviation on the image due to the turning of the host vehicle 10 is performed. As shown in FIG. 18, when the host vehicle turns, for example, to the left by the turning angle θr during the period from time k to (k + 1), on the image obtained by the camera, as shown in FIG. This is a process for correcting this. Specifically, the corrected coordinates (Xr, Yr, Zr) are calculated by applying the real space coordinates (X, Y, Z) to the following equation (5). The calculated real space position data (Xr, Yr, Zr) is stored in the memory in association with each object. In the following description, the coordinates after the turning angle correction are displayed as (X, Y, Z).

In step S23, for the same object, N pieces of real space position data (for example, about N = 10) after turning angle correction obtained during the period of ΔT, that is, time-series data are used. An approximate straight line LMV corresponding to the relative movement vector between the object and the vehicle 10 is obtained. Specifically, when a direction vector L = (lx, ly, lz) (| L | = 1) indicating the direction of the approximate straight line LMV, a straight line represented by the following formula (6) is obtained.

Here, u is a parameter that takes an arbitrary value, and Xav, Yav, and Zav are the average value of the X coordinate, the average value of the Y coordinate, and the average value of the Z coordinate of the real space position data string, respectively. Equation (6) becomes the following equation (6a) if the parameter u is deleted.
(X-Xav) / lx = (Y-Yav) / ly = (Z-Zav) / lz
... (6a)

  FIG. 20 is a diagram for explaining the approximate straight line LMV, and P (0), P (1), P (2),..., P (N-2), P (N-1) in FIG. The time series data after the turning angle correction is shown, and the approximate straight line LMV passes through the average position coordinates Pav (= (Xav, Yav, Zav)) of this time series data, and the average value of the square of the distance from each data point Is obtained as a straight line that minimizes. Here, the numerical value in () attached to P indicating the coordinates of each data point indicates that the data increases as the value increases. For example, P (0) indicates the latest position coordinates, P (1) indicates the position coordinates one sample period before, and P (2) indicates the position coordinates two sample periods before. The same applies to D (j), X (j), Y (j), Z (j) and the like in the following description.

More specifically, a vector D (j) = (DX (j), DY (j), DZ (j) from the average position coordinate Pav to the coordinates P (0) to P (N−1) of each data point. ) = (X (j) −Xav, Y (j) −Yav, Z (j) −Zav)) and the inner product s of the direction vector L are calculated by the following equation (7), and the variance of the inner product s is maximum. A direction vector L = (lx, ly, lz) is obtained.
s = lx · DX (j) + ly · DY (j) + lz · DZ (j) (7)

The variance-covariance matrix V of the coordinates of each data point is expressed by the following equation (8). Since the eigenvalue σ of this matrix corresponds to the variance of the inner product s, the maximum of the three eigenvalues calculated from this matrix is The eigenvector corresponding to the eigenvalue is the direction vector L to be obtained. In order to calculate eigenvalues and eigenvectors from the matrix of equation (8), a method known as the Jacobian method (for example, shown in “Numerical Calculation Handbook” (Ohm)) is used.

Next, the latest position coordinate P (0) = (X (0), Y (0), Z (0)) and (N−1) position coordinate P (N−1) before the sample (before time ΔT) = (X (N-1), Y (N-1), Z (N-1)) is corrected to a position on the approximate straight line LMV. Specifically, by applying the Z coordinates Z (0) and Z (N−1) to the equation (6a), that is, according to the following equation (9), the corrected position coordinates Pv (0) = (Xv (0), Yv (0), Zv (0)) and Pv (N-1) = (Xv (N-1), Yv (N-1), Zv (N-1)) are obtained.

  A relative movement vector is obtained as a vector from the position coordinates Pv (N−1) calculated by Expression (9) toward Pv (0). In this way, by calculating an approximate straight line that approximates the relative movement locus of the object with respect to the host vehicle 10 from a plurality (N) of data within the monitoring period ΔT, the influence of the position detection error is reduced. Thus, the possibility of collision with the object can be predicted more accurately.

  Returning to FIG. 3, in step S <b> 24, it is determined based on the relative movement vector calculated in step S <b> 23 whether or not the object is highly likely to collide with the host vehicle 10, and if so, the speaker 3 and the HUD 4. A process of issuing a warning to the driver by voice and display using After step S24 is executed, the process returns to step S11.

  As described above, in the present embodiment, when the object image in the reference image is in contact with the image edge, the determination point representing the position of the object image is shifted from the center of gravity of the area of the object image. Is moved to a point located at the end of the circumscribed rectangle of the object image (or a point located at the end of the object image) close to the image end of the object image, and the relative movement vector is calculated using the moved determination point. Since the calculation is made, the relative movement direction of the object can be accurately detected.

  Also, for example, as shown in FIG. 13B, based on the object image OBJR2 included in the reference image, a correlation operation is performed on the search image, and the corresponding object image COBJL2 obtained as a result is When contacting the end, a correlation operation is performed on the reference image based on the corresponding target object image COBJL2, and the parallax is determined based on the corresponding target object image COBJR2 and the corresponding target object image COBJL2. Since the distance from the calculated parallax to the target is calculated, the correct distance can be calculated even when the corresponding target image in the search image extracted by the correlation calculation touches the image end. can do.

  In the present embodiment, the image processing unit 2 constitutes an image edge determining unit, a determination point moving unit, and a moving direction detecting unit. More specifically, steps S43 to S45 in FIG. 4 correspond to the image edge determination means, steps S41, S42, S46, S47 in FIG. 4 and step S21 in FIG. 3 correspond to the determination point moving means. Step S23 of No. 3 corresponds to the moving direction detecting means.

(Second Embodiment)
FIG. 21 is a flowchart of obstacle detection / alarm processing according to the second embodiment of the present invention. In this process, step S17a is added to the process shown in FIG. 3, and step S31 in FIG. 3 is changed to step S31a. FIG. 22 is a flowchart showing the distance calculation process in step S31a. In this process, steps S43, S44 and S47 are deleted from the process shown in FIG. 4, and step S46 is changed to step S46a. Except for the above points, the second embodiment is the same as the first embodiment.

In step S17a, among the objects extracted in step S17, a process in which the object image in the reference image is in contact with the image end is excluded from the monitoring target.
In step S31a, the distance calculation process shown in FIG. 22 is executed. If it is determined in step S42 that the degree of correlation is high, it is determined whether or not the corresponding object image obtained by the correlation calculation is in contact with the image end of the search image (step S45). The distance is calculated from (Step S48). On the other hand, when the corresponding target object image is in contact with the image end, processing for removing the target object from the monitoring target is performed (step S46a), and this processing is immediately terminated.

  As described above, in the present embodiment, the object whose object image is in contact with the image end of the reference image and the object whose corresponding object image extracted by the correlation calculation is in contact with the image end of the search image are excluded from the monitoring target. Therefore, it is possible to prevent such a movement direction of the object and the distance to the object from being recognized erroneously.

In the present embodiment, the image processing unit 2 constitutes an image edge determination unit, and more specifically, step S45 in FIG. 22 corresponds to the image edge determination unit.
The present invention is not limited to the embodiment described above, and various modifications can be made. For example, in this embodiment, an infrared camera is used as a camera for obtaining an object image. However, for example, as shown in Japanese Patent Laid-Open No. 9-226490, a television camera that can detect only normal visible light is used. Also good. However, by using an infrared camera, it is possible to simplify the extraction process of animals or traveling vehicles, and it can be realized even if the computing capability of the computing device is relatively low.

It is a block diagram which shows the structure of the moving target object detection apparatus concerning one Embodiment of this invention. It is a figure for demonstrating the attachment position of the camera shown in FIG. It is a flowchart which shows the procedure of the process by the image processing unit of FIG. It is a flowchart which shows the distance calculation process of FIG. 3 in detail. In order to explain a gray scale image obtained by an infrared camera, it is a diagram showing hatching in an intermediate gradation part. It is a figure which shows the black area | region which hatched, in order to demonstrate the image which binarized the gray scale image. It is a figure for demonstrating the conversion process and labeling to run length data. It is a figure for demonstrating tracking between the time of a target object. It is a figure for demonstrating the search target image in a right image, and the search area | region set to a left image. It is a figure for demonstrating the correlation calculation process which made the search area | region object. It is a figure for demonstrating the case where a target object touches the edge part of an image. It is a figure for demonstrating the case where the target object image in a search image touches the edge part of an image. It is a figure for demonstrating the case where a target object image contacts the edge part of an image. It is a figure for demonstrating the calculation method of a parallax. It is a figure for demonstrating the method of calculating distance from parallax. It is a figure which shows the coordinate system in this embodiment. It is a figure for demonstrating the movement of a judgment point. It is a figure for demonstrating a turning angle correction | amendment. It is a figure which shows the shift | offset | difference of the target object position on the image which generate | occur | produces by turning of a vehicle. It is a figure for demonstrating the calculation method of a relative movement vector. It is a flowchart which shows the procedure of the process by the image processing unit in the 2nd Embodiment of this invention. It is a flowchart which shows the distance calculation process of FIG. 21 in detail. It is a figure for demonstrating the problem of the conventional method. It is a figure for demonstrating the problem of the conventional method.

Explanation of symbols

1R, 1L infrared camera (camera)
2 Image processing unit (image edge judging means, judging point moving means, moving direction detecting means)
5 Yaw rate sensor 6 Vehicle speed sensor

Claims (3)

In the moving object detection device for detecting the moving direction of the object based on the image information of the object obtained by the two cameras arranged apart from each other,
Image edge determination means for determining that an object image obtained by at least one of the two cameras is in contact with the edge of the image;
For the object image determined to be in contact with the edge of the image by the image edge determination means, a determination point representing the position of the object image is determined from the area centroid of the object image, and the center of the image Determination point moving means for moving to a predetermined point close to
A moving object detecting device comprising: a moving direction detecting means for detecting a moving direction of the object based on the determination point after the movement. Of the image information of the object obtained by the two cameras, correlation calculation is performed using the image information of the object determined to be in contact with the edge of the image by the image edge determination means as the reference image and the other as the search image. By further comprising a distance calculating means for calculating the distance of the object,
The moving direction detection unit detects a moving direction of the object based on a determination point after movement by the determination point moving unit and a distance calculated by the distance calculation unit. The moving object detection device according to 1.   When it is determined by the image edge determination means that both of the object images obtained by the two cameras are in contact with the edge of the image, the distance calculation means is an image having a smaller display area of the object image. The moving object detection device according to claim 2, wherein the moving object detection device is used as a reference image.

Images