JP2010092437A - Vehicle periphery monitoring device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To determine a quadruped which moves laterally in front of a vehicle. <P>SOLUTION: Based on a picked up image, a moving object which is moving in the vehicle width direction in front of the vehicle is extracted. When the temporal change of an aspect ratio of a circumscribed quadrangle circumscribed to the moving object is small and the temporal change of the object form picked up in the lower region of the circumscribed quadrangle is large, the moving object is determined to be a quadruped. In the other example, when the temporal change of the object form picked up in the upper region of the circumscribed quadrangle circumscribed to the moving object is small and the temporal change of the object form picked up in the lower region of the circumscribed quadrangle is large, the moving object is determined to be a quadruped. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an apparatus for monitoring the periphery of a vehicle, and more specifically to an apparatus for determining a quadruped animal such as a deer and a bear moving in front of the vehicle.

  Conventionally, an apparatus has been proposed in which an infrared camera is mounted on a vehicle, a pedestrian is detected from captured images around the vehicle imaged by the camera, and the information is provided to the driver of the vehicle. However, an object that may come into contact with the vehicle is not limited to a pedestrian. For example, if a large animal such as a deer is present in front of the host vehicle, the vehicle may be affected.

Patent Document 1 below discloses a technique for detecting an animal by distinguishing it from a pedestrian. According to this method, an animal is detected by extracting an object having a predetermined shape representing an animal from a single captured image by a shape matching method.
JP 2007-310705 A

  Since the quadruped that crosses the front of the vehicle is moving while moving its four legs, the shape of the quadruped animal differs depending on the timing of imaging. Therefore, even if the shape matching technique of a predetermined shape is applied to a single captured image, the detection accuracy may be lowered for detection of such a moving quadruped animal. Therefore, there is a demand for a technique that can detect a crossing quadruped animal with better accuracy.

  According to one aspect of the present invention, a vehicle periphery monitoring device that monitors the periphery of a vehicle using a captured image obtained by an imaging unit mounted on the vehicle extracts an object from the captured image and applies the object to the object. Means for setting a circumscribed rectangle that circumscribes the object, storage means for storing image data of the object to which the circumscribed rectangle is set, time-series, tracking the position of the extracted object, and in front of the vehicle Moving object specifying means for specifying a moving object moving in the width direction, and image data in which the circumscribed rectangle at one time is set from the time-series image data for the specified moving object Time series data extraction means for extracting image data in which the circumscribed rectangle at the other time is set, an aspect ratio of the circumscribed rectangle at the one time, and at the other time The magnitude of the change between the aspect ratio of the circumscribed square is small, and the shape of the moving object imaged in the lower area of the circumscribed square at the one time and the circumscribed at the other time Determination means for determining that the moving object is a quadruped when the magnitude of the change between the moving object and the shape of the moving object imaged in the lower rectangular area is large.

  According to another aspect of the present invention, a vehicle periphery monitoring device that monitors a periphery of a vehicle using a captured image obtained by an imaging unit mounted on the vehicle extracts an object from the captured image and applies the object to the object. Means for setting a circumscribed rectangle that circumscribes the object, storage means for storing image data of the object to which the circumscribed rectangle is set, time-series, tracking the position of the extracted object, and in front of the vehicle Moving object specifying means for specifying a moving object moving in the width direction, and image data in which the circumscribed rectangle at one time is set from the time-series image data for the specified moving object And time-series data extracting means for extracting image data in which the circumscribed rectangle at another time is set, and the transfer imaged in the upper area of the circumscribed rectangle at the one time. The magnitude of the change between the shape of the object and the shape of the moving object imaged in the upper area of the circumscribed rectangle at the other time is small, and the circumscribed rectangle at the one time is When the magnitude of the change between the shape of the moving object imaged in the lower area and the shape of the moving object imaged in the lower area of the circumscribed rectangle at the other time is large, Determining means for determining that the moving object is a quadruped animal.

  According to the present invention, since the quadruped animal is determined based on the temporal change in the shape captured in the lower region of the object image, the quadruped moving in the lateral direction can be detected with better accuracy. Can be determined. In addition, by taking into account the temporal change in the size of the object image or the temporal change in the shape captured in the upper region, it is distinguished with better accuracy from other objects such as pedestrians and cars, Quadruped animals can be determined.

  Other features and advantages of the present invention will be apparent from the detailed description that follows.

  Next, an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of a vehicle periphery monitoring device according to an embodiment of the present invention. The apparatus is mounted on a vehicle and includes two infrared cameras 1R and 1L capable of detecting far-infrared rays, a yaw rate sensor 5 that detects the yaw rate of the vehicle, and a vehicle speed sensor 6 that detects a travel speed (vehicle speed) VCAR of the vehicle. A brake sensor 7 for detecting the operation amount of the brake, an image processing unit 2 for detecting an object in front of the vehicle based on image data obtained by the cameras 1R and 1L, and a sound based on the detection result And a head-up display (hereinafter referred to as HUD) 4 for displaying an image obtained by the camera 1R or 1L and for causing the driver to recognize an object in front of the vehicle. It has.

  As shown in FIG. 2, the cameras 1 </ b> R and 1 </ b> L are disposed in a symmetric position with respect to the central axis passing through the center of the vehicle width at the front portion of the vehicle 10. The two cameras 1R and 1L are fixed to the vehicle so that their optical axes are parallel to each other and their height from the road surface is equal. The infrared cameras 1R and 1L have a characteristic that the level of the output signal becomes higher (that is, the luminance in the captured image becomes higher) as the temperature of the object is higher.

  The image processing unit 2 includes an A / D conversion circuit that converts an input analog signal into a digital signal, an image memory that stores a digitized image signal, a central processing unit (CPU) that performs various arithmetic processing, and a data RAM (Random Access Memory) used to store data, ROM (Read Only Memory) that stores programs executed by the CPU and data used (including tables and maps), driving signals for the speaker 3, display signals for the HUD 4, etc. Is provided. The output signals of the cameras 1R and 1L and the output signals of the sensors 5 to 7 are converted into digital signals and input to the CPU. As shown in FIG. 2, the HUD 4 is provided so that a screen 4 a is displayed at a front position of the driver on the front window of the vehicle 10. Thus, the driver can visually recognize the screen displayed on the HUD 4.

  FIG. 3 is a flowchart showing a process executed by the image processing unit 2. The process is performed at predetermined time intervals.

  In steps S11 to S13, the output signals of the cameras 1R and 1L (that is, captured image data) are received as input, A / D converted, and stored in the image memory. The stored image data is a gray scale image including luminance information.

  Referring now to FIG. 4, (a) and (b) schematically show a right image and a left image of a grayscale image obtained by the cameras 1R and 1L, respectively. The hatched area is an intermediate gradation (gray) area, and represents a difference in gradation depending on the type of hatching. A region not hatched has a high luminance level (that is, a high temperature) and is a region of an object (hereinafter referred to as a high luminance region) displayed on the screen as white or a gradation color close to white. . In the right image and the left image, the horizontal position on the image of the same object is shifted, and the distance to the object can be calculated by this shift (parallax).

  Returning to FIG. 3, in step S14, the right image is used as a reference image (alternatively, the left image may be used as a reference image), and the image signal is binarized. Specifically, a process is performed in which a region brighter than the luminance threshold ITH determined in advance by simulation or the like is set to “1” (white) and a dark region is set to “0” (black). FIG. 5 shows an image obtained by binarizing the image of FIG. In this figure, the black region is represented by a hatched region, and the white region (high luminance region) is represented by a region that is not hatched.

  In step S15, the binarized image data is converted into run-length data. FIG. 6 is a diagram for explaining this process. In FIG. 6 (a), regions that are whitened by binarization are represented by L1 to L8. Each of the lines L1 to L8 has a width of one pixel in the y direction. The lines L1 to L8 are actually arranged without gaps in the y direction, but are shown separated in the figure for convenience. The lines L1 to L8 have lengths of 2 pixels, 2 pixels, 3 pixels, 8 pixels, 7 pixels, 8 pixels, 8 pixels, and 8 pixels, respectively, in the x direction. The run-length data indicates the lines L1 to L8 by the coordinates of the start point (leftmost pixel of each line) of each line and the length (number of pixels) from the start point to the end point (rightmost pixel of each line). )). For example, since the line L3 includes three pixels (x3, y5), (x4, y5), and (x5, y5), the line L3 is represented by run-length data (x3, y5, 3).

  In steps S16 and S17, as shown in FIG. 6B, the object is labeled and a process of extracting the object is performed. That is, of the lines L1 to L8 converted to run length data, the lines L1 to L3 having portions overlapping in the y direction are regarded as one object 1, the lines L4 to L8 are regarded as one object 2, and the run length data The object labels 1 and 2 are given. By this process, for example, four high-luminance regions shown in FIG. 5 are extracted as the objects 1 to 4, respectively.

  In step S18, as shown in FIG. 6C, the center G of the extracted object, the area S, and the aspect ratio ASPECT of the circumscribed rectangle circumscribing the extracted object as shown by the broken line. Is calculated. The area S is calculated by integrating the lengths of run length data for the same object. The coordinates of the center of gravity G are calculated as the x coordinate of a line that bisects the area S in the x direction and the y coordinate of a line that bisects the area S in the y direction. The aspect ratio ASPECT is calculated as a ratio Dy / Dx between the length Dy in the y direction and the length Dx in the x direction of the circumscribed square. Note that the position of the center of gravity G may be substituted by the position of the center of gravity of the circumscribed rectangle.

  In step S19, tracking of the target object for the time (tracking), that is, recognition of the same target object is performed every predetermined sampling period. The sampling period may be the same as the period in which the process of FIG. 3 is performed. Specifically, when the time obtained by discretizing the time t as an analog quantity with the sampling period is k and the objects 1 and 2 are extracted at the time k as shown in FIG. The identity determination between the objects 1 and 2 and the objects 3 and 4 extracted at time (k + 1) which is the next sampling period is performed. In this example, if the following conditions 1) to 3) are satisfied, it is determined that the objects 1 and 2 and the objects 3 and 4 are the same, and the objects 3 and 4 extracted this time are the same. Is changed to labels 1 and 2, respectively, to perform inter-time tracking.

  1) The coordinates of the position of the center of gravity G on the image of the object i (= 1, 2) at the time k are (xi (k), yi (k)), and the object j (= 3) at the time (k + 1). , 4) where | xj (k + 1) −xi (k) | <Δx, and | yj (, where (xj (k + 1), yj (k + 1)) are coordinates of the position of the center of gravity G on the image k + 1) −yi (k) | <Δy. Here, Δx and Δy are predetermined allowable values of the movement amount on the image in the x direction and the y direction, respectively.

  2) The area on the image of the object i = (1,2) at time k is Si (k), and the area on the image of the object j (= 3,4) at time (k + 1) is Sj ( When k + 1), Sj (k + 1) / Si (k) <1 ± ΔS. Here, ΔS is a predetermined allowable value of the area change.

  3) The aspect ratio of the circumscribed rectangle of the object i (= 1, 2) at time k is ASPECTTi (k), and the aspect ratio of the circumscribed rectangle of the object j (= 3,4) at time (k + 1) is ASPECTj ( When k + 1), ASPECTj (k + 1) / ASPECTTi (k) <1 ± ΔASPECT. Here, ΔASPECT is a predetermined allowable value of the aspect ratio change.

  In the example of FIG. 7, between (a) and (b), the objects 1 and 3 satisfy the identity determination condition, and the objects 2 and 4 satisfy the identity determination condition. Therefore, the objects 3 and 4 are determined to be the same as the objects 1 and 2, respectively.

  The above condition is based on the knowledge that if the object is the same between time k and (k + 1), the changes in the center of gravity G, area S, and aspect ratio ASPECT should be less than or equal to a predetermined tolerance. Yes. In order to perform better identity determination, it is preferable to determine identity when all the above three conditions are satisfied, but it is the same when any one or two of the three conditions are satisfied. You may make it determine with sex.

  In this way, at each sampling period, at least for the extracted target object, at least a given label, image data of the target object surrounded by a circumscribed rectangle (binarized), position of the target object (this embodiment) Then, the position coordinates of the center of gravity G) and the aspect ratio ASPECT of the circumscribed rectangle are stored in the memory as time series data. The time series data includes data on N captured images obtained at least within a predetermined monitoring period ΔT.

  Note that the processes in steps S14 to S19 described above are executed for a binarized reference image (in this example, the right image).

  Returning to FIG. 3, in step S20, the vehicle speed VCAR detected by the vehicle speed sensor 6 and the yaw rate YR detected by the yaw rate sensor 5 are read and the yaw rate YR is integrated over time, whereby the turning angle θr of the vehicle 10 (see FIG. 13). (To be described later with reference).

  On the other hand, in steps S31 to S33, a process of calculating a distance z from the vehicle 10 to the object is performed in parallel with the processes of steps S19 and S20. Since this calculation requires a longer time than steps S19 and S20, it may be executed at a cycle longer than that of steps S19 and S20 (for example, a cycle about three times the execution cycle of steps S11 to S20).

  In step S31, by selecting one of the objects tracked by the binarized image of the reference image (in this example, the right image), as shown in FIG. The search image R1 (here, the image region surrounded by the circumscribed rectangle is set as the search image) is extracted. In step S32, a search area in which an image area (hereinafter referred to as a corresponding image) in which the same object as the search image is captured is set in the left image, and a correlation is established between the reference image and the search area. An operation is executed to extract a corresponding image.

In order to reduce the processing load of the correlation calculation, it is preferable to set a search region R2 in the left image as shown in FIG. 8B according to each vertex coordinate of the search image R1 (for example, each vertex) A region within a predetermined range including the coordinates can be set as the search region R2). A luminance difference total value C (a, b) indicating the level of correlation with the search image R1 in the search region R2 is calculated by the following equation (1), and the total value C (a, b) is minimized. A region is extracted as a corresponding image. This correlation calculation is performed using a grayscale image, not a binary image. Alternatively, when there is past position data for the same object, an area R2a (indicated by a broken line in FIG. 8B) narrower than the search area R2 may be set as the search area based on the position data. Good. Thereby, the load of the correlation calculation process can be further reduced.

  Here, the search image R1 has M × N number of pixels, and IR (m, n) is a luminance value at the position of the coordinate (m, n) in the search image R1 shown in FIG. (A + m−M, b + n−N) is a luminance value at the position of the coordinate (m, n) in the local region R3 having the same shape as the search image R1, with the coordinate (a, b) in the search region R2 as a base point. It is. The position of the corresponding image is specified by determining the position where the luminance difference sum C (a, b) is minimized by changing the coordinates (a, b) of the base point. Thus, as shown in FIG. 10, the search image R1 and the corresponding image R4 corresponding thereto are extracted.

In step S33, the distance dR (number of pixels) between the centroid position of the search image R1 and the image center line (a line that bisects the captured image in the x direction) LCTR of the captured image and the centroid position and the image center line LCTR of the corresponding image R4. Distance dL (number of pixels) is calculated and applied to equation (2) to calculate the distance z to the object of the vehicle 10.

  Here, as shown in FIG. 11, B is the base length, that is, the distance in the x direction (horizontal direction) between the center position of the image sensor 11R of the camera 1R and the center position of the image sensor 11L of the camera 1L (that is, both F represents the focal length of the lenses 12R and 12L provided in the cameras 1R and 1L, and p represents the pixel spacing in the image sensors 11R and 11L. Δd (= dR + dL) indicates the magnitude of parallax.

In step S21, the coordinates (x, y) in the image of the position of the object (as described above, the position of the center of gravity G in this embodiment) and the distance z calculated by Expression (2) are expressed in Expression (3). Apply and convert to real space coordinates (X, Y, Z). Here, as shown in FIG. 12A, the real space coordinates (X, Y, Z) are set to the origin O as the midpoint position (position fixed to the vehicle) of the camera 1R and 1L attachment positions. As shown in the figure, it is expressed in a coordinate system in which the X axis is defined in the vehicle width direction of the vehicle 10, the Y axis is defined in the vehicle height direction, and the Z axis is defined in the traveling direction of the vehicle 10. As shown in FIG. 12B, the coordinates on the image are represented by a coordinate system in which the center of the image is the origin, the horizontal direction is the x axis, and the vertical direction is the y axis.

  Here, (xc, yc) is the coordinate (x, y) on the right image based on the relative position relationship between the mounting position of the camera 1R and the origin O of the real space coordinate system and the image. Are converted into coordinates in a virtual image that coincides with the center of the image. F is a ratio between the focal length F and the pixel interval p.

  In step S22, the turning angle correction for correcting the positional deviation on the image due to the turning of the vehicle 10 is performed. As shown in FIG. 13A, when the vehicle 10 turns, for example, to the left by the turning angle θr during the period from time k to (k + 1), the image obtained by the camera is shown in FIG. 13B. As shown, it is shifted in the x direction by Δx. Therefore, this is corrected.

Specifically, the real space coordinates (X, Y, Z) are applied to Equation (4) to calculate the corrected coordinates (Xr, Yr, Zr). The calculated real space position data (Xr, Yr, Zr) for the N captured images obtained within at least the predetermined monitoring period ΔT is stored in the memory in time series in association with each object. . In the following description, the corrected coordinates are indicated as (X, Y, Z).

In step S23, for the same object, from N pieces of real space position data (for example, about N = 10) after turning angle correction calculated for N pieces of captured images obtained within a predetermined monitor ΔT. Then, an approximate straight line LMV corresponding to the relative movement vector of the object with respect to the vehicle 10 is obtained. Specifically, when the direction vector L = (lx, ly, lz) (| L | = 1) indicating the direction of the approximate straight line LMV, the straight line represented by the equation (5) is obtained.

Here, u is a parameter that takes an arbitrary value. Xav, Yav, and Zav are the average value of the X coordinate, the average value of the Y coordinate, and the average value of the Z coordinate of the real space position data string, respectively. By deleting the parameter u, the equation (5) is expressed as the following equation (5a).
(X-Xav) / lx = (Y-Yav) / ly = (Z-Zav) / lz
(5a)

  FIG. 14 is a diagram for explaining the approximate straight line LMV. P (0), P (1), P (2),..., P (N-2), P (N-1) indicate time-series data after the turning angle correction, and the approximate straight line LMV is It is obtained as a straight line that passes through the average position coordinates Pav (= (Xav, Yav, Zav)) of the time series data and has the minimum mean value of the square of the distance from each data point. Here, the numerical value in () attached to P indicating the coordinates of each data point indicates that the data is past data as the value increases. For example, P (0) indicates the latest position coordinates of the sample period, that is, P (1) indicates position coordinates one sample period before, and P (2) indicates position coordinates two sample periods before. The same applies to D (j), X (j), Y (j), Z (j) and the like in the following description.

More specifically, a vector D (j) = (DX (j), DY (j), DZ (j) from the average position coordinate Pav toward the coordinates P (0) to P (N−1) of each data point. )) = (X (j) −Xav, Y (j) −Yav, Z (j) −Zav)) and the inner product s of the direction vector L are calculated by the equation (6), and the variance of the inner product s is maximum. A direction vector (lx, ly, lz) is obtained.

The variance-covariance matrix V of the coordinates of each data point is expressed by Equation (7), and the eigenvalue σ of this matrix corresponds to the variance of the inner product s. Therefore, the largest eigenvalue among the three eigenvalues calculated from this matrix The eigenvector corresponding to is the direction vector L to be obtained. In order to calculate eigenvalues and eigenvectors from the matrix of equation (7), a method known as the Jacobian method can be used.

Next, the latest position coordinate P (0) = (X (0), Y (0), Z (0)) and the position coordinate P (N−) before (N−1) (that is, before time ΔT). 1) = (X (N−1), Y (N−1), Z (N−1)) is corrected to a position on the approximate straight line LMV. Specifically, the corrected position coordinates Pv (0) = (Xv (0) are obtained from Expression (8) obtained by applying Z coordinates Z (0) and Z (n−1) to Expression (5a). ), Yv (0), Zv (0)) and Pv (N-1) = (Xv (N-1), Yv (N-1), Zv (N-1)).

  A vector from the position coordinates Pv (N−1) calculated by Expression (8) toward Pv (0) is calculated as a relative movement vector.

  In this way, by calculating the approximate straight line LMV that approximates the relative movement trajectory of the object with respect to the vehicle 10 from a plurality (N) of data within the monitoring period ΔT, and determining the relative movement vector, the influence of the position detection error is obtained. And the possibility of collision with the object can be predicted more accurately.

  In step S24, an object determination process is performed to determine whether or not the object is to be a target for alerting. If it is determined that it is a target for alerting, the process proceeds to step S25 to perform an alarm determination process. In the alarm determination process, it is determined whether or not an alarm is actually output to the driver. If the determination result is affirmative, an alarm is output.

  The present invention relates to determination of a quadruped moving in the vehicle width direction (X direction) in front of the vehicle. The determination processing of the quadruped is performed in step 24, and details will be described later. . If it is determined that the animal is a quadruped, it is determined that it is a target for alerting as described above. Of course, in addition to a quadruped animal, the determination of a pedestrian may be performed in step 24. If it is determined that the person is a pedestrian, the target is determined to be alerted. In step S24, a process for determining whether the object is an artificial structure may be performed. If it is determined as an artificial structure, it may be excluded from the alerting target. The determination process for a pedestrian and an artificial structure can be performed by any appropriate technique (for example, described in JP-A-2006-185434).

  Next, determination of a quadruped moving in the vehicle width direction (X direction) in front of the vehicle according to one embodiment of the present invention will be described. Before describing a specific determination method, the movement of an object will be described with reference to FIG. 15, and the principle of determination will be described with reference to FIG.

  Referring to FIG. 15, an area (imaging area) AR0 that can be monitored by the cameras 1R and 1L is shown. The area AR1 corresponds to an area corresponding to a range obtained by adding a margin β (for example, about 50 to 100 cm) on both sides of the vehicle width α of the vehicle 10, in other words, the central axis of the vehicle 10 in the vehicle width direction. It is an area having a width of (α / 2 + β) on both sides, and is an approach determination area with a high possibility of collision if the object continues to exist as it is. The areas AR2 and AR3 are areas where the absolute value of the X coordinate is larger than that of the approach determination area (outside in the lateral direction of the approach determination area), and an object in this area may enter the approach determination area ( Therefore, it is called an intrusion determination area). These areas AR1 to AR3 have a predetermined height H in the Y direction and are limited by a predetermined distance Z1 in the Z direction.

  In this embodiment, the latest position Pv (0) of the target object exists in the areas AR1 to AR3, and the target object moving in the X direction in front of the vehicle 10 is set as the determination target. Further, the movement in the X direction includes not only movement from the positive direction of the X axis to the negative direction but also movement from the negative direction to the positive direction.

  Next, referring to FIG. 16, the object moving in the vehicle width direction (X direction) in front of the vehicle is (a) an animal, (b) a pedestrian, and (c) an automobile. A time-series change of an image captured by the camera 1R is shown. As shown in the upper right of the figure, the square M additionally displayed on the captured image (in this figure, indicated by a white line on the captured image) is the extraction described above with reference to FIG. The circumscribed rectangle is set to circumscribe the target object. A line ML is drawn that halves each circumscribed square M in the y direction. Thus, the circumscribed square M is divided into an upper area Mu and a lower area Ml in the y direction.

  The numerical value in () given to M indicates that the data is past data as the value increases. For example, M (0) corresponds to the latest position coordinate Pv (0) of the object, and indicates a circumscribed rectangle specified for the object extracted from the captured image in the current sampling cycle. M (1) corresponds to the position coordinate Pv (1), and indicates a circumscribed rectangle specified for the object extracted from the captured image one sampling period before.

  The animal in (a) moves by moving the four legs. Therefore, the shape of the upper body part, that is, the body part imaged in the upper area Mu of the circumscribed square M hardly changes with time, but the shape of the lower body part, that is, the quadruped part imaged in the lower area Ml changes. Since it moves with four legs, the temporal change in the size of the circumscribed square M is small.

  The pedestrian in (b) moves with two legs. Accordingly, the shape of the lower body imaged in the lower region Ml of the circumscribed rectangle M changes with time, and the size of the circumscribed rectangle M also changes with it (the more the legs are spread, the more the x direction of the circumscribed rectangle M moves). Will be larger). Further, as the arm moves, the shape of the upper body imaged in the upper region Mu of the circumscribed rectangle M is also likely to change.

  Since the car in (c) is an artifact, the shape of the upper part of the car imaged in the upper area Mu of the circumscribed square M and the lower part of the car imaged in the lower area Ml of the circumscribed square M The shape of seldom changes with time.

  The present invention has been made based on the findings of (a) to (c) above, and for the object moving in the X direction in front of the vehicle, the temporal change in the shape of the lower body being imaged was examined, An object having a large temporal change is determined as a quadruped animal. In addition, by examining temporal changes in the size of the entire object being imaged and the shape of the upper body being imaged, a quadruped animal is distinguished from a pedestrian and a car.

Specifically, for the circumscribed rectangle of the object moving in the X direction in front of the vehicle,
1) Examine the time-series change of the aspect ratio ASPECT of the circumscribed rectangle M.
2) Examine time-series changes in the shape of the object imaged in the upper region Mu of the circumscribed rectangle M.
3) The time-series change of the object shape imaged in the lower area Ml of the circumscribed rectangle M is examined.

  In the case of a pedestrian, the size of the circumscribed rectangle M changes as the shape of the lower body changes. Therefore, according to the above 1), those having a large change in aspect ratio ASPECT are highly likely to be pedestrians and are determined not to be quadrupeds.

  In the case of a pedestrian, since the shape of the upper body is likely to change, the shape of the object imaged in the upper region Mu of the circumscribed rectangle M is likely to change. Therefore, according to the above 2), it is highly likely that an object having a large change in the shape of the object in the upper region Mu is a pedestrian and is not a quadruped.

  Moreover, since the shape of a movable artificial object such as an automobile does not change, the shape of the object in the lower region Ml of the circumscribed rectangle M hardly changes. Therefore, according to the above 3), a small change in the shape of the object in the lower region Ml is highly likely to be a movable artifact such as an automobile, and is determined not to be a quadruped.

  As a result of these three determinations, the size of the circumscribed square M and the shape imaged in the upper area Mu of the circumscribed square M hardly change, but the object imaged in the lower area Ml is extracted. It is determined that this represents a quadruped animal.

  The above 1) and 2) are determinations for distinguishing quadrupeds from pedestrians. In this embodiment, both the determinations 1) and 2) are performed in order to increase the determination accuracy. However, only one of the determinations may be made.

  FIG. 17 shows a flow of a quadruped animal determination process based on the above findings, which is performed in step S24 of FIG.

In step S41, an object moving in the X direction in front of the vehicle 10 is further extracted from the objects extracted in step S17 of FIG. In this embodiment, it is checked whether the equations (9) to (11) are established for the object extracted in step S17 in FIG. 3, and the established object is extracted as a moving object.

| Xv (N−1) −Xv (0) |> predetermined value THx (9)
Vs = (Zv (N−1) −Zv (0)) / ΔT (10a)
Zv (0) / Vs <T (10)
| Yv (0) | <H (11)

  The above equations (9) to (11) use the relative movement vector calculated as described above. Since the relative movement vector represents the movement of the object with respect to the vehicle 10, it can be checked from the relative movement vector whether or not the object has moved in the X direction within a predetermined time.

  As described above, the position coordinates of the object move from Pv (N−1) to Pv (0) within the predetermined monitoring period ΔT. Here, the X, Y, and Z coordinates of Pv are as shown in Expression (8). As shown in the equation (9), it is checked whether or not the movement distance | Xv (N−1) −Xv (0) | in the X direction is larger than the predetermined value THx within the period of ΔT. In this case, it is determined that the object is moving in the X direction. The predetermined value THx can be set arbitrarily.

  Preferably, it is possible to determine whether or not the formula (10) is satisfied, in order to investigate the possibility of collision between the vehicle 10 and the object. Specifically, in order to check whether or not the object is approaching the vehicle 10 at a predetermined speed or higher in the Z direction (distance direction), the relative speed Vs in the Z direction is calculated by the equation (10a). Zv (0) is the latest distance detection value of the object, and Zv (N−1) is the distance detection value before time ΔT. T is a predetermined margin time, and is intended to determine the possibility of a collision by a time T before the predicted collision time, and can be set to about 2 to 5 seconds, for example. That Formula (10) is materialized indicates that the object is approaching the vehicle 10. Note that the predetermined value Z1 in the Z direction described with reference to FIG. 15 corresponds to Vs × T.

  More preferably, it is determined whether or not the object indicates the size of the animal by determining whether or not Expression (11) is satisfied. In this embodiment, height is used. If the height of the object in the Y direction is smaller than a predetermined value H, it is determined that it corresponds to the size of the animal. H can be set to about twice the vehicle height of the vehicle 10, for example, and corresponds to the predetermined value H in the Y direction described with reference to FIG.

  In this way, an object that satisfies the above three expressions (9), (10), and (11) is extracted as an object that moves in front of the vehicle 10 in the X direction.

  In step S42, the object assigned the same label Lb1 as the label given to the moving object extracted in step S41 (referred to as Lb1) is searched from the time-series data stored in the memory. . As described above, in the process of step S19 in FIG. 3, the object is tracked for the time, the same label is given to the object determined to be the same, and information on the object is stored as time-series data. Is remembered. Therefore, by searching the time series data stored in the memory, the image data, the position (in this embodiment, the position of the center of gravity G) surrounded by the circumscribed rectangle for the moving object to which the label Lb1 is assigned, The aspect ratio ASPECT of the circumscribed rectangle can be obtained in time series.

  In this embodiment, the period to be searched in step S42 is ΔT, and therefore N positions Pv (0) to Pv (N−1) of the objects respectively obtained from the N captured images. ) N circumscribed squares M (0) to M (N−1) and image data (binarized) surrounded by the circumscribed squares exist.

  As described above, the search target period in step S42 in this embodiment is the same as the above-described relative movement vector monitoring period ΔT, but they may be different. For example, the number of captured images used in step S42 may be less than N.

In step S43, the magnitude of the change in the aspect ratio ASPECT of the circumscribed rectangle in the monitoring period ΔT is examined. As an example of this technique, the aspect ratio ASPECT (0) of the latest circumscribed rectangle M (0) and the aspect ratio of the circumscribed rectangle M (j) (j is 1 to (N-1)) searched from the memory. The difference Dap (j) of the ratio ASPECT (j) is calculated as in Expression (12).
Difference Dap (j)
= | ASPECT (j) of M (j) −ASPECT (0) of M (0) |
j = 1 to (N-1) (12)

An average value Dap_av of the difference is calculated by averaging (N−1) differences Dap (j) calculated according to Expression (12) as shown in Expression (13).
Dap_av
= (Dap (1) + Dap (2) +... Dap (N-1)) / (N-1)
(13)

  The average value Dap_av of the difference represents how much the aspect ratio ASPECT of the circumscribed rectangle changes within the period ΔT with respect to the reference aspect ratio ASPECT (0), and therefore the aspect ratio in the period ΔT. It can be considered to represent the magnitude of the change in ASPECT.

  Note that the magnitude of the change in aspect ratio ASPECT may be examined by other methods. For example, in the monitoring period ΔT, the transition of N aspect ratios of the aspect ratio ASPECT (0) of M (0) to the aspect ratio ASPECT (N−1) of M (N−1) is examined, and the maximum aspect ratio ASPECT is obtained. The absolute value of the difference between the value and the minimum value may be used to represent the magnitude of the change in aspect ratio ASPECT. As a further alternative, the absolute value of the difference between the maximum and minimum values of the difference Dap (j) may be used to represent the magnitude of the change in aspect ratio ASPECT.

  In step S44, the magnitude of the change in the object shape in the upper area (Mu in FIG. 16) of the circumscribed rectangle in the monitoring period ΔT is examined. In this embodiment, the shape of the object imaged in the upper area of the latest circumscribed rectangle M (0) and the upper area of the circumscribed rectangle M (j) (j = 1 to N−1) searched from the memory. The difference from the shape of the object being imaged is examined.

  This difference can be examined by any appropriate technique. For example, using a known shape matching technique, the object shape in the upper region of the circumscribed rectangle M (0) and the circumscribed rectangle M (j ) And an object shape in the upper region are taken, and a value E representing the height of the correlation is calculated. It shows that the difference between both shapes is so large that the value E showing the calculated correlation height is large.

  Specifically, the image of the upper region of the circumscribed rectangle M (0) is used as a reference image, and this is aligned with the image of the upper region of the circumscribed rectangle M (j) (referred to as a target image). Both images are binary images. For example, the center of the circumscribed square M (0) (the intersection of the diagonal lines of the circumscribed square) and the center of the circumscribed square M (j) can be aligned. Alternatively, the position of the center of gravity G (0) representing the position of the object calculated as described above for the circumscribed rectangle M (0) and the position of the object similarly calculated for the circumscribed rectangle M (j) are represented. The position of the center of gravity G (j) may be matched.

The number of pixels of the reference image of the circumscribed rectangle M (0) is P × Q, and the coordinates of each pixel are represented by (p, q). As shown in Expression (14), in the aligned state, the luminance value I ₀ (p, q) of each pixel of the reference image and the luminance value I _j of the corresponding pixel of the target image of the circumscribed rectangle M (j) The sum Eu (j) of absolute values of differences from (p, q) is calculated. For a certain pixel in the reference image, when the corresponding pixel does not exist in the target image (for example, when the number of pixels in the reference image is larger than the number of pixels in the target image), a black value (1) for I _j Can be set. The sum Eu (j) represents the degree of correlation between the two, and is referred to herein as the sum value.

  The total value Eu (j) is the shape of the object imaged in the upper area of the latest circumscribed rectangle M (0) and the shape of the object imaged in the upper area of the past circumscribed rectangle M (j). Represents the difference. It shows that the difference between both is so large that total value Eu is high.

The average value Eu_av of the total value is calculated by averaging the (N−1) total values Eu (j) according to the equation (14) as in the equation (15).
Eu_av = (Eu (1) + Eu (2) +... Eu (N-1)) / (N-1)
(15)

  The average value Eu_av of the total sum represents how much the shape of the object in the upper region of the circumscribed rectangle changes within the period ΔT with respect to the shape of the object in the upper region of the reference circumscribed rectangle M (0). Therefore, it can be considered to represent the magnitude of the change in the shape of the object in the upper region within the period of ΔT. The larger the average value Eu_av, the larger the magnitude of the change.

  Note that the size of the change in the shape of the object in the upper region may be examined by another method. For example, the transition of N total values from the total value Eu (0) to Eu (N−1) in the period of ΔT is examined, and the absolute value of the difference between the maximum value and the minimum value of the total value is calculated in the upper region. May be used to represent the magnitude of the change in the shape of the object.

  In taking the correlation, alternatively, from the aligned state, the reference image is scanned within a predetermined range with respect to the target image, and the total value Eu (j) is calculated at each scanning position. A total value may be used. Even if there is a difference in size between the reference image and the target image, the correlation between the two can be obtained with better accuracy.

  In step S45, the magnitude of the change in the object shape in the lower area of the circumscribed rectangle (Ml in FIG. 16) in the monitoring period ΔT is examined. In this embodiment, the shape of the object imaged in the lower area of the latest circumscribed rectangle M (0) and the shape of the object imaged in the lower area of the circumscribed rectangle M (j) searched from the memory. Investigate the difference. This is calculated by the same method as that calculated for the upper area of the circumscribed rectangle in step S44, and thus the description thereof is omitted. As a result, a total value El (j) representing the correlation for the lower region is calculated, and an average value El_av representing the magnitude of the change in the shape of the object in the lower region of the circumscribed rectangle within the period of ΔT is calculated. Calculated. The larger the average value El_av, the larger the magnitude of the change.

  Similarly to the description about step S44, the magnitude of the change in the shape of the object in the lower region may be examined by another method. For example, the transition of N total values from the total value El (0) to El (N-1) is examined, and the absolute value of the difference between the maximum value and the minimum value of the total value is determined as the object in the lower region. You may use as a thing showing the magnitude | size of a shape change.

  Note that the shape matching in steps S43 and S44 may alternatively be performed using a grayscale image.

  Next, in step S46, it is determined whether or not the magnitude of the change in aspect ratio ASPECT calculated in step S43 is small. In this embodiment, it is determined whether the average difference value Dap_av is equal to or less than a predetermined value. This step is a process for determining 1) described above. Therefore, the predetermined value is set in advance to a value that can distinguish a pedestrian and an animal through simulation or the like. If the average difference value Dap_av is equal to or less than the predetermined value, it indicates that the change in the aspect ratio ASPECT during the monitoring period ΔT is small, and the process proceeds to step S47. If the average difference value Dap_av is larger than a predetermined value, it indicates that the change in aspect ratio ASPECT is large. In this case, it is highly likely that the moving object represents a pedestrian and is not a quadruped (S49).

  In step S47, it is determined whether or not the magnitude of the change in the object shape in the upper area of the circumscribed rectangle calculated in step S44 is small. In this embodiment, it is determined whether the average value Eu_av of the total value is equal to or less than a predetermined value. This step is a process for determining 2) described above. Therefore, the predetermined value is set in advance to a value that can distinguish a pedestrian and an animal through simulation or the like. If the average value Eu_av of the total value in the monitoring period ΔT is equal to or less than the predetermined value, it indicates that the change in the shape of the object in the upper region is small, and the process proceeds to step S48. If the average value Eu_av of the total sum is larger than a predetermined value, it indicates that the change in the shape of the object in the upper region is large. In this case, it is highly likely that the moving object represents a pedestrian and is not a quadruped (S49).

  In step S48, it is determined whether the magnitude of the change in object shape in the lower area of the circumscribed rectangle calculated in step S45 is large. In this embodiment, it is determined whether the average value El_av of the total value is equal to or greater than a predetermined value. This step is a process for determining 3) described above. Therefore, the predetermined value is set in advance to a value that can distinguish an artificial movable object such as an automobile from an animal through simulation or the like. If the average value El_av of the total value in the monitoring period ΔT is equal to or greater than the predetermined value, it indicates that the change in the shape of the object in the lower region is large, and the process proceeds to step S50. If the average value El_av of the total value is smaller than a predetermined value, it indicates that the change in the shape of the object in the lower region is small. In this case, the moving object is highly likely to represent an artificial movable object such as an automobile, and is determined not to be a quadruped animal (S49).

  Thus, when all the determinations in steps S46 to S48 are Yes, that is, in the monitoring period ΔT, the change in the aspect ratio of the circumscribed rectangle is small, and the shape of the object imaged in the upper area of the circumscribed rectangle When the magnitude of the change is small and the magnitude of the change in the shape of the object imaged in the lower area of the circumscribed rectangle is large, it is determined that the object is a quadruped (S50). If it is determined that the animal is a quadruped animal, the object is determined to be an alert target.

  On the other hand, if it is determined that the object is not a quadruped animal, it may be determined by another method whether or not the object is a pedestrian.

  As described above, if it is determined that the animal is a quadruped animal, it is determined whether or not an alarm is actually output in the alarm determination process in step S25 of FIG. This determination process will be described.

  In this embodiment, it is determined whether an alarm output is actually performed according to the operation of the brake. Specifically, it is determined from the output of the brake sensor 7 whether or not the driver of the vehicle 10 is performing a brake operation. If the brake operation is not performed, an alarm is output. When the brake operation is performed, the acceleration Gs generated by the braking operation is calculated (the deceleration direction is positive). When the acceleration Gs is equal to or less than a predetermined threshold GTH (Gs ≦ GTH), an alarm is output. When Gs> GTH, it is determined that the collision is avoided by the brake operation, and the process is terminated. As a result, when an appropriate brake operation is performed, an alarm is not issued, so that the driver is not bothered excessively. However, alternatively, an alarm may be issued without determining brake operation.

Here, the threshold value GTH can be determined as in Expression (16). This is a value corresponding to a condition in which the vehicle 10 stops at a travel distance equal to or less than the distance Zv (0) when the brake acceleration Gs is maintained as it is.

  As for the alarm output, an audio alarm is issued through the speaker 3 and, as shown in FIG. 18, an image obtained by the camera 1R, for example, is displayed on the screen 4a by the HUD 4, and the moving quadruped is highlighted. To do. The highlighting may be performed by any method, for example, it can be highlighted by surrounding it with a colored frame. In this way, the driver can more reliably recognize a moving animal with a high possibility of collision. The alarm output may be performed using either one of the alarm and the image display.

  Note that the warning determination process can be performed not only on a moving quadruped animal but also on an object that is determined to be a target for alerting, such as a pedestrian, as described above.

  In the above-described embodiment, the circumscribed rectangle M (0) of the captured image corresponding to the position Pv (0) of the moving object at the time when the moving object is determined in step S41 is used as a reference, and the past captured image is more than this. The quadruped animal determination process is performed on the circumscribed rectangle M (j). Alternatively, a circumscribed rectangle of a captured image at a time different from the time when it is determined as a moving object may be used as a reference. For example, with respect to the determined moving object, the circumscribed rectangle M (5) of the captured image corresponding to the position Pv (5) is used as a reference, and the captured images at future positions Pv (0) to Pv (4). In addition, the aspect ratio ASPECT, time-series changes of the upper shape and the lower shape may be examined for the captured images of the past positions Pv (6) to Pv (N-1).

  Further, in the above embodiment, in order to determine a quadruped animal with better accuracy, the number of images to be compared with the reference image is plural (N), but may alternatively be one. In that case, the aspect ratio ASPECT, the upper shape, and the lower shape are compared between the circumscribed square of the current captured image and the circumscribed square of the captured image for a predetermined time, for example, and the magnitude of the change between the two is determined. Can be sought.

  Further, in the above embodiment, it is determined whether or not the object is moving in the X direction using the relative movement vector. Alternatively, more simply, it is checked whether the absolute value of the difference between the X coordinate of P (0) and the X coordinate of P (N-1) is greater than a predetermined value THx. You may make it judge that it is a thing.

  Further, in the above-described embodiment, it is determined whether the object is moving in the X direction by the same method regardless of where the object exists in the areas AR1 to AR3 (FIG. 15). Alternatively, the determination method may be changed for each region. For example, the latest position Pv (0) of the object is in the approach determination region AR1 and X as described in step S41 above. If it is determined that it is moving in the direction, the determination process of FIG. 17 may be performed. On the other hand, when the latest position Pv (0) of the object is in the intrusion determination area AR2, it is checked whether or not the moving direction is the positive to negative direction of the X axis (that is, Xv (N-1)). > Xv (0)), if so, the determination process of FIG. 17 is performed. That is, when the object is present in the area AR2, the movement from the negative to the positive direction indicates that the object is moving away from the vehicle 10, and in this case, the quadruped animal determination process is performed. Do not do it. On the other hand, when the latest position Pv (0) of the object is in the intrusion determination area AR3, it is checked whether the moving direction is from the negative to the positive direction of the X axis (that is, Xv (N−1) ) <Xv (0)), if so, the determination process of FIG. 17 is performed. That is, when the object is present in the area AR3, the movement from the positive direction to the negative direction indicates that the object is moving away from the vehicle 10, and in this case, the quadruped animal determination process is performed. Do not do it. In this way, the determination method may be changed for each region.

  In addition, this invention is not restricted to said embodiment, Various deformation | transformation forms are possible. For example, in the above-described embodiment, an infrared camera is used as the imaging unit. However, for example, a television camera that can detect only normal visible light may be used (for example, JP-A-2-26490). However, by using an infrared camera, the object extraction process can be further simplified, and the calculation load can be reduced.

  As described above, specific embodiments of the present invention have been described. However, the present invention is not limited to these embodiments.

The block diagram which shows the structure of the periphery monitoring apparatus according to one Example of this invention. The figure for demonstrating the attachment position of the camera according to one Example of this invention. 3 is a flowchart illustrating a process in an image processing unit according to an embodiment of the present invention. The figure which shows an example of the gray scale image obtained by the camera according to one Example of this invention. The figure which shows an example of the image which binarized the gray scale image according to one Example of this invention. The figure for demonstrating the conversion process and labeling process to run length data according to one Example of this invention. The figure for demonstrating the time tracking of the target object according to one Example of this invention. The figure which shows the search area | region in the right image according to one Example of this invention, and the search area | region set to a left image. The figure for demonstrating the correlation calculation for the search area | region according to one Example of this invention. The figure for demonstrating the calculation method of the parallax according to one Example of this invention. The figure for demonstrating the method of calculating distance from parallax according to one Example of this invention. The figure which shows the real space coordinate system and image coordinate system according to one Example of this invention. The figure for demonstrating the turning angle correction | amendment according to one Example of this invention. The figure for demonstrating the calculation method of a relative movement vector according to one Example of this invention. The figure which shows the area division ahead of the vehicle according to one Example of this invention. The figure for demonstrating the principle of the quadruped animal moving to the horizontal direction according to one Example of this invention. 4 is a flowchart of a quadruped animal determination process according to one embodiment of the present invention. The figure for demonstrating an example of the alarm output according to one Example of this invention.

Explanation of symbols

1R, 1L infrared camera (imaging means)
2 Image processing unit 3 Speaker 4 Head-up display

Claims (2)

A vehicle periphery monitoring device that monitors the periphery of a vehicle using a captured image obtained by an imaging means mounted on the vehicle,
Means for extracting an object from the captured image and setting a circumscribed rectangle circumscribing the object;
Storage means for storing, in time series, image data of an object on which the circumscribed rectangle is set;
A moving object specifying means for tracking the position of the extracted object and specifying a moving object moving in the vehicle width direction in front of the vehicle;
Time-series data for extracting the image data in which the circumscribed rectangle at one time is set and the image data in which the circumscribed rectangle at another time is set from the time-series image data for the identified moving object. Extraction means;
The magnitude of the change between the aspect ratio of the circumscribed rectangle at the one time and the aspect ratio of the circumscribed rectangle at the other time is small, and an image is captured in the lower area of the circumscribed rectangle at the one time When the magnitude of the change between the shape of the moving object being made and the shape of the moving object imaged in the lower area of the circumscribed rectangle at the other time is large, the moving object is A determination means for determining that the animal is a quadruped animal;
A vehicle periphery monitoring device comprising: A vehicle periphery monitoring device that monitors the periphery of a vehicle using a captured image obtained by an imaging means mounted on the vehicle,
Means for extracting an object from the captured image and setting a circumscribed rectangle circumscribing the object;
Storage means for storing, in time series, image data of an object on which the circumscribed rectangle is set;
A moving object specifying means for tracking the position of the extracted object and specifying a moving object moving in the vehicle width direction in front of the vehicle;
Time-series data for extracting the image data in which the circumscribed rectangle at one time is set and the image data in which the circumscribed rectangle at another time is set from the time-series image data for the identified moving object. Extraction means;
Between the shape of the moving object imaged in the upper area of the circumscribed rectangle at the one time and the shape of the moving object imaged in the upper area of the circumscribed rectangle at the other time The magnitude of the change is small, and the shape of the moving object imaged in the lower area of the circumscribed square at the one time and the image of the lower area of the circumscribed square at the other time A determining means for determining that the moving object is a quadruped when the magnitude of the change between the moving object and the shape of the moving object is large;
A vehicle periphery monitoring device comprising: