JP5904925B2 - Vehicle periphery monitoring device - Google Patents

Description

  The present invention relates to a vehicle periphery monitoring device that monitors the periphery of a vehicle based on a captured image of a camera attached to the vehicle.

  2. Description of the Related Art Conventionally, a vehicle periphery monitoring device that extracts an image portion of an object (monitoring object such as a pedestrian) from an image captured by a camera attached to the vehicle and calculates the position of the object in real space. It has been proposed (see, for example, Patent Document 1).

  In the vehicle periphery monitoring device described in Patent Document 1, the size of the image portion of the target object extracted from the image captured by one camera is set on a real space set by assuming the type of the target object. The distance between the object and the vehicle is calculated by applying to the correlation between the distance between the object and the vehicle and the size of the image portion of the object on the image.

International Publication No. 2012/029372

  In the vehicle periphery monitoring device described in Patent Document 1 described above, the image portion of the target object corresponds to the actual size of the target object (the size of the target object in real space). This is a precondition for accurately calculating the distance between the vehicle and the object. However, depending on the imaging state of the camera, for example, the image portion of the leg portion of the object is not extracted from the captured image, and the lower end of the image portion of the object is the original position (when the image portion of the leg portion is extracted). Position).

  In this case, since the size (height) of the image portion of the object is different from the original size (a size corresponding to the size of the actual object), the image portion of the object The distance between the object and the vehicle cannot be accurately calculated based on the size of the vehicle.

  The present invention has been made in view of such a background, and the position on the captured image corresponding to the ground contact position of the target object in the real space even in an imaging state in which the image portion at the lower end of the target object cannot be extracted from the captured image. It is an object of the present invention to provide a vehicle periphery monitoring device that can estimate the vehicle speed.

The present invention has been made to achieve the above object, and the vehicle periphery monitoring device of the present invention comprises:
A captured image acquisition unit for acquiring an image captured by a camera mounted on the vehicle;
An object extraction unit for extracting an image portion of the object from the captured image;
A road boundary extraction unit that extracts an image part of a boundary between a roadway and a roadside belt or a sidewalk from the captured image;
When the image portion of the object is located above the boundary image portion, based on the intersection of the straight line drawn vertically downward from the image portion of the object and the boundary image portion, in real space And a lower end position estimation unit that estimates a position on the captured image corresponding to the ground contact position of the object (first invention).

  According to the first invention, in the image portion of the target object extracted by the target portion extraction section, the lower end image portion (for example, the leg image portion when the target object is a pedestrian) is not extracted. Even in such a case, based on the intersection of the straight line drawn vertically downward from the image portion of the object and the image portion at the boundary between the roadway and the roadside belt or sidewalk by the lower end position estimation unit, the real space The position on the captured image corresponding to the ground contact position of the target object can be estimated.

In the first invention,
The road boundary extraction unit draws the straight line vertically downward from a horizontal center position of the image portion of the object when the image portion of the object is a pedestrian image portion. (Second invention).

According to the second invention, when the image portion of the object is a pedestrian image portion, the object in the real space is drawn by drawing the straight line vertically downward from the horizontal center position of the image portion of the object. The estimation accuracy of the position on the captured image corresponding to the contact position of the object can be increased.

In the first invention or the second invention,
The road boundary extracting unit extracts an image portion of the boundary using an edge filter for oblique line extraction (third invention).

  According to the third aspect, by using the edge filter, it is possible to easily extract the image portion of the boundary that is assumed to be a diagonal line in the captured image.

In the third invention,
The road boundary extraction unit selects an edge line whose length is equal to or greater than a first predetermined value and whose slope or curvature is closest to a second predetermined value from among the edge lines extracted using the edge filter. (4th invention).

  According to the fourth invention, it is possible to select an edge line that is highly likely to be the image portion of the boundary from among the edge lines extracted using the edge filter.

In the fourth invention,
A road shape recognition unit for recognizing the shape of the road on which the vehicle is traveling;
A vehicle behavior detection unit for detecting the behavior of the vehicle,
The road boundary extraction unit determines the second predetermined value based on one or both of a shape of a road on which the vehicle is traveling and a behavior of the vehicle (fifth invention). ).

  According to the fifth aspect, the shape of the image portion of the road, the roadside belt, and the sidewalk in the captured image changes according to the shape of the road on which the vehicle is traveling and the behavior of the vehicle. Therefore, the road boundary extraction unit determines the second predetermined value based on one or both of the shape of the road on which the vehicle is traveling and the behavior of the vehicle, thereby obtaining an image of the boundary. The extraction accuracy of the part can be increased.

In any one of the first to fifth inventions,
The road boundary extraction unit is configured to extract the image portion of the boundary from a boundary search region set in advance as a region where the image portion of the boundary is assumed to be located in the captured image. Characteristic (sixth invention).

  According to the sixth aspect of the present invention, it is possible to reduce the amount of calculation required for extracting the boundary image portion by narrowing down the search range when extracting the boundary image portion to the boundary search region.

In any one of the first to sixth inventions,
The lower end position estimation unit is configured to be perpendicular to the image portion of the object when the position of the vanishing point of the road in the captured image is deviated by a predetermined distance or more from a reference position set assuming a straight flat road. The position on the captured image corresponding to the ground contact position of the object in real space, the position obtained by correcting the position of the intersection of the straight line drawn downward and the image portion of the boundary according to the direction of the shift (7th invention).

  In the seventh invention, the position of the vanishing point of the road in the captured image is set assuming a straight flat road according to the shape of the road (the uphill road, the curved road, etc.) on which the vehicle is traveling. Deviation from the reference position. The direction in which the vanishing point is shifted corresponds to the shape of the road. Further, a direction where an intersection of a straight line drawn vertically downward from the image portion of the object and the image portion of the boundary deviates from an actual position on the captured image corresponding to a ground contact position in the real space of the object Corresponds to the shape of the road.

  Therefore, by correcting the position of the intersection of the straight line drawn vertically downward from the image part of the object and the image part of the boundary according to the direction of the vanishing point shift by the lower end position estimation unit, The estimation accuracy of the position on the captured image corresponding to the ground contact position in the real space of the object can be increased.

Explanatory drawing of the attachment aspect to the vehicle of a vehicle periphery monitoring apparatus. The block diagram of a vehicle periphery monitoring apparatus. The operation | movement flowchart of a vehicle periphery monitoring apparatus. Explanatory drawing of pedestrian determination. Explanatory drawing of the movement of the pedestrian crossing a road. Explanatory drawing of the correlation with the height and distance of the image part of a pedestrian. Explanatory drawing of the captured image under intense heat environment. The flowchart of the process which calculates the height of the image part of a target object. Explanatory drawing of the extraction process of a roadside edge. Explanatory drawing of the captured image when the vehicle is drive | working the straight flat road. Explanatory drawing of the captured image when the vehicle turns right and is traveling downhill. Explanatory drawing of the change of the magnitude | size of the image part of the target object in a time-sequential image. Explanatory drawing of the estimation process of the movement vector of the target object in real space. Explanatory drawing of the calculation process of the change rate of the image part of the object by pattern matching.

  Embodiments of the present invention will be described with reference to FIGS. Referring to FIG. 1, a vehicle periphery monitoring device 10 of this embodiment is used by being mounted on a vehicle 1, and the vehicle 1 includes an infrared camera 2 (corresponding to the camera of the present invention) capable of detecting far infrared rays. Is provided. The infrared camera 2 has sensitivity in the far-infrared region, and has a characteristic that the level of the output video signal increases (the brightness of the video signal increases) as the temperature of the object to be imaged increases.

  The infrared camera 2 is attached to the front portion of the vehicle 1 in order to image the front of the vehicle 1. The front portion of the vehicle 1 is the origin O, the left-right direction of the vehicle 1 is the X axis, the vertical direction is the Y axis, A real space coordinate system in which the front-rear direction is the Z axis is defined. Instead of the infrared camera 2, a camera having sensitivity in other wavelength regions such as visible light may be used.

  Next, referring to FIG. 2, the vehicle periphery monitoring device 10 includes a yaw rate sensor 3 (corresponding to a vehicle behavior detection unit of the present invention) that detects the yaw rate of the vehicle 1, and a vehicle speed that detects the traveling speed of the vehicle 1. There is a possibility that the sensor 4, the brake sensor 5 that detects the amount of brake operation by the driver, the speaker 6 for alerting by voice, the image captured by the infrared camera 2, and the vehicle 1 may be contacted. Head up display 7 (hereinafter referred to as HUD 7) for displaying a high object to the driver and the position of the vehicle 1 having map data and detected by GPS (Global Positioning System) The navigation device 8 (including the function of the road shape recognition unit of the present invention) that provides travel route information to the destination is connected with reference to the map data. . As shown in FIG. 1, the HUD 7 is provided so that the screen 7 a is displayed at a front position on the driver side of the front window of the vehicle 1.

  The vehicle periphery monitoring device 10 is an electronic unit configured by a CPU, a memory (not shown), and the like. The captured image acquisition unit 11 converts a video signal output from the infrared camera 2 into digital data and converts it into an image memory ( The CPU has a function of performing various arithmetic processes on the captured image in front of the vehicle 1 captured in the image memory and captured in the image memory.

  Then, by causing the CPU to execute a control program for the vehicle periphery monitoring device 10, the CPU extracts the object extraction unit 12 that extracts an image portion having a predetermined condition from the image captured by the infrared camera 2, and the extracted image An object type determination unit 13 that determines the type of an object in the real space corresponding to the part, a road boundary extraction unit 14 that extracts an image part of the boundary between the roadway and the sidewalk or the roadside belt, and a lower end position of the object in the real space The lower end position estimation unit 15 that estimates the position on the captured image corresponding to the distance, the distance calculation unit 16 that calculates the distance between the object and the vehicle 1, the real space position calculation unit 17 that calculates the real space position of the object, It functions as a movement vector calculation unit 18 that calculates the movement vector of the object, and an alarm determination unit 19 that determines whether or not the object is to be alarmed based on the movement vector.

  Next, a series of vehicle periphery monitoring processing by the vehicle periphery monitoring device 10 will be described according to the flowchart shown in FIG. The vehicle periphery monitoring device 10 monitors the periphery of the vehicle 1 by executing the process according to the flowchart shown in FIG.

[Image capture processing]
STEP 1 is a process performed by the captured image acquisition unit 11. The captured image acquisition unit 11 inputs a video signal output from the infrared camera 2 and takes in a grayscale image obtained by converting the video signal into digital gradation (luminance) data into an image memory.

[Object extraction process]
The subsequent STEP 2 is processing by the object extraction unit 12. The object extraction unit 12 sets, for each pixel of the grayscale image, a pixel whose luminance is equal to or higher than a predetermined threshold value as “1” (white) and a pixel whose luminance is smaller than the threshold value as “0” (black) 2 A binarization process is performed to obtain a binary image. Then, the target object extraction unit 12 calculates run length data of each white region in the binary image, and performs a labeling process or the like to extract an image portion of the object.

[Object type determination processing]
The next STEP 3 is processing by the object type determination unit 13. The object type determination unit 13 is a pedestrian whose object crosses the road (hereinafter referred to as a crossing pedestrian) based on the feature amount of the image portion of the object (hereinafter referred to as the object) extracted in STEP2. Judge whether there is.

  Here, FIG. 4 exemplifies an image of a crossing pedestrian, and the object type determination unit 13 includes regions M (1) and M (2) above the image portion Tk estimated to be the head of the crossing pedestrian. ) And lower regions M (3) to M (8) are searched to recognize the features of each region.

  Then, the object type determination unit 13 does not have a feature in the upper regions M (1) and M (2) of the image portion Tk, and crosses pedestrians in the lower regions M (3) to M (8). When the square shape at the lower end (two oblique edges intersecting above) is recognized, the type of the object is determined to be a crossing pedestrian.

  In addition, the object type determination unit 13 sets a target object region Rk that includes the image portion Tk that is estimated as the head to M (7) at the lower end. Note that the determination of a crossing pedestrian may be based on the complexity of the image, changes in luminance dispersion, periodicity, or the like.

[Distance calculation processing]
The following STEP 4 to STEP 5 are processes by the distance calculation unit 16, and STEP 20 is a process by the road boundary extraction unit 14, the lower end position estimation unit 15, and the distance calculation unit 16. The distance calculation unit 16 determines whether or not it is determined in STEP 4 that the type of the object is a crossing pedestrian, and when it is determined that the object is a crossing pedestrian (in this case, the object and its image portion). (It is assumed that the shape change is large and the degree of shape change exceeds a predetermined level), the process branches to STEP 20, and the first distance calculation process is executed.

  On the other hand, when it is determined that the type of the object is not a crossing pedestrian (in this case, it is assumed that the shape change of the object and its image portion is small and the degree of shape change is below the predetermined level). Advances to STEP5, and the distance calculation unit 16 executes a second distance calculation process.

  Here, as will be described later, the second distance calculation process is performed based on the rate of change in the size of the image portion of the same object extracted from the time-series image captured by the infrared camera 2. 1 is calculated.

  Then, as shown in FIG. 5, the crossing pedestrian crosses the front of the vehicle 1 by moving both hands and both feet greatly as shown by Ik1 to Ik4 in FIG. The shape (width W etc.) of the image part of a pedestrian changes greatly. Therefore, there is a case where the image part of the crossing pedestrian cannot be extracted as the image part of the same object between the time-series images taken by the infrared camera 2, and the change rate of the size can be obtained even if it can be extracted. It is difficult to obtain with high accuracy.

  Therefore, when it is determined that the type of the object is a crossing pedestrian, the distance calculation unit 16 determines the object and the vehicle based on the height of the image portion of the object in the single captured image in STEP 20. 1st distance calculation processing which calculates the distance with 1 is performed.

[First distance calculation processing]
As shown in FIG. 6, when it is assumed that the height of the pedestrian is constant (for example, 170 cm), as the distance between the pedestrian and the vehicle 1 becomes longer, the pedestrian in the captured image Im1 of the infrared camera 2 becomes larger. The height H of the image parts 20-22 becomes low. Therefore, the distance calculation unit 16 assumes that the height of the pedestrian in the real space is constant, and the distance L in the real space between the pedestrian and the vehicle 1 and the image portion of the pedestrian on the captured image. The height in the real space between the object and the vehicle 1 is calculated by applying the height of the image portion of the object on the captured image to the map or correlation equation in which the correlation with the height H is set. .

  Here, as shown in FIG. 7 (a), the temperature difference between the pedestrian and the road becomes small under the extreme heat environment in which the outside air temperature rises to near the body temperature of the pedestrian. The brightness difference between the pedestrian image portion 30a, the road image portion 31 and the wall image portion 32 in the captured image Im2 (grayscale image) is reduced.

  Therefore, when the captured image Im2 is binarized, as shown in FIG. 7B, the pedestrian's image portion 30b in the binary image Im3 is partially lost (the pixel is “0”). May become). In this case, as shown in FIG. 7A, the image portion of the pedestrian whose height is H1 is extracted as the height H2 as shown in FIG. 7B. , The error when the distance between the pedestrian and the vehicle 1 is calculated based on H2 increases.

  Also, even when a visible camera is used instead of an infrared camera, when the luminance difference between the pedestrian image part and the road image part is small, a part of the pedestrian leg part disappears in the same manner. There is.

  Therefore, in order to suppress the occurrence of such a distance calculation error, the road boundary extraction unit 14 and the lower end position estimation unit 15 perform processing according to the flowchart shown in FIG. The lower end position (the position on the captured image corresponding to the ground contact position of the pedestrian's leg in real space) is estimated.

  Steps 50 to 53 in FIG. 8 are processes by the road boundary extraction unit 14. In STEP 50, the road boundary extraction unit 14 in the captured image Im4 (grayscale image or binary image) by the infrared camera 2 as shown in FIG. 9A, boundaries 40, 41 between the roadway and the roadside belt or sidewalk. Regions 43 and 44 that are assumed to be located are set as boundary search regions.

  In subsequent STEP 51, the road boundary extraction unit 14 applies an edge filter for detecting a diagonal line for extracting a diagonal line rising to the right for the boundary search region 43 on the left side, and thereby the edge line of the boundary 40 (hereinafter referred to as a left road side edge). To extract. Further, the road boundary extraction unit 14 applies an edge filter for oblique line detection that extracts a downward slanting oblique line to the right boundary search area 44 to extract an edge line of the boundary 41 (hereinafter referred to as a right road side edge). To do.

  In the next STEP 52, the road boundary extraction unit 14 recognizes the shape of the road on which the vehicle 1 is traveling from the information (position information and map information of the vehicle 1) from the navigation device 8, and according to the shape of the road. The determination reference value of the slope of the left road side edge and the right road side edge (the reference value of the slope of the left road side edge and the right road side edge assumed from the shape of the road) is set.

  For example, since FIG. 9A shows a situation where the vehicle 1 is traveling on a straight flat road, the road boundary extraction unit 14 captures the left road-side edge and the image when the vehicle 1 travels on a straight flat road. An inclination assuming a right road side edge is set as a criterion. FIG. 9B shows a situation where the vehicle 1 is turning right on a curved road, so the road boundary extraction unit 14 shows the left road edge and the right road edge when imaged in a situation where the curved road is turned right. The assumed inclination is set as the determination reference value. For curved roads, the curvature may be used as a criterion value instead of the slope.

  Note that the determination reference value is set by estimating the shape of the image portion of the road in the captured image according to the behavior of the vehicle 1 detected by the yaw rate sensor 3 or the like instead of the shape of the road recognized by the navigation device 8. May be. Alternatively, the determination reference value may be set based on both the shape of the road recognized by the navigation device 8 and the behavior of the vehicle 1 detected by the yaw rate sensor 3 or the like.

  In the next STEP 53, the road boundary extracting unit 14 uses the extracted left road side edge and right road side edge whose length is equal to or greater than a predetermined value and whose slope is closest to the determination reference value. Sort as edge and right road edge.

  The subsequent STEP 54 is processing by the lower end position estimation unit 15. As shown in FIG. 10, the lower-end position estimation unit 15 is configured such that the pedestrian image portion 63 extracted by the object extraction unit 12 in the binary image Im6 is extracted by the road boundary extraction unit 14. A straight line 65 and a left road-side edge drawn vertically downward (downward in the y-coordinate) from the center position 64 in the horizontal direction (x-coordinate direction) of the image portion 63 of the pedestrian when positioned above the 61. An intersection 66 with 61 is obtained.

  In addition, when calculating | requiring an intersection about the left road side edge and the image part of the pedestrian of the upper side, you may make it draw a straight line from the right end of the horizontal direction of a pedestrian's image part to a perpendicular downward direction. Further, when obtaining the intersection point for the right road side edge and the image portion of the pedestrian above it, a straight line may be drawn vertically downward from the horizontal left end of the image portion of the pedestrian.

  Then, the lower end position estimation unit 15 estimates the intersection point 66 as the original lower end position of the pedestrian image portion 63 (a position on the captured image corresponding to the pedestrian's ground contact position in real space, hereinafter referred to as the lower end estimated position). To do. In FIG. 10, 62 is a right road edge, and 60 is a vanishing point of the road.

  Thus, by obtaining the estimated lower end position of the pedestrian image part 63, as described above with reference to FIG. 7A and FIG. Even in the case of disappearance, the position on the captured image corresponding to the ground contact position of the pedestrian in the real space can be estimated.

  Note that FIG. 10 is a captured image in which the left road side edge 61 and the right road side edge 62 are extracted in a situation where the vehicle 1 is traveling on a straight flat road, but as illustrated in FIG. In the captured image when the vehicle 1 is turning right, the positional relationship between the pedestrian image portion 74 and the right road side edge 72 and vanishing point 73 is different.

  In the captured image Im7 in FIG. 11A, the intersection 76 between the straight line 75 and the right road side edge 72 drawn vertically downward from the pedestrian image portion 74 is shifted from the lower end position of the original image portion 74. Will become bigger.

  For this reason, the lower end position estimating unit 15 determines a predetermined position from the reference position 70 that is set assuming the position of the vanishing point in the captured image when the vanishing point 73 is traveling on a straight flat road in the captured image Im7. When the displacement is more than the distance, the displacement direction of the vanishing point 73 with respect to the reference position 70 is determined. When the displacement direction is the right direction, the point 77 obtained by moving the intersection 76 upward along the right road edge 72 is the lower end. Estimated position. And it suppresses that the estimation precision of a lower end estimated position falls by this.

  FIG. 11B shows a captured image in which the left road side edge 71 and the right road side edge 72 are extracted in a situation where the vehicle 1 is traveling on a downhill. An intersection 86 between the straight line 85 drawn vertically downward from the portion 84 and the left road side edge 81 is greatly displaced from the lower end position of the original image portion 84. In FIG. 11B, the vanishing point 83 is shifted upward from the reference position 80.

  For this reason, the lower end position estimating unit 15 sets a point 87 obtained by moving the intersection 86 upward along the left road side edge 81 in the captured image Im8 in which the vanishing point 83 is shifted upward with respect to the reference position 80. And And it suppresses that the estimation precision of a lower end estimated position falls by this.

  In FIG. 11A, the relationship between the right road side edge 72 and the pedestrian image portion 74 when the vanishing point 73 is shifted to the right side with respect to the reference position 70 has been described. Even when the point is shifted to the left side, it is possible to suppress the estimation accuracy of the lower end estimated position from being lowered by performing the same process on the left road side edge and the image portion of the pedestrian. Further, even if correction based on such vanishing point shift is not performed, the effect of the present invention can be obtained.

  Then, in STEP 55, the distance calculation unit 16 calculates the height from the upper end to the lower end estimated position of the pedestrian image part as the height of the pedestrian image part. This is applied to a map or correlation equation in which the correlation between the distance L in the real space with the vehicle 1 and the height H of the image portion of the pedestrian on the captured image is set. Calculate the distance in space.

[Second distance calculation processing]
On the other hand, when it is determined that the type of the object is not a crossing pedestrian, the distance calculation unit 16 calculates the distance between the object and the vehicle 1 by executing a second distance calculation process in STEP 5. In the second distance calculation process, as shown in FIG. 12, the image Im9 captured in the previous control cycle (imaging time t ₁₁ ) and the image Im10 captured in the current control cycle (imaging time t ₁₂ ) The processing for tracking the image portion of the same object is performed. Note that the tracking process is described in detail in, for example, Japanese Patent Application Laid-Open No. 2007-213561, and thus the description thereof is omitted here.

Then, the distance calculation unit 16, by the following equation (1), by dividing the width w ₁₁ of the image portion 91 in the image Im9 width w ₁₂ of the image portion 92 in the image IM10, to calculate the change rate Rate. The relative speed Vs between the vehicle 1 and the object is approximated by the traveling speed of the vehicle 1 detected by the vehicle speed sensor 4.

However, w ₁₁ : the width of the image portion of the object at the time of previous imaging (imaging time t ₁₁ ), w ₁₂ : the width of the image portion of the object at the time of current imaging (imaging time t ₁₂ ), f: f = F (focal length of infrared camera 2) / p (pixel pitch of captured image), W: width of object in real space, Z ₁ : from vehicle 1 to object at the time of previous imaging (imaging time t ₁₁ ) Distance, Z ₂ : Distance from vehicle 1 to object at the time of current imaging (imaging time t ₁₂ ), Vs: Relative speed between vehicle and object, dT: Imaging interval, Tr: Time of arrival of own vehicle (object) Estimated time until the vehicle 1 reaches the vehicle 1).

Subsequently, the distance calculation unit 16 calculates the relative speed Vs between the vehicle 1 and the object (= the traveling speed Vj of the vehicle 1 + the moving speed Vd of the object) and the traveling speed Vj of the vehicle 1 in the above equation (1). The distance Z ₂ from the vehicle 1 to the object at the time of the current imaging is calculated by the following equation (2) which is assumed to be sufficiently higher than the moving speed Vd of the object and is replaced with the traveling speed Vj of the vehicle 1. calculate.

However, Z ₂ : distance from the vehicle 1 to the object at the time of the current imaging, Rate: rate of change, Vj: travel speed of the vehicle 1, dT: imaging interval.

[Real space position calculation processing]
The next STEP 6 is processing by the real space position calculation unit 17. The real space position calculation unit 17 calculates a distance Z ₁ from the vehicle 1 to the object at the time of the previous imaging using the following equation (3).

Where Z ₁ is the distance from the vehicle 1 to the object at the previous imaging, Z ₂ is the distance from the vehicle 1 to the object at the current imaging, Vj is the traveling speed of the vehicle 1, and dT is the imaging interval.

  Then, the real space position calculation unit 17 calculates the real space position of the object at the current and previous imaging from the position of the image portion of the object in the current and previous binary images.

Here, FIG. 13A shows the position Pi_2 (x ₁₂ , y ₁₂ ) of the image portion of the current object on the binary image Im11 and the position Pi_1 (x ₁₁ , y of the image portion of the previous object. ₁₁ ), the vertical axis y is set in the vertical direction of the image, and the horizontal axis x is set in the horizontal direction of the image.

FIG. 13B shows the movement of the object in real space, where the Z axis is set in the traveling direction of the vehicle 1 and the X axis is set in the direction orthogonal to the Z axis. In the figure, Pr_2 (X ₁₂ , Y ₁₂ , Z ₁₂ ) indicates the position of the object at the time of the current imaging, and Pr_1 (X ₁₁ , Y ₁₁ , Z ₁₁ ) indicates the position of the object at the time of the previous imaging. Show. Vm is a movement vector of an object in real space estimated from Pr_2 and Pr_1.

The real space position calculation unit 17 calculates the real space coordinates Pr_2 (X ₁₂ , Y ₁₂ , Z ₁₂ ) of the object at the time of the current imaging by the following formula (4), and the previous formula (5) The real space coordinates Pr_1 (X ₁₁ , Y ₁₁ , Z ₁₁ ) of the object at the time of imaging are calculated. Note that Z ₁₁ = Z ₁ and Z ₁₂ = Z ₂ .

Where X ₁₂ , Y ₁₂ are real space coordinate values of the object at the time of the current imaging, x ₁₂ , y ₁₂ are coordinate values of the image portion of the object in the current binary image, and Z _{2 is} at the time of the current imaging. Distance from vehicle to object, f: f = F (focal length of infrared camera) / p (pixel pitch of captured image).

However, X ₁₁ , Y ₁₁ : Real space coordinate value of the object at the previous imaging, x ₁₁ , y ₁₁ : Coordinate value of the image part of the object in the previous binary image, Z ₁ : At the previous imaging Distance from vehicle to object, f: f = F (focal length of infrared camera) / p (pixel pitch of captured image).

  In addition, the real space position calculation unit 17 performs a turning angle correction that corrects a positional shift on the image due to the turning of the vehicle 1 based on the turning angle recognized from the detection signal YR of the yaw rate sensor 3. Specifically, when the turning angle of the vehicle 1 between the previous imaging and the current imaging is θr, the real space coordinate value is corrected by the following equation (6).

However, Xr, Yr, Zr: real space coordinate value after turning angle correction, θr: turning angle, Xo, Yo, Zo: real space coordinate value before turning angle correction.

[Movement vector estimation processing]
The following STEP 7 is processing by the movement vector calculation unit 18. As shown in FIG. 13B, the movement vector calculation unit 18 calculates the object and the host vehicle from the real space position Pr_1 at the previous imaging time and the real space position Pr_2 at the current imaging time for the same target object. An approximate straight line Vm corresponding to a relative movement vector with respect to 1 is obtained.

  Note that the relative movement vector may be obtained using the real space position of the object at a plurality of past time points. Moreover, the specific calculation process of an approximate straight line is based on the method described in Unexamined-Japanese-Patent No. 2001-6096, for example.

[Alarm judgment processing]
Subsequent STEP 9 and STEP 30 to STEP 31 are processes by the alarm determination unit 19. In STEP 9, the alarm determination unit 19 alerts the object when the object exists in the approach determination area in front of the vehicle 1 or when the movement vector of the object is in the approach determination area. set to target.

  When the object is set as an alarm target, the alarm determination unit 19 further determines whether or not a brake operation is performed by the driver from the output of the brake sensor 5. When the brake operation is performed and the acceleration of the vehicle 1 (in this case, the deceleration direction is positive) is greater than a preset acceleration threshold value (it is assumed that an appropriate brake operation is performed by the driver). Is determined that the alarm output is unnecessary because the avoidance operation has been performed, and the process proceeds to STEP 10.

  On the other hand, if the brake operation is not performed or the acceleration of the vehicle 1 is equal to or less than the acceleration threshold value, the process branches to STEP30. Then, the alarm determination unit 19 outputs an alarm sound from the speaker 6 at STEP 30, displays an emphasized image of the target object on the HUD 7 at STEP 31, and proceeds to STEP 10.

  In the present embodiment, the distance calculation unit 16 calculates the rate of change Rate by the time tracking process of the image portion of the same object between the binary images shown in FIG. 12 in the second distance calculation process. The rate of change Rate may be calculated by the inter-phase calculation of the image portion of the object shown in FIG. Referring to FIG. 14, Im12 is a gray scale image at the time of previous imaging, and 101 indicates an image portion of the object. Im13 is a grayscale image at the time of the current imaging, and 102 indicates an image portion of the object.

  Then, the distance calculation unit 16 reduces or enlarges the size of the image portion 102 of the target object in the current grayscale image Im13 (when the target object approaches the host vehicle) or enlarges the target object from the host vehicle. Then, the degree of correlation with the image portion 101 of the object at the previous imaging is calculated. Specifically, as illustrated, an image 110 obtained by multiplying the image portion 102 by 1.5, an image 111 by 1.25 times, an image 112 by 1.0 times, an image 113 by 0.75 times, an image by 0.5 times The degree of correlation between the image 114 and the image portion 101 is calculated. Then, the distance calculation unit 16 determines the magnification of the image portion 102 when the degree of correlation is the highest as the rate of change Rate.

  Further, in the present embodiment, the configuration in which the front of the vehicle is imaged is shown, but it is also possible to determine the possibility of contact with the monitoring object by imaging other directions such as the rear and side of the vehicle. Good.

  In the present embodiment, the specific type of the present invention is a crossing pedestrian, but the shape of the image portion between time-series captured images such as a large wild animal crossing the road is the same object. For other types of objects that are assumed to change to an extent that it is difficult to extract as an image part of the vehicle, the distance from the vehicle 1 is assumed in advance, assuming the size (height, width, etc.) of the type of object. The present invention can be applied by setting the correlation between the image and the image portion of the object in the captured image.

  Further, in this embodiment, the crossing pedestrian is determined and the first distance calculation process and the second distance calculation process are switched. However, an object (a vehicle or a predetermined object) whose image shape change between time series images is small. Whether the object is a stationary object or the like, and if it is determined that the object has a small shape change, the second distance calculation process calculates the distance between the vehicle and the object, and the object has a small shape change. If it is not determined that the distance between the vehicle 1 and the object may be calculated by the first distance calculation process.

  In this case, instead of the determination of the crossing pedestrian in STEP 4 in FIG. 3, it is determined whether or not the object has a small shape change. If the object has a small shape change, the process proceeds to STEP 5 and the second distance calculation process is performed. Thus, the distance between the vehicle 1 and the object is calculated, and when the shape change is not small, the process branches to STEP 20, and the distance between the vehicle 1 and the object is calculated by the first distance calculation process.

  Further, in the present embodiment, in order to obtain the size (height) of the image portion of the object when performing the first distance calculation process, the original lower end position (lower end estimated position) of the image portion of the object is estimated. However, the application range of the present invention is not limited to this, and the present invention can be applied if it is necessary to obtain a position on the captured image corresponding to the ground contact position of the object in real space.

  In the present embodiment, as described above with reference to FIGS. 10, 11 (a), and 11 (b), processing for determining the lower end estimated position by performing correction based on the difference in vanishing point position. Although the correction is made based on the vehicle behavior (detected by the yaw rate, tilt sensor, etc.), the shape of the road recognized from the captured image, information from the navigation device, etc., the lower end estimated position is determined. Also good.

  DESCRIPTION OF SYMBOLS 1 ... Vehicle, 2 ... Infrared camera, 3 ... Yaw rate sensor, 4 ... Vehicle speed sensor, 5 ... Brake sensor, 6 ... Speaker, 7 ... HUD, 8 ... Navigation apparatus, 10 ... Vehicle periphery monitoring apparatus, 11 ... Captured image acquisition part , 12 ... object extraction unit, 13 ... object type determination unit, 14 ... road boundary extraction unit, 15 ... lower end position estimation unit, 16 ... distance calculation unit, 17 ... real space position calculation unit, 18 ... movement vector calculation unit, 19 ... Alarm judgment part.

Claims (7)

A captured image acquisition unit for acquiring an image captured by a camera mounted on the vehicle;
An object extraction unit for extracting an image portion of the object from the captured image;
A road boundary extraction unit that extracts an image part of a boundary between a roadway and a roadside belt or a sidewalk from the captured image;
When the image portion of the object is located above the boundary image portion, based on the intersection of the straight line drawn vertically downward from the image portion of the object and the boundary image portion, in real space A vehicle periphery monitoring device, comprising: a lower end position estimation unit that estimates a position on the captured image corresponding to a ground contact position of the object. The vehicle periphery monitoring device according to claim 1,
The road boundary extraction unit draws the straight line vertically downward from a horizontal center position of the image portion of the object when the image portion of the object is a pedestrian image portion. Vehicle periphery monitoring device. In the vehicle periphery monitoring device according to claim 1 or 2,
The vehicle periphery monitoring device, wherein the road boundary extraction unit extracts an image portion of the boundary using an edge filter for oblique line extraction. In the vehicle periphery monitoring device according to claim 3,
The road boundary extracting unit extracts, as an image portion of the boundary, a vehicle periphery monitoring, from among the edge lines extracted using the edge filter, an edge line whose slope is closest to a predetermined determination reference value apparatus. In the vehicle periphery monitoring device according to claim 4,
A road shape recognition unit for recognizing the shape of the road on which the vehicle is traveling;
A vehicle behavior detection unit for detecting the behavior of the vehicle,
The vehicle boundary monitoring device, wherein the road boundary extraction unit determines the determination reference value based on one or both of a shape of a road on which the vehicle is traveling and a behavior of the vehicle. In the vehicle periphery monitoring device according to any one of claims 1 to 5,
The vehicle boundary monitoring device, wherein the road boundary extraction unit extracts an image portion of the boundary from a boundary search region set in advance as a region where the image portion of the boundary is assumed to be located in the captured image. . In the vehicle periphery monitoring device according to any one of claims 1 to 6,
The lower end position estimation unit is configured to be perpendicular to the image portion of the object when the position of the vanishing point of the road in the captured image is deviated by a predetermined distance or more from a reference position set assuming a straight flat road. The position on the captured image corresponding to the ground contact position of the object in real space, the position obtained by correcting the position of the intersection of the straight line drawn downward and the image portion of the boundary according to the direction of the shift A vehicle periphery monitoring device characterized by estimating as follows.