JP5587250B2 - Ranging device - Google Patents

Description

  The present invention relates to a distance measuring device that measures a distance between an object existing around a vehicle and a host vehicle using a captured image of an in-vehicle camera.

  For the purpose of driving support of the vehicle, a captured image around the vehicle is acquired by an in-vehicle camera mounted on the front part of the vehicle, and based on the captured image, an object such as a pedestrian existing around the vehicle and the own vehicle A technique for measuring the distance between the two is conventionally known.

  For example, when the optical axis of the in-vehicle camera is substantially parallel to the road surface on which the vehicle is traveling (the vehicle existing road surface), when the object existing in the front location on the vehicle existing road surface is imaged by the in-vehicle camera, Estimate the distance between the target object and the host vehicle from the vertical deviation between the position of the grounding point of the target object and the center of the captured image, the focal length of the in-vehicle camera, and the height of the in-vehicle camera. be able to.

  However, in this case, if the road surface on which the object exists (the road surface on which the object touches) has a gradient with respect to the vehicle existing road surface, the object in the captured image depends on the gradient. The position of the grounding point of the object will shift up and down.

  For this reason, in order to accurately measure the distance between the target object and the host vehicle using the vertical deviation between the position of the grounding point of the target object and the center of the image in the captured image, the target object Therefore, it is necessary to appropriately compensate for the influence caused by the road surface gradient.

  On the other hand, as a technique for acquiring information related to a road surface gradient, for example, techniques disclosed in Patent Documents 1 and 2 are known.

  The technique found in Patent Document 1 estimates the inclination angle of a road surface based on the output of an acceleration sensor mounted on a vehicle. In addition, the technique shown in Patent Document 2 projects laser light onto a road surface in front of the vehicle, and based on the temporal change pattern of the received light intensity of the reflected light, the presence or absence of the road surface gradient in front of the vehicle, and the gradient thereof. It is judged whether it is an ascending slope or a descending slope.

JP 2006-194677 A JP 2003-65740 A

  However, since the technique shown in Patent Document 1 only knows the inclination angle of the road surface on which the host vehicle is present, it cannot acquire information on the gradient of the road surface on which the object around the vehicle exists.

  Further, in the technique shown in Patent Document 2, it is only possible to determine whether there is a road surface gradient ahead of the vehicle and whether the gradient is an ascending gradient or a descending gradient. It is difficult to grasp what influence the position of the ground contact point of the object has on the image.

  Therefore, it is difficult to apply road surface gradient information obtained by the techniques shown in Patent Documents 1 and 2 to a technique for measuring the distance between the object and the host vehicle based on the captured image.

  The present invention has been made in view of such a background, and appropriately compensates for the influence of the gradient of the road surface on which the object is present, and uses the image captured by the in-vehicle camera to determine the distance between the object and the host vehicle. An object of the present invention is to provide a distance measuring device capable of measuring with high accuracy.

  First, basic technical matters relating to the present invention will be described.

  As shown in FIG. 1A, a situation is assumed in which a vehicle 1 equipped with an in-vehicle camera 2 is traveling on a flat road surface 51. In this situation, there is an object 53 (pedestrian in the illustrated example) on which the distance D from the vehicle 1 (the host vehicle 1) is to be measured on the road surface 52 in front of the vehicle 1.

  In this case, the road surface 52 on which the object 53 is present (the road surface 52 on which the object 53 is grounded) is the road surface on the same plane as the road surface 51 on which the vehicle 1 is present (the road surface 51 on which the vehicle 1 is grounded) ( Road surface 51 and the same road surface). Hereinafter, the road surfaces 51 and 52 may be referred to as the own vehicle existence road surface 51 and the target object existence road surface 52, respectively.

  Moreover, FIG.1 (b) has shown roughly the picked-up image imaged with the vehicle-mounted camera 2 in the condition shown to Fig.1 (a). The portions denoted by reference numerals 51a, 52a, and 53a indicate images of the own vehicle existence road surface 51, the object existence road surface 52, and the object 53, respectively.

  In the situation shown in FIG. 1A, as shown in the figure, the angle formed by the optical axis Lc of the in-vehicle camera 2 with respect to the own vehicle existing road surface 51 is α, and the grounding point P53 of the object 53 from the in-vehicle camera 2 (the object 53). And an angle formed by the straight line L1 to the optical axis Lc of the in-vehicle camera 2 with respect to the optical axis Lc of the in-vehicle camera 2 and Hc from the own vehicle-existing road surface 51.

  Here, the height Hc of the in-vehicle camera 2 (hereinafter sometimes referred to as the camera height Hc) is more specifically the height of the optical center C of the in-vehicle camera 2 from the own vehicle existing road surface 51. More specifically, the straight line L1 related to the angle γ is a straight line from the optical center C of the in-vehicle camera 2 to the ground point P53 of the object 53. More specifically, the angles α and γ are angles around an axis in the normal direction of a plane that includes the optical axis Lc of the in-vehicle camera 2 and stands perpendicular to the vehicle-existing road surface 51. The angles α and γ correspond to the angle in the pitch direction of the vehicle 1 (that is, the angle around the axis of the vehicle 1 in the vehicle width direction) in the situation shown in FIG.

  The angle α is a situation in which the optical axis Lc of the in-vehicle camera 2 extends obliquely downward with respect to a plane parallel to the own vehicle existing road surface 51 through the optical center C of the in-vehicle camera 2 (FIG. 1A). In the situation shown in FIG. 2) and a negative angle in a situation where the optical axis Lc of the in-vehicle camera 2 extends obliquely upward with respect to the plane. The angle γ is a positive angle when the straight line L1 extends obliquely downward with respect to the optical axis Lc of the in-vehicle camera 2, and the straight line L1 is obliquely downward with respect to the optical axis Lc of the in-vehicle camera 2. A negative angle is assumed in the situation extending to

  In FIG. 1A, α ≠ 0 [deg], but α = 0 [deg] may be used.

  When the angles α and γ are defined as described above, the distance D of the object 53 from the host vehicle 1 (the distance in the direction parallel to the host vehicle existing road surface 51), the angles α and γ, and the camera height Hc The relationship of following Formula (1) is materialized between.

D = Hc / tan (α + γ) (1)

Next, as shown in FIG. 2A, a situation is assumed in which the object presence road surface 52 in front of the vehicle 1 is a road surface (inclined road surface) having a gradient with respect to the host vehicle presence road surface 51. . This situation is different from the situation shown in FIG. 1A only in the gradient state of the object existence road surface 52.

  In addition, FIG.2 (b) has shown roughly the picked-up image imaged with the vehicle-mounted camera 2 in the condition shown to Fig.2 (a). Similarly to the case of FIG. 1B, the portions denoted by reference numerals 51 a, 52 a, and 53 a indicate images of the own vehicle existence road surface 51, the object existence road surface 52, and the object 53, respectively.

  In the situation shown in FIG. 2A, angles α and γ in the figure are angles defined in the same manner as in the situation shown in FIG. In addition, in this situation, a point located below the in-vehicle camera 2 in the own vehicle existence road surface 51 (specifically, a vertical line dropped from the optical center C of the in-vehicle camera 2 to the own vehicle existence road surface 51 is the own vehicle existence road surface. The angle formed by the straight line L2 extending from the intersection 51 to the ground contact point P53 of the object 53 with respect to the own vehicle existing road surface 51 is β (> 0).

  Note that the angle β is similar to the angles α and γ and includes an optical axis Lc of the in-vehicle camera 2 and an angle around a normal axis of a plane that stands perpendicular to the vehicle-existing road surface 51 ( In the situation shown in FIG. 1B, the angle of the vehicle 1 in the pitch direction).

  When the angle β is defined in addition to the angles α and γ as described above, the distance D from the host vehicle 1 to the object 53, the angles α, γ, β, and the camera height Hc are as follows. The relationship of Formula (2) is materialized.

D = Hc / (tan (α + γ) + tanβ) (2)

In this case, in the equation (2), if β = 0, the equation (2) matches the equation (1). The situation shown in FIG. 1A corresponds to the situation where β in FIG. 2A is set to “0”.

  Accordingly, in any of the situations shown in FIGS. 1A and 2A, the value of the camera height Hc of the in-vehicle camera 2 and the values of the angles α, γ, and β (values in units of [rad]) ), The distance D between the object 53 and the host vehicle 1 can be specified by the calculation of the right side of the above equation (2) from these values.

  Here, since the camera height Hc is defined by the mounting position of the in-vehicle camera 2 with respect to the vehicle body of the vehicle 1, in general, the value can be set in advance as a known value.

  Further, the angle γ can be estimated based on the captured image of the in-vehicle camera 2. Specifically, as shown in FIG. 1B or FIG. 2B, the on-image object grounding point P53a, which is a point formed by projecting the grounding point P53 of the object 53 onto the captured image, and the captured image. Assuming that the vertical position deviation between the image center Pca (the intersection of the imaging surface of the in-vehicle camera 2 and the optical axis Lc) is Δy, the following equation (3) is established. Note that F is a focal length of the in-vehicle camera 2.

tanγ = Δy / F (3)

Accordingly, in the captured image, if the vertical position deviation Δy between the image object grounding point P53a and the image center Pca is specified, the value of Δy and the value of the focal length F (this is generally a known value). The angle γ can be estimated based on the above equation (3).

  Supplementally, on the object existence road surface 52, an in-vehicle camera 2 represents an arbitrary stationary point (a point fixed with respect to the object existence road surface 52) at which the distance from the vehicle 1 is the same as the ground contact point P53 of the object 53. For example, as shown in FIG. 1B or FIG. 2B, the point projected on the captured image extends in the horizontal direction (horizontal direction) of the captured image through the object grounding point P53a on the image. This is a point on the line L53a (for example, the point P1 in the figure).

  In this case, the vertical distance between this point (for example, P1) and the image center Pca coincides with the vertical distance Δy between the image object grounding point P53a and the image center Pca. Therefore, the angle γ can also be estimated based on the equation (3) from the interval Δy between the point (for example, P1) obtained by projecting the stationary point on the captured image and the image center Pca.

  The angle α is defined according to the posture of the vehicle body of the vehicle 1 (the posture in the increasing / decreasing direction of the angle α). Therefore, if the attitude of the vehicle body of the vehicle 1 is measured using an appropriate sensor such as a gyro sensor (may be estimated using an appropriate model), the value of the angle α can be estimated.

  For example, in the situation shown in FIG. 1A or 2A, if the inclination angle (pitch angle) of the vehicle body of the vehicle 1 is measured, the measured value of the pitch angle and the vehicle body of the vehicle 1 are measured. The angle α can be estimated from the direction of the optical axis Lc (direction in the pitch direction with respect to the vehicle body) determined by the mounting posture of the in-vehicle camera 2.

  In the situation where the vehicle 1 is traveling on a smooth road surface 51 (the own vehicle existing road surface 51), the vehicle body of the vehicle 1 generally has a posture in the front-rear direction of the vehicle body substantially equal to the own vehicle existing road surface 51. The posture is kept parallel. Therefore, in such a situation, the value of the angle α can be set in advance as a known value without measuring the posture of the vehicle body of the vehicle 1. For example, when the direction of the optical axis Lc of the in-vehicle camera 2 is substantially the same as the front-rear direction of the vehicle body of the vehicle 1, the value of the angle α can be regarded as approximately “0”.

  Further, the angle β can be estimated as described below by using two captured images captured at different times by the in-vehicle camera 2.

  First, as shown in FIG. 3A, it is assumed that a flat road surface S (plane) existing in front of the in-vehicle camera 2 is imaged at different times t1 and t2. A point obtained by projecting an arbitrary stationary point P on the road surface S onto a captured image at time t1 (hereinafter referred to as a first captured image) is P1, and a captured image at time t2 (hereinafter referred to as a second image). A point that is projected on the captured image) is defined as P2.

  In this case, the projection point P1 in the first captured image and the projection point P2 in the second captured image are each extracted as a feature point in each captured image, and the position of each point in the captured image is specified. To do.

Then, a position vector representing the position of the projection point P1 in the first captured image in homogeneous coordinates is ↑ P1 (≡ [x1, y1,1] ^T ), and the position of the projection point P2 in the second captured image is the same. A position vector expressed in the next coordinate is represented by ↑ P2 (≡ [x2, y2,1] ^T ).

  Note that x1 and y1 are the coordinate position in the horizontal direction and the coordinate position in the vertical direction (vertical direction) of the projection point P1 in the first captured image, respectively. Similarly, x2 and y2 are the coordinate position in the horizontal direction and the coordinate position in the vertical direction (vertical direction) of the projection point P2 in the second captured image, respectively. The subscript “T” means transposition.

  Further, a translation vector representing the spatial translational displacement of the position of the in-vehicle camera 2 in the period between the time t1 and the time t2 is represented by ↑ t, and the in-vehicle camera 2 in the period between the time t1 and the time t2. Let R be a rotation matrix representing the spatial rotational displacement of the posture. In other words, the translation vector ↑ t and the rotation matrix R represent the spatial translational displacement and the rotational displacement of the camera coordinate system in the period between the time t1 and the time t2, respectively.

  Note that the rotation matrix R is an orthogonal matrix. The camera coordinate system is a coordinate system fixed to the vehicle-mounted camera 2. More specifically, the camera coordinate system has, for example, the optical center C of the in-vehicle camera 2 as the origin, the horizontal direction of the captured image of the in-vehicle camera 2 in the X-axis direction, the vertical direction (vertical direction) in the Y-axis direction, and light. This is a three-axis coordinate system in which the direction of the axis Lc is the Z-axis direction.

  Further, as shown in FIG. 3B, a unit vector in the normal direction of the road surface S (hereinafter simply referred to as a normal vector) is represented by ↑ n, one of the times t1 and t2, for example, at time t1. Let d be the distance between the optical center C of the in-vehicle camera 2 and the road surface S. These ↑ n and d are plane parameters that define the spatial posture and position of the road surface S (plane), respectively.

In addition, since the road surface S is generally substantially parallel to the vehicle width direction, the road surface S is substantially parallel to the lateral direction of the captured image of the in-vehicle camera 2 (X-axis direction of the camera coordinate system). Therefore, when the inclination angle of the road surface S with respect to the optical axis Lc of the in-vehicle camera 2 at time t1 is θ as shown in FIG. 3B, the normal vector ↑ n of the road surface S is the camera coordinate system at time t1. ↑ n = [0, cos θ, sin θ] ^T is expressed using the coordinate component at.

  Here, the relationship between the position vector ↑ P1 of the projection point P1 of the first captured image and the position vector ↑ P2 of the projection point P2 of the second captured image is associated by plane projective transformation. Specifically, H is a homography matrix (planar projection transformation matrix) for converting the position vector ↑ P1 of the projection point P1 of the first captured image into the position vector ↑ P2 of the projection point P2 of the second captured image. Then, as shown in the following equation (4), ↑ P2 is a vector obtained by multiplying H ・ ↑ P1 by a certain proportionality constant k.

↑ P2 = k ・ H ・ ↑ P1 (4)

The homography matrix H is given by the following equation (5) using the translation vector ↑ t, the rotation matrix R, and the plane parameters ↑ n, d of the road surface S.

H = ^{RT- RT} , ↑ t, (↑ n / d) ^T (5)

Therefore, the following formula (6) is obtained by applying the formula (5) to the formula (4).

↑ P2 = k · (R ^T −R ^T · ↑ t · (↑ n / d) ^T ) · ↑ P1 (6)

In this equation (6), if ↑ P1, ↑ P2, R, and ↑ t are known, the value of (cosθ) / d or (sinθ) / d that is a component of ↑ n / d is obtained as follows. be able to.

That is, the translation vector ↑ t seen in the camera coordinate system at time t1 is expressed as ↑ t = [t1, t2, t3] ^T using coordinate components. The component display of the transposed matrix ^RT of the rotation matrix R is given by the following equation (7).

Further, variables a and b are defined as a≡ (cos θ) / d and b≡ (sin θ) / d, respectively, and ↑ n / d = [0, a, b] ^T is set. At this time, the expression (6) is expressed by the following expression (8) using the respective components of ↑ P1, ↑ P2, R ^T , ↑ t, and ↑ n / d.

Then, when k is deleted from the first row and the second row using the equation in the third row of the equation (8), the following equations (10a) and (10b) are obtained.

It should be noted that the determinant of the coefficient matrix (second-order square matrix with (ef) as the first row component and (hi) as the second row component) of the variables a and b related to the equations (10a) and (10b) is “ Therefore, the values of the variables a and b cannot be determined using these equations (10a) and (10b) as simultaneous equations.

  On the other hand, since a≡ (cos θ) / d and b≡ (sin θ) / d, the following equation (10c) is established.

a ² + b ² = 1 / d ² (10c)

Therefore, the values of a and b can be calculated by using the equations (10a) to (10c) as simultaneous equations. For example, when b is deleted using the formula obtained by adding both sides of the formulas (10a) and (10b) and the formula (10c), the following formula (11) is obtained.

Therefore, the value of the variable a can be obtained as a solution of the quadratic equation shown by the equation (11). In this case, there are generally two values of a that are the solutions of equation (11), but of these two values, the value of a that satisfies equations (10a) and (10b). Is finally selected as the value of a that satisfies all of the above formulas (10a) to (10c).

  Of course, a can be deleted from the equations (10a) to (10c) and the value of b can be determined in the same manner as described above. Moreover, the other value can also be determined from one value of a and b by the formula (10a) or (10b).

  Thus, if ↑ P1, ↑ P2, R, and ↑ t are known, the value of the variable a or b can be calculated based on the equations (10a) to (10c). Since a≡ (cos θ) / d and b≡ (sin θ) / d, if the value of the variable a or b is calculated as described above, the following equation (12a) or (12b ), The inclination angle θ of the road surface S with respect to the optical axis Lc of the in-vehicle camera 2 can be obtained.

θ = cos ⁻¹ (a · d) (12a)
θ = sin ⁻¹ (b · d) (12b)

Next, it is assumed that the first captured image captured by the vehicle-mounted camera 2 at time t1 is an image captured in the situation of FIG. 2A described above, and the imaging time t2 of the second captured image is the time t1. It is assumed that it is immediately after or immediately before. It is assumed that the time difference between the times t1 and t2 is a short time difference that makes the change in the distance between the object 53 and the vehicle 1 within the time interval sufficiently small.

  The stationary point P projected onto the first captured image and the second captured image at times t1 and t2, respectively, is the same as or substantially the same as the ground point P53 of the object 53 at the distance from the host vehicle 1 at the time t1. It is assumed that the object is a stationary point on the road surface 52. The stationary point P may be a stationary point that coincides with the grounding point P53 of the object 53 at time t1.

  Further, in the situation shown in FIG. 2 (a), the normal direction of the plane that stands up perpendicularly to the own vehicle existing road surface 51 including the optical axis Lc of the in-vehicle camera 2 (the vehicle in the situation of FIG. 1 is assumed to be a virtual road surface parallel to the vehicle width direction (direction perpendicular to the paper surface) and including the straight line L2 as a virtual road surface on which the object 53 exists. This is referred to as an object existence virtual road surface Sa. Then, the normal vector of the object existence virtual road surface Sa and the distance from the optical center C of the in-vehicle camera 2 to the object existence virtual road surface Sa at time t1 are newly set as ↑ n and d, respectively.

  At this time, when the plane parameters ↑ n and d of the object existence virtual road surface Sa are applied to the right side of the equation (6), the position vector ↑ P1 of the projection point P1 of the first captured image and the second captured image It is considered that the relationship represented by the equation (6) is approximately established with the position vector ↑ P2 of the projection point P2.

  Accordingly, the translation vector ↑ t and the rotation matrix R representing the change in the position and orientation of the in-vehicle camera 2 in the period between the time t1 and the time t2, respectively, and the distance from the host vehicle 1 at the time t1 are the object 53. Position vectors ↑ P1 and ↑ P2 of projection points P1 and P2 obtained by projecting a stationary point P on the object presence road surface 52 that is the same as the ground contact point P53 to the first and second captured images, respectively. When the variable a (≡ (cos θ) / d) or b (≡ (sin θ) / d) is calculated as described above, the value of the variable a or b is approximately the object existing virtual road surface Sa. It corresponds to the value of the component of ↑ n / d corresponding to.

  The distance d between the object existence virtual road surface Sa and the optical center C of the in-vehicle camera 2 at time t1 is considered to approximately match the camera height Hc.

  Therefore, by using the camera height Hc as the value of d, from the value of d (= Hc) and the value of the variable a or b, the above equation (12a) or equation (12b) is used. The inclination angle θ of the object existence virtual road surface Sa with respect to the optical axis Lc of the in-vehicle camera 2 (in other words, the inclination angle of the straight line L2 with respect to the optical axis Lc in FIG. 2A) can be estimated.

  On the other hand, as can be seen with reference to FIG. 2A, the inclination angle θ of the object existence virtual road surface Sa with respect to the optical axis Lc of the in-vehicle camera 2 is θ = α + β. Accordingly, the angle β necessary for measuring the distance D of the object 53 from the host vehicle 1 based on the equation (2) is the inclination angle θ of the object existence virtual road surface Sa estimated as described above. From the angle α, it can be calculated by the following equation (13).

β = θ−α (13)

As described above, it is possible to estimate the value of the angle β using two captured images captured by the in-vehicle camera 2 at different times t1 and t2.

  If the values of the angles γ, α, and β are estimated as described above, the distance D between the object 53 and the host vehicle 1 can be measured based on the equation (2).

  The present invention will be described below based on the technical matters described above.

A distance measuring device of the present invention is a distance measuring device that measures a distance between an object on a road surface imaged by a vehicle-mounted camera and the host vehicle on which the vehicle-mounted camera is mounted,
From two captured images captured by the in-vehicle camera at different image capturing times, the two captured images are stationary points on the object existing road surface that is the road surface on which the object exists, and the distance from the own vehicle is the two captured images. A feature point formed by projecting a stationary point that is the same as the ground contact point of the object at the imaging time of the first captured image, which is one of the two, to each of the two captured images is extracted, and the two image capturing is performed. Distance measurement feature point extracting means for specifying the position of the feature point in each of the images;
Camera motion parameter measuring means for measuring camera motion parameters representing changes in the position and orientation of the in-vehicle camera in a period between the respective imaging times of the two captured images;
Based on the position of the feature point specified in the first captured image, a value of a first reference angle that is an angle formed by a straight line from the in-vehicle camera to the stationary point with respect to the optical axis of the in-vehicle camera is estimated. First reference angle estimation means;
The own vehicle existence road surface, which is the road surface on which the own vehicle exists at the imaging time of the first captured image, the position of the feature point in each of the two captured images, the height of the in-vehicle camera, and the imaging time of the first captured image Based on an arithmetic expression representing a relationship between a second reference angle that is an angle formed with respect to the optical axis of the in-vehicle camera from a point located below the in-vehicle camera to the stationary point From the measured value of the camera motion parameter, the position of the feature point specified in each of the two captured images, and the set value of the height of the in-vehicle camera, the second reference as an unknown in the arithmetic expression Second reference angle estimation means for estimating an angle value;
Distance estimating means for estimating a distance between the object and the host vehicle from at least an estimated value of the first reference angle, an estimated value of the second reference angle, and a set value of a height of the in-vehicle camera; (First invention).

  More specifically, the first reference angle and the second reference angle in the first invention are plane methods that include the optical axis of the in-vehicle camera and that stand perpendicular to the road surface on which the vehicle exists. An angle around an axis in the linear direction or substantially the same direction. The same applies to the third reference angle described later.

  According to the first aspect of the present invention, the distance from the own vehicle is the same as the ground contact point of the object at the imaging time of the first captured image of the two captured images by the distance measurement feature point extraction unit. The feature points formed by projecting the still points on the road surface where the object exists are projected onto the two captured images are extracted. Then, the position of the feature point in each of the two captured images is specified. As a result, the position vectors ↑ P1 and ↑ P2 described above are specified.

  The feature point in the first captured image is, for example, a point formed by projecting the ground point of the object at the imaging time of the first captured image onto the first captured image (FIG. 1B or FIG. 2). b) In the example shown, it can be extracted as a point having local features such as luminance on a horizontal line passing through the point P53a). Then, the feature point in the other captured image (second captured image) can be extracted as a point having the same feature as the feature point of the first captured image in the second captured image. As such a feature point, for example, a Harris corner point, a minimum eigenvalue point, or the like can be used.

  The camera motion parameter measuring means measures camera motion parameters representing changes in the position and orientation of the in-vehicle camera in the period between the respective imaging times of the two captured images. The camera motion parameters are parameters that can define the translation vector ↑ t and the rotation matrix R, and may be the translation vector ↑ t and the rotation matrix R itself.

  The camera motion parameters such as the translation vector ↑ t and the rotation matrix R can be measured based on the output of a sensor such as a gyro sensor mounted on the vehicle, for example. Or it is also possible to measure using the picked-up image of a vehicle-mounted camera. In that case, camera motion parameters (for example, the translation vector ↑ t and the rotation matrix R) can be calculated by a known image processing method based on, for example, Structure from Motion (hereinafter, sometimes referred to as SfM).

  Furthermore, the value of the first reference angle is estimated by the first reference angle estimation means based on the position of the feature point in the first captured image. This first reference angle means the angle γ in FIG. 1 (a) or FIG. 2 (a). Therefore, the value of the first reference angle (γ) can be estimated based on the position of the feature point in the first captured image. More specifically, the value of the first reference angle (γ) can be estimated based on the equation (3).

  Further, the value of the second reference angle is estimated by the second reference angle estimation means. The second reference angle is such that a straight line extending from a point located below the vehicle-mounted camera to a stationary point on the road surface where the vehicle exists is a road surface where the vehicle is present at the time of capturing the first captured image. This is the angle formed with respect to the optical axis of the camera. Therefore, the second reference angle corresponds to the inclination angle θ of the object existence virtual road surface (Sa) with respect to the optical axis of the in-vehicle camera.

  Therefore, between the camera motion parameter, the position of the feature point in each of the two captured images, the height of the in-vehicle camera, and the second reference angle, the equations (10a) to (10c) are used. The relationship shown is approximately established. Then, based on an arithmetic expression representing the relationship, from the measured value of the camera motion parameter, the position of the feature point specified in each of the two captured images, and the set value of the height of the in-vehicle camera The value of the second reference angle (θ) can be estimated.

  And in 1st invention, the distance between the said target object and the own vehicle from the estimated value of the said 1st reference angle, the estimated value of the said 2nd reference angle, and the setting value of the height of the said vehicle-mounted camera at least. Is estimated by the distance estimation means.

  In this case, for example, in a situation where the vehicle is traveling on a flat road surface, the angle α in the equation (2) (the angle formed by the optical axis of the in-vehicle camera with respect to the vehicle existing road surface) is approximately It becomes a constant value (known value). For this reason, from the estimated value of the first reference angle (γ), the estimated value of the second reference angle (θ), and the set value of the height of the in-vehicle camera, the value of the angle α is a predetermined constant value. The distance between the object and the host vehicle can be estimated based on the equations (2) and (13).

  Alternatively, the angle α may be estimated by an appropriate method. In that case, from the estimated value of the angle α, the estimated value of the first reference angle (γ), the estimated value of the second reference angle (θ), and the set value of the height of the in-vehicle camera, Based on the equations (2) and (13), the distance between the object and the host vehicle can be estimated.

  According to the first aspect of the present invention, even if the target object existence road surface has a gradient with respect to the own vehicle existence road surface, the degree of the gradient is reflected and the distance between the object and the own vehicle is reduced. The distance can be estimated. Therefore, it is possible to accurately measure the distance between the target object and the host vehicle using the image captured by the in-vehicle camera while appropriately compensating for the influence of the gradient of the target object road surface.

  In the first invention, the camera motion parameter measuring means includes, for example, a translation vector that is a displacement vector of the position of the in-vehicle camera in a period between the imaging times of the two captured images, and the in-vehicle camera. It is a means for measuring, as the camera motion parameter, a rotation matrix that represents the angle change amount of the posture. These translation vector and rotation matrix mean the translation vector ↑ t and the rotation matrix R, respectively.

  In this case, the second reference angle estimation means specifically estimates the value of the second reference angle (θ) based on the expressions (10a) to (10c) as the arithmetic expressions. (Second invention).

  Here, the formulas (10a) to (10c) and the meaning or definition of each variable are described again as follows.

According to the second aspect of the invention, based on the above equations (10a) to (10c), the measured value of the camera motion parameter (the value of each component of the translation vector ↑ t and the value of each component of the rotation matrix R) ) And the position of the feature point specified in each of the two captured images (values of the respective components of the position vectors ↑ P1 and ↑ P2), the value of the variable a or b is obtained as described above. Can do. Then, from the value of the variable a or b and the set value (Hc) of the vehicle-mounted camera height as the value of d, the estimated value of the second reference angle (θ) according to the equation (12a) or (12b). Can be requested.

In the first invention, third reference angle estimation means for estimating a value of a third reference angle, which is an angle formed by an optical axis of the vehicle-mounted camera at the imaging time of the first captured image with respect to the own vehicle existing road surface. Further comprising
The distance estimating means is configured to calculate the object based on the estimated value of the first reference angle, the estimated value of the second reference angle, the estimated value of the third reference angle, and the set value of the height of the in-vehicle camera. It is preferable to estimate the distance between the vehicle and the host vehicle (third invention).

  According to the third aspect of the invention, the third reference angle estimation unit estimates the third reference angle. This third reference angle means the angle α in FIG. 1 (a) or FIG. 1 (b). And the estimated value of this 3rd reference angle ((alpha)) is used in order to estimate the distance between the said target object and the own vehicle. As a result, even when the actual value of the third reference angle (α) varies during travel of the vehicle, it is possible to accurately estimate the distance between the object and the host vehicle.

  In the third aspect of the invention, the third reference angle can be estimated by various methods. As the method, the following method can be used.

  That is, the camera motion parameter measuring means measures a translation vector that is a displacement vector of the position of the vehicle-mounted camera in a period between the imaging times of the two captured images as a component of the camera motion parameter. In this case, the third reference angle estimation means estimates the value of the third reference angle based on at least the direction of the measured translational movement vector with respect to the optical axis of the in-vehicle camera (fourth invention). ).

  In this case, since the direction of the translational movement vector is substantially parallel to the road surface on which the host vehicle exists, the third reference angle (α) is basically determined by the vehicle-mounted camera of the translational movement vector. It is defined according to the direction with respect to the optical axis. Therefore, the value of the third reference angle (α) can be estimated based on the direction of the translation vector relative to the optical axis of the in-vehicle camera.

  The third reference angle may be estimated by the following method, for example. That is, the camera motion parameter measuring means calculates a translation vector that is a displacement vector of the position of the vehicle-mounted camera and an angle change amount of the posture of the vehicle-mounted camera in a period between the respective imaging times of the two captured images. A rotation matrix that represents the camera motion parameter, the A provisional estimation of the third reference angle based on the direction of the measured translational movement vector relative to the optical axis of the in-vehicle camera. A third reference angle A estimating means for sequentially obtaining a value, and a third reference angle for successively obtaining a B provisional estimated value of the third reference angle by integrating the amount of angle change indicated by the measured rotation matrix. B estimation means, and the third reference angle estimation means includes an average value of the A reference provisional value of the third reference angle for a predetermined period and a B provisional estimation. Of said predetermined calculating a deviation between the average value of the periodically between content, determining an estimated value of the third reference angle by correcting the first B provisional estimated value according to the deviation (fifth invention).

  According to this, the third reference angle A estimating means is similar to the third reference angle estimating means of the fourth aspect of the invention based on the direction of the translation vector (direction with respect to the optical axis of the in-vehicle camera). 3 Estimate the value of the reference angle (α). However, in the fifth invention, the estimated value is the A-th provisional estimated value (temporary estimated value of the third reference angle (α)).

  Further, the third reference angle B-estimating means integrates the angle change amount of the third reference angle (α) obtained as the angle change amount of the attitude of the in-vehicle camera indicated by the rotation matrix, thereby obtaining a third A provisional estimated value (Bth provisional estimated value) of the reference angle (α) is sequentially obtained.

  Here, since the A-th provisional estimated value can be calculated without requiring an integration operation, no error accumulates in the A-th provisional estimated value. However, since the A-th provisional estimated value is directly affected by an instantaneous change in the posture of the vehicle, an instantaneous error may occur with respect to the actual value of the third reference angle (α). is there.

  On the other hand, since the B-th provisional estimated value is obtained by integrating the angle change amount as a relative change amount between different times of the in-vehicle camera posture, an instantaneous change in the vehicle posture is obtained. It is hard to be influenced by. However, the B-th provisional estimated value is likely to include an error in the initial value of the B-th provisional estimated value and a cumulative error. As a result, the B-th provisional estimated value tends to cause an offset with respect to the actual value of the third reference angle (α).

  Therefore, in the fifth aspect of the invention, the third reference angle estimation means includes an average value for the predetermined period of the A provisional estimated value of the third reference angle (α) and an average of the B reference provisional value for the predetermined period. A deviation from the value is calculated, and the estimated value of the third reference angle (α) is determined by correcting the B-th provisional estimated value according to the deviation.

  In this case, the deviation corresponds to an offset of the B-th provisional estimated value with respect to the actual value of the third reference angle (α). For this reason, it is possible to remove the offset from the B-th provisional estimated value by correcting the A-th provisional estimated value according to the deviation. As a result, the estimated value of the third reference angle (α) obtained by the correction can be more accurately matched with the actual value of the third reference angle (α). As a result, the measurement accuracy of the distance between the object and the host vehicle can be further enhanced.

  Note that a Kalman filter is used from the A-th provisional estimated value of the third reference angle (α) and the B-th estimated value that has not been accumulated (that is, the amount of change in the attitude of the in-vehicle camera 2 per unit time). The third reference angle may be estimated.

  The third to fifth inventions may be combined with the second invention.

FIG. 1A is a diagram illustrating an example of a state of a vehicle and an object, and FIG. 1B is a diagram illustrating an outline of a captured image captured by a vehicle-mounted camera in the state of FIG. FIG. 2A is a diagram illustrating another example of the state of the vehicle and the object, and FIG. 2B is a diagram illustrating an outline of a captured image captured by the in-vehicle camera in the situation of FIG. FIG. 3A is a diagram showing an imaging state at different times t1 and t2 by the in-vehicle camera, and FIG. 3B is a diagram showing an imaging state at time t1 in FIG. The block diagram which shows the structure of the ranging apparatus in one Embodiment of this invention. FIGS. 5A and 5B are diagrams for explaining processing of the 3A reference angle A-th estimation unit in FIG. 4.

  An embodiment of the present invention will be described below with reference to FIGS.

  Referring to FIG. 4, the distance measuring device 10 of the present embodiment is mounted on the vehicle 1 shown in FIGS. 1A and 2A and is connected to the in-vehicle camera 2. The arithmetic processing unit 11 is provided. In addition, in FIG. 1A and FIG. 2A, the vehicle 1 is schematically illustrated, but the actual vehicle 1 is, for example, an automobile.

  In the present embodiment, the in-vehicle camera 2 is mounted on the front of the vehicle 1 to monitor the front of the vehicle 1 and captures an image in front of the vehicle 1. In this case, the in-vehicle camera 2 is attached to the vehicle body of the vehicle 1 such that its optical axis Lc is directed in the front-rear direction of the vehicle 1. However, the optical axis Lc of the in-vehicle camera 2 may have a slight inclination with respect to the longitudinal direction of the vehicle 1.

  The in-vehicle camera 2 is constituted by a CCD camera, a CMOS camera, or the like, and generates and outputs an image signal of a captured image. The captured image may be either a color image or a monotone image.

  The in-vehicle camera 2 may be a camera mounted on the vehicle 1 so as to capture a rear or side image of the vehicle 1.

  The arithmetic processing unit 11 is an electronic circuit unit including a CPU, a RAM, a ROM, an interface circuit, etc. (not shown), and receives an image signal generated by the in-vehicle camera 2.

  The arithmetic processing unit 11 includes, as a function realized by executing the installed program, an image acquisition unit 12 that acquires a captured image of the in-vehicle camera 2, and the target 53 and the host vehicle 1 from the captured image. A distance-measuring feature point extraction unit 13 for extracting feature points used for measuring a distance, a camera motion estimation unit 14 for estimating (measuring) a motion parameter of the in-vehicle camera 2, and first to third reference angles, respectively. Distance estimation for obtaining an estimated value (measured value) of the distance between the object 53 and the vehicle 1 and the first reference angle estimation unit 15, the second reference angle estimation unit 16, and the third reference angle estimation unit 17 to be estimated Part 18. The arithmetic processing unit 11 sequentially executes the processing of these functional units at a predetermined arithmetic processing cycle, thereby estimating the distance D between the object 53 existing in front of the vehicle 1 and the host vehicle 1. Are calculated sequentially.

  The details of the overall processing of the arithmetic processing unit 11 including the processing of each functional unit of the arithmetic processing unit 11 will be described below.

  The in-vehicle camera 2 captures an image in front of the vehicle 1 at a timing instructed by the arithmetic processing unit 11 at a predetermined arithmetic processing cycle. And the image signal produced | generated by the imaging is taken in sequentially by the image acquisition part 12 of the arithmetic processing unit 11 from the vehicle-mounted camera 2. FIG. The image acquisition unit 12 converts an image signal (an image signal for each pixel) that is an analog signal input from the in-vehicle camera 2 into digital data, and stores and stores the digital signal in an image memory (not shown).

  In this case, the image acquisition unit 12 periodically updates a plurality of captured images, for example, a set of two captured images captured at different times (times at time intervals of a predetermined calculation processing cycle) by the in-vehicle camera 2. However, it is stored in the image memory. In the following description, the imaging times of two captured images stored and held in the image memory are referred to as times t1 and t2, the captured image at time t1 is referred to as a first captured image, and the captured image at time t2 is referred to as a second captured image. Sometimes. Note that the time t2 may be either the time after the time t1 or the previous time.

  Two captured images (first captured image and second captured image) stored and held in the image memory by the processing of the image acquisition unit 12 are given to the distance measurement feature point extraction unit 13 and the camera motion estimation unit 14, Each process is executed next.

  The distance measurement feature point extraction unit 13 performs the first picked-up image of the contact point P53 (contact point between the target object 53 and the target object existing road surface 52) of the target object 53 whose distance from the host vehicle 1 is to be measured. The processing of the on-image object grounding point specifying unit 13a for identifying the on-image object grounding point P53a projected on the vehicle and the distance from the own vehicle 1 are the same (or almost the same) as the grounding point P53. It is roughly divided into a feature point search unit 13b for searching for feature points P1 and P2 obtained by projecting a stationary point P on the object presence road surface 52 on the first and second captured images.

  The on-image object grounding point specifying unit 13a executes processing for specifying the on-image object grounding point P53a as follows.

  The on-image target ground point specifying unit 13a first searches the given first captured image for an image portion of a predetermined type of target object 53 whose distance to the host vehicle 1 is to be measured. In the present embodiment, the predetermined type of object 53 is, for example, a pedestrian (person). Then, the on-image object grounding point specifying unit 13a searches the first captured image for an image portion having a shape pattern or the like characteristic to the pedestrian, whereby the object 53 ( The image portion of (pedestrian) is extracted. For example, in the situation shown in FIG. 1 (a) or FIG. 2 (a), the portion indicated by reference numeral 53a in FIG. 1 (b) or FIG. 2 (b) is extracted as the image portion of the object 53 (pedestrian). Is done.

  Various methods are known for searching for a pedestrian image portion from a captured image, and a pedestrian as an object 53 is extracted from the first captured image using the known method. You can do it.

  Then, the on-image object ground point identification unit 13a identifies the on-image object ground point P53a of the object 53 in the first captured image extracted as described above. In this case, as shown in FIG. 1B or FIG. 2B, for example, the lowest point of the image 53a of the object 53 in the first captured image is specified as the on-image object ground point P53a.

  In addition, when the image of the front-end | tip part of a pedestrian's leg cannot be obtained clearly, this pedestrian in the 1st picked-up image assumes that the pedestrian's figure is a standard figure, for example. The position of the object ground contact point P53a on the image may be estimated from the positions of the head and shoulders of the image.

  Next, the process of the feature point search unit 13b is executed as follows.

  In the present embodiment, the in-vehicle camera 2 is mounted on the vehicle 1 so that the lateral direction of the captured image (the x-axis direction of the camera coordinate system) is substantially parallel to the own vehicle existing road surface 51.

  For this reason, the point formed by projecting a stationary point on the object existence road surface 52 such that the distance from the own vehicle 1 is the same distance as the object ground contact point P53 is basically the following: As shown in FIG. 1B or FIG. 2B, it exists on a straight line L53a extending in the horizontal direction (x-axis direction of the camera coordinate system) through the object grounding point P53 on the image.

  Therefore, the feature point search unit 13b includes feature points (for example, FIG. 1B) that exist in the image 52a of the target object road surface 52 on the straight line L53a in the image region in the vicinity of the image 53a of the target object 53. Alternatively, the point P1) in FIG. 2B is searched, and the first imaging of the point is a stationary point on the object existence road surface 52 (a stationary point where the distance from the vehicle 1 is the same as the object grounding point P53). It is extracted as a feature point projected on the image.

  Such a feature point generally shows a characteristic (distinguishable from other local regions) size or change in state quantities such as luminance, saturation, and hue in the local region including the point. It is such a point. For example, a Harris corner point, a minimum eigenvalue point, or the like is searched for as the feature point.

  After extracting the feature points in the first captured image in this way, the feature point search unit 13b then searches the second captured image for points having the same features as the feature points of the first captured image, and searches for them. A point is extracted as a feature point obtained by projecting the stationary point (a stationary point on the object existence road surface 52 whose distance from the vehicle 1 is the same as the object grounding point P53) onto the second captured image. The search process in the second captured image may be performed using a known image processing method such as block matching.

  As a result of the above processing of the object grounding point specifying unit 13a and the feature point searching unit 13b on the image, the first image of a stationary point on the object existing road surface 52 where the distance from the vehicle 1 is the same as the object grounding point P53 is obtained. A feature point formed by projecting on the image and a feature point formed by projecting the stationary point on the second captured image are extracted. The feature points extracted from the first captured image and the second captured image in this way are the feature points P1 and P2 shown in FIG.

  Then, the distance measurement feature point extraction unit 13 obtains and outputs the positions of the feature points P1 and P2 extracted in this manner in the captured images (the coordinate components of the position vectors ↑ P1 and ↑ P2).

  Supplementally, in this embodiment, since the object 53 (pedestrian) is a moving object, the object ground contact point P53 is generally at the imaging time t1 of the first captured image and the imaging time t2 of the second captured image. Does not match. For this reason, in this embodiment, a point formed by projecting a stationary point different from the object ground point P53 on each captured image is extracted from each captured image as the feature point.

  However, the object 53 whose distance is to be measured may not be a moving body such as a pedestrian, but may be a fixed installation object (stationary object). In that case, you may make it extract the point formed by projecting the grounding point of the fixed installation object as a target object on each captured image from each captured image as the said feature point.

  In the present embodiment, the camera motion estimation unit 14 includes two given captured images (first captured image and second captured image), a frame rate indicating the number of captured images per unit time of the in-vehicle camera 2, As an estimated value (measured value) of the displacement vector of the spatial position of the in-vehicle camera 2 using a known image processing technique based on SfM (Structure from Motion) from the vehicle speed detected by the vehicle speed sensor that does not perform The translation vector ↑ t and the rotation matrix R indicating the estimated value (measured value) of the angular change amount of the spatial orientation of the in-vehicle camera 2 in the period between the imaging times t1 and t2 of the two captured images. Is calculated.

  The translation vector ↑ t and the rotation matrix R are obtained from feature points in each captured image of a stationary object (fixed installation) image projected on the first captured image and the second captured image, for example. Known methods based on SfM for optical flow (eg “Structure from Motion without Correspondence” / Frank Dellaert, Steven M. Seitz, Charles E. Thorpe, Sebastian Thrun / Computer Sience Department & Robotics Institute Canegie Mellon University, Pittsborgh PA 15213 ).

  Supplementally, the translational movement vector calculated based on SfM has an indefinite scale, but the scale of the translational movement vector ↑ t is determined based on the frame rate and the detected vehicle speed.

  Since the in-vehicle camera 2 moves integrally with the vehicle body of the vehicle 1, an attitude sensor such as a gyro sensor for detecting the attitude of the vehicle body and an acceleration sensor for detecting the acceleration of the vehicle body are mounted on the vehicle 1. If there is, for example, the translation vector ↑ t or the rotation matrix R may be measured based on the outputs of these sensors.

  The arithmetic processing unit 11 performs the processing of the distance measurement feature point calculation unit 13 and the camera motion estimation unit 14 as described above, and then performs the first reference angle estimation unit 15, the second reference angle estimation unit 16, the first The process of the 3 reference angle estimation part 17 is each performed. Note that the process of the first reference angle estimation unit 15 may be executed in parallel with or before the process of the camera motion estimation unit 14. Further, the process of the third reference angle estimation unit 17 may be executed in parallel with or before the process of the distance measurement feature point calculation unit 13.

The position of the feature point P1 in the first captured image (or the position vector ↑ P1) is input to the first reference angle estimation unit 15 from the distance measurement feature point extraction unit 13. The first reference angle estimator 15 then determines the positional deviation Δy in the vertical direction (y-axis direction of the camera coordinate system) between the image center Pca of the first captured image and the input feature point P1 (FIG. 1). (See (b) or FIG. 2 (b)). Then, the first reference angle estimation unit 15 calculates the first deviation from the positional deviation Δy and the value of the focal length F of the in-vehicle camera 2 stored and held in advance in a storage device (not shown) based on the equation (3). An estimated value (= tan ⁻¹ (Δy / F)) of the angle γ as the reference angle is calculated.

  The second reference angle estimator 16 includes the positions of the feature points P1 and P2 (or the position vectors ↑ P1 and ↑ P2) in the first captured image and the second captured image from the distance measurement feature point extraction unit 13, respectively. And the translation movement vector ↑ t and the component values of the rotation matrix R are input from the camera motion estimation unit 14.

  Then, the second reference angle estimation unit 16 includes component values of the position vectors ↑ P1 and ↑ P2 of the feature points P1 and P2 indicated by the input values from the distance measurement feature point extraction unit 13, and a camera motion estimation unit. 14 between the translation movement vector ↑ t and the rotation matrix R inputted from 14 and d (the object existing virtual road surface Sa and the optical center C of the vehicle-mounted camera 2 at time t1) in the equation (10c). Based on the above formulas (10a) to (10c), the angle θ as the second reference angle is calculated from the value of the camera height Hc stored and held in advance in a storage device (not shown) as an approximate set value of Calculate an estimate.

  Specifically, for example, the values of the coefficients r, s, and t of the quadratic equation of the formula (11) are defined in the proviso for the formulas (10a) to (10c) and the proviso for the formula (11). Is calculated from the component values of ↑ P1, ↑ P2, ↑ t, R. Then, using the values of the coefficients r, s, and t, the value of the solution of the quadratic equation of Equation (11) is calculated as the value of the variable a (≡ (cos θ) / d).

  In this case, as described above, there are generally two values of a which are the solutions of the equation (11), but of these two values, the equations (10a) and (10b) are satisfied. The value of a to be obtained is selected.

  Then, the estimated value of the second reference angle θ is calculated from the value of the variable a and the set value (Hc) of d by the equation (12a).

  Instead of the variable a, the value of the variable b (≡ (sin θ) / d) is calculated, and the value of the variable b and the set value (Hc) of d are calculated according to the equation (12b). 2 An estimated value of the reference angle θ may be calculated.

  The third reference angle estimation unit 17 receives the translation movement vector ↑ t and the component values of the rotation matrix R from the camera motion estimation unit 14.

  The third reference angle estimator 17 executes a process of calculating the A-th provisional estimated value αa and the B-th tentative estimated value αb of the angle α as the third reference angle, respectively. 17a and a third reference angle B-th provisional estimation unit 17b.

  The third reference angle A provisional estimation unit 17a determines the third reference angle α of the third reference angle α based on the direction of the input translational movement vector ↑ t (direction relative to the optical axis Lc of the vehicle-mounted camera 2 at the imaging time t1). A provisional estimated value αa is calculated.

  In this case, the translation vector ↑ t can be regarded as a vector that is substantially parallel (substantially parallel to the traveling direction of the vehicle 1) to the own vehicle existing road surface 51 where the vehicle 1 is traveling at the time t1.

  Therefore, the direction of the translation vector ↑ t with respect to the optical axis Lc of the in-vehicle camera 2 (or the direction of the optical axis Lc with respect to the translation vector ↑ t) is such that the optical axis Lc of the in-vehicle camera 2 is relative to the own vehicle existing road surface 51. The angle formed depends on the third reference angle α.

  That is, as shown in FIGS. 5A and 5B, the translational movement vector ↑ t seen in the camera coordinate system of the vehicle-mounted camera 2 at time t1 is relative to the optical axis Lc (Z-axis) of the vehicle-mounted camera 2. The formed angle αa corresponds to the third reference angle α.

  Therefore, in the present embodiment, the third reference angle A-provisional estimation unit 17a determines an angle αa formed by the input translational movement vector ↑ t with respect to the XZ plane of the camera coordinate system of the vehicle-mounted camera 2 at time t1. Calculated as a provisional estimated value of the third reference angle α.

Specifically, the third reference angle A temporary estimation unit 17a calculates the component value of the translation vector ↑ t (= [t1, t2, t3] ^T ) in the camera coordinate system of the in-vehicle camera 2 at time t1. The A-th provisional estimated value αa is calculated by the following equation (14).

Also, the third reference angle B-th provisional estimation unit 17b calculates a B-th provisional estimated value αb of the third reference angle α based on the input rotation matrix R.

  Specifically, the third reference angle B-provisional estimation unit 17b determines the amount of change in the angle of the spatial orientation of the in-vehicle camera 2 indicated by the input rotation matrix R (the imaging times t1 and t2 of the two captured images). Of the change in angle in the pitch direction of the vehicle 1 (the direction around the axis where the change in the third reference angle α occurs) is sequentially accumulated (accumulated addition) in the calculation processing cycle of the calculation processing unit 11. By doing so, the B-th provisional estimated value αb of the third reference angle α is calculated.

  The B-th provisional estimated value αb is initialized when the vehicle 1 starts to operate. In this case, as an initial value of the B-th provisional estimated value αb, for example, a value (for example, “0”) defined in advance as the value of the third reference angle α when the vehicle 1 is stopped on a horizontal road surface. ) Or the value of the third reference angle α finally calculated during the previous driving of the vehicle 1 is set.

  Through the processing of the third reference angle A estimation unit 17a and the third reference angle B estimation unit 17b described above, the A provisional estimated value αa of the third reference angle α based on the translation vector ↑ t and the rotation matrix R And the B provisional estimated value αb of the third reference angle α based on the above are sequentially calculated at a predetermined calculation processing cycle.

  Here, since the A-th provisional estimated value αa based on the translational movement vector ↑ t is directly affected by the instantaneous fluctuation of the posture of the vehicle body of the vehicle 1, it is instantaneous with respect to the actual third reference angle. Although a deviation is likely to occur, a steady offset with respect to the actual third reference angle α generally does not occur.

  On the other hand, the waveform of the B-th provisional estimated value αb based on the rotation matrix R matches the actual third reference angle α relatively accurately, but has a steady offset with respect to the actual third reference angle α. Prone to occur. This offset is caused by an error in the initial value of αb or an accumulation of instantaneous errors.

  Therefore, in the present embodiment, based on the above-described properties of the A-th provisional estimated value αa and the B-th provisional estimated value αb, the third reference angle estimation unit 17 estimates the third reference angle α as described below. Are sequentially determined.

  That is, the third reference angle estimation unit 17 stores and holds the A-th provisional estimated value αa and the B-th provisional estimated value αb calculated in a period from the current time to a predetermined time before in time series. In this case, the stored A-th provisional estimated value αa and B-th provisional estimated value αb are updated each time new (latest) αa and αb are calculated.

  The third reference angle estimator 17 first stores the latest data for a predetermined time interval (from the current time to the past time before the predetermined time interval) among the time-series data of αa and αb stored and held. Data within the period of time) is extracted, and the average value of the extracted data of αa and αb is calculated. That is, the third reference angle estimation unit 17 calculates the average value αa_ave of αa and the average value αb_ave of αb in the period from the current time to the past time before a predetermined time interval. The predetermined time interval may be a fixed time, but in the present embodiment, for example, it is set according to the vehicle speed of the vehicle 1 (the higher the vehicle speed, the shorter the predetermined time interval is set).

  Next, the third reference angle estimation unit 17 calculates a deviation (= αa_ave−αb_ave) between the average value αa_ave of the A-th provisional estimated value αa and the average value αb_ave of the B-th provisional estimated value αb. Then, the third reference angle estimation unit 17 corrects the B provisional estimated value αb (the latest value of αb) at the current time based on this deviation, thereby obtaining the estimated value of the third reference angle α at the current time. decide. Specifically, the third reference angle estimation unit 17 determines the estimated value of the third reference angle α at the current time by adding the deviation to the B-th provisional estimated value αb at the current time.

  The estimated value of the third reference angle α is sequentially determined by the processing of the third reference angle estimation unit 17 described above. The A-th provisional estimation is obtained by calculating the deviation between αa and αb at each time in the period from the current time to the past time before the predetermined time interval, and obtaining the average value of the deviation within the period. The deviation (= αa_ave−αb_ave) between the average value αa_ave of the values αa and the average value αb_ave of the B-th provisional estimated value αb may be calculated.

  The estimated value of the first reference angle γ, the estimated value of the second reference angle θ, and the estimated value of the third reference angle α obtained by the first to third reference angle estimators 15 to 17 as described above are the distances, respectively. Input to the estimation unit 18. The distance estimation unit 18 calculates an estimated value of the distance D between the object 53 and the host vehicle 1 from these input values.

  Specifically, the distance estimator 18 subtracts the estimated value of the third reference angle α from the estimated value of the second reference angle θ (according to the above equation (13)), as shown in FIG. An estimated value of the angle β, that is, the angle β formed by the straight line L2 (or the object existence virtual road surface Sa) in the drawing with respect to the own vehicle existence road surface 51 is calculated. Then, the distance measurement value calculation unit 18 calculates the above equation (2) from the estimated value of the angle β, the estimated value of the first reference angle γ, the estimated value of the third reference angle α, and the value of the camera height Hc. ) To calculate the estimated value of the distance D between the object 53 and the host vehicle 1.

  The above is the detail of the process which the arithmetic processing unit 11 in this embodiment performs. By this processing, when there is an object 53 to be measured for the distance to the host vehicle 1 on the road surface in front of the vehicle 1, the distance D between the target object 53 and the host vehicle 1 is successively determined. Will be measured.

  Here, a supplementary description will be given of the correspondence between the present embodiment and the present invention. In the present embodiment, the distance measurement feature point extraction unit 13, the camera motion estimation unit 14, the first reference angle estimation unit 15, the second reference angle estimation unit 16, the third reference angle estimation unit 17, and the distance estimation unit 18 The feature point extracting means for ranging, the camera motion parameter measuring means, the first reference angle estimating means, the second reference angle estimating means, the third reference angle estimating means, and the distance estimating means according to the present invention are realized.

  Furthermore, the third reference angle A estimation unit 17a and the third reference angle B estimation unit 17b according to the present invention are realized by the third reference angle A estimation unit 17a and the third reference angle B estimation unit 17b, respectively.

  According to the embodiment described above, the degree of the gradient of the object existing road surface 52 ahead of the vehicle 1 by estimating the second reference angle θ in addition to the first reference angle γ and the third reference angle α. Can be reflected in the estimation process of the distance D between the object 53 and the host vehicle 1, and the estimated value of the distance D can be obtained by the equation (2). For this reason, the influence D of the gradient of the object existing road surface 53 with respect to the own vehicle existing road surface 51 can be appropriately compensated, and the distance D between the object 53 and the own vehicle 1 can be accurately measured.

  Moreover, since the 2nd reference angle (theta) used in order to estimate the distance D between the target object 53 and the own vehicle 1 can be estimated using the picked-up image by the single vehicle-mounted camera 2, the estimation The third reference angle θ can be estimated without requiring a dedicated sensor or the like.

  In the third reference angle α estimation process used to estimate the distance D between the object 53 and the host vehicle 1, the B-th provisional estimated value αb calculated at a predetermined calculation processing cycle is The difference (= αa_ave−αb_ave) between the average value αa_ave of the A-th provisional estimated value αa and the average value αb_ave of the B-th estimated value αb within the period from the current time to the past time before a predetermined time interval. By correcting, the final estimated value of the third reference angle α is sequentially determined.

  As a result, the estimated value of the third reference angle α is determined so that the B-th provisional estimated value αb is equal to the third reference angle α due to the initial value error of the B-th provisional estimated value αb or the accumulation of instantaneous errors. The offset generated with respect to the actual value is determined so as to be removed from the B-th provisional estimated value αb. Therefore, an estimated value that accurately matches the actual value of the third reference angle α can be obtained.

  Further, the predetermined time interval that defines the sampling periods of αa and αb used for obtaining the deviation (= αa_ave−αb_ave) is set to a shorter time as the vehicle speed increases. For this reason, the period of the predetermined time interval is set to a period in which a large variation is unlikely to occur in the posture of the vehicle 1 (particularly the posture in the pitch direction) within the period.

  As a result, the deviation (= αa_ave−αb_ave) is calculated using the average values αa_ave and αb_ave with high reliability (high S / N ratio) of the A-th provisional estimated value αa and the B-th provisional estimated value αb. be able to. For this reason, the accuracy of the estimated value of the third reference angle α can be effectively increased.

  As a result, according to the present embodiment, the distance D between the object 53 and the host vehicle 1 can be accurately estimated with an inexpensive configuration using the single vehicle-mounted camera 2.

  In the embodiment described above, the third reference angle α is sequentially estimated. However, in a situation where the vehicle 1 is traveling on a flat road surface, the value of the third reference angle α is It is maintained almost constant. The value of the third reference angle α is a known value defined by the mounting posture of the in-vehicle camera 2 with respect to the vehicle body. Therefore, in such a situation, the distance D between the object 53 and the host vehicle 1 is estimated using the value of α as a predetermined default value without sequentially estimating the third reference angle α. Also good.

  Moreover, although the said embodiment demonstrated taking the case where the target object 53 was made into the pedestrian as an example, the target object 53 may be moving bodies other than a pedestrian, for example, animals other than a pedestrian, other vehicles, etc. . Alternatively, the object 53 may be a fixed ground object (stationary object). In the case where the object 53 is a fixed ground object, in the process of the distance measurement feature point extraction unit 13, as described above, a point obtained by projecting the ground point of the object 53 onto each captured image is the feature point P1. , P2 may be used.

  DESCRIPTION OF SYMBOLS 1 ... Vehicle, 2 ... Car-mounted camera, 10 ... Distance measuring device, 13 ... Feature point extraction part for distance measurement (feature point extraction means for distance measurement), 14 ... Camera motion estimation part (camera motion parameter measurement means), 15 ... 1st reference angle estimation part (1st reference angle estimation means), 16 ... 2nd reference angle estimation part (2nd reference angle estimation means), 17 ... 3rd reference angle estimation part (3rd reference angle estimation means), 17a ... 3rd reference angle A estimation part (3rd reference angle A estimation means), 17b ... 3rd reference angle B estimation part (3rd reference angle B estimation means), 18 ... Distance estimation part (distance estimation means) ).

Claims (5)

A distance measuring device that measures a distance between an object on a road surface imaged by an in-vehicle camera and the vehicle on which the in-vehicle camera is mounted,
From two captured images captured by the in-vehicle camera at different image capturing times, the two captured images are stationary points on the object existing road surface that is the road surface on which the object exists, and the distance from the own vehicle is the two captured images. A feature point formed by projecting a stationary point that is the same as the ground contact point of the object at the imaging time of the first captured image, which is one of the two, to each of the two captured images is extracted, and the two image capturing is performed. Distance measurement feature point extracting means for specifying the position of the feature point in each of the images;
Camera motion parameter measuring means for measuring camera motion parameters representing changes in the position and orientation of the in-vehicle camera in a period between the respective imaging times of the two captured images;
Based on the position of the feature point specified in the first captured image, a value of a first reference angle that is an angle formed by a straight line from the in-vehicle camera to the stationary point with respect to the optical axis of the in-vehicle camera is estimated. First reference angle estimation means;
The own vehicle existence road surface, which is the road surface on which the own vehicle exists at the imaging time of the first captured image, the position of the feature point in each of the two captured images, the height of the in-vehicle camera, and the imaging time of the first captured image Based on an arithmetic expression representing a relationship between a second reference angle that is an angle formed with respect to the optical axis of the in-vehicle camera from a point located below the in-vehicle camera to the stationary point From the measured value of the camera motion parameter, the position of the feature point specified in each of the two captured images, and the set value of the height of the in-vehicle camera, the second reference as an unknown in the arithmetic expression Second reference angle estimation means for estimating an angle value;
Distance estimating means for estimating a distance between the object and the host vehicle from at least an estimated value of the first reference angle, an estimated value of the second reference angle, and a set value of a height of the in-vehicle camera; A distance measuring device comprising: The distance measuring device according to claim 1,
The camera motion parameter measuring means is a translation vector that is a displacement vector of the position of the in-vehicle camera in a period between the respective imaging times of the two captured images, and a rotation that represents an angle change amount of the attitude of the in-vehicle camera. A means for measuring a matrix as the camera motion parameter,
The second reference angle estimation means estimates the value of the second reference angle based on the following expressions (10a) to (10c) as the arithmetic expressions.
The distance measuring device according to claim 1,
Further comprising third reference angle estimation means for estimating a value of a third reference angle, which is an angle formed by the optical axis of the vehicle-mounted camera at the imaging time of the first captured image with respect to the own vehicle existing road surface,
The distance estimating means is configured to calculate the object based on the estimated value of the first reference angle, the estimated value of the second reference angle, the estimated value of the third reference angle, and the set value of the height of the in-vehicle camera. A distance measuring device for estimating a distance between the vehicle and the host vehicle. The distance measuring device according to claim 3,
The camera motion parameter measuring unit is a unit that measures, as a component of the camera motion parameter, a translation vector that is a displacement vector of the position of the vehicle-mounted camera in a period between the imaging times of the two captured images. ,
The distance measuring device, wherein the third reference angle estimating means estimates a value of the third reference angle based on at least a direction of the measured translational movement vector with respect to an optical axis of the in-vehicle camera. The distance measuring device according to claim 3,
The camera motion parameter measuring means is a translation vector that is a displacement vector of the position of the in-vehicle camera in a period between the respective imaging times of the two captured images, and a rotation that represents an angle change amount of the attitude of the in-vehicle camera. A means for sequentially measuring a matrix as the camera motion parameter,
Third reference angle A estimation means for successively obtaining an A provisional estimated value of the third reference angle based on the direction of the measured translational movement vector with respect to the optical axis of the in-vehicle camera, and the measured rotation Further comprising third reference angle B estimation means for successively obtaining the B provisional estimated value of the third reference angle by integrating the amount of angle change indicated by the matrix,
The third reference angle estimation means calculates a deviation between an average value of the A reference provisional value of the third reference angle for a predetermined period and an average value of the B provisional estimation value for the predetermined period, and the deviation And determining the estimated value of the third reference angle by correcting the B-th provisional estimated value in accordance with the distance measuring device.