JP2013003110A - Vehicle state detection apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve detection accuracy in a vehicle state detection apparatus for detecting a state of its own vehicle based on image data imaged by a camera 20 installed in the vehicle.SOLUTION: A control device 10 (a CPU 11) acquires a plurality of image data imaged by the camera 20, extracts a feature point of each of the plurality of image data and derives a plurality of optical flows by tracking positions of respective feature points in the image data continued with the lapse of time (S10 to S30). Optical flows having the same direction and the same size are classified as one group out of the plurality of derived optical flows, the optical flows in the group including most optical flows are selected as own-vehicle optical flows corresponding to the own vehicle and the state of the own vehicle is detected based on the selected own-vehicle optical flows (S50 to S70).

Description

  The present invention relates to a vehicle state detection device that detects the state of a host vehicle based on image data captured by an imaging device installed in the vehicle.

  Conventionally, there has been a driving support system disclosed in Patent Document 1 as an example of a technique for detecting the state of the host vehicle based on image data captured by an imaging device installed in the vehicle.

  This driving support system detects (estimates) the state of the host vehicle (vehicle speed and rotation angle) without requiring a vehicle speed sensor and a steering angle sensor. Specifically, feature points are extracted from image data obtained by an imaging device (camera) installed in the vehicle. Then, an optical flow (movement vector) is derived by performing a feature point tracking process between image data arranged in time series. And the state of the own vehicle is detected based on this optical flow.

JP 2009-17462 A

  By the way, in this driving support system, the feature points to be extracted are fixed points such as intersections or end points of white lines drawn on the road surface (that is, stationary objects (fixed or drawn on the road surface, etc.). Part of a stationary object) is assumed.

  However, not only a stationary object but also an object that moves relative to the road surface (moving object) may exist in the imaging region of the imaging device. In this case, the imaging device images a moving object together with a stationary object. In this way, when the imaging device images a moving object together with a stationary object, the driving support system extracts the stationary object (that is, a part of the stationary object) and the moving object (that is, a part of the moving object) as feature points. End up. Therefore, the driving support system not only derives an optical flow (hereinafter also referred to as an optical flow of a stationary object) with respect to a feature point extracted from a stationary object, but also applies an optical flow (hereinafter referred to as a feature point extracted from a moving object). , Also referred to as an optical flow of a moving object). That is, the optical flow derived by the driving support system includes an optical flow of a stationary object and an optical flow of a moving object. Thus, the driving support system detects the state of the host vehicle based on the optical flow of the stationary object and the optical flow of the moving object.

  Originally, the driving support system detects the state of the vehicle based on the optical flow of the feature points extracted from the stationary object, and thus based on the optical flow of the stationary object and the optical flow of the moving object. If the state of the host vehicle is detected, the detection accuracy may be reduced.

  The present invention has been made in view of the above problems, and is a vehicle state detection device that detects the state of the host vehicle based on image data captured by an imaging device installed in the vehicle, and has detection accuracy. The purpose is to improve.

In order to achieve the above object, the invention described in claim 1
A vehicle state detection device that detects the state of the host vehicle based on image data captured by an imaging device installed in the vehicle,
Image data acquisition means (S10) for acquiring a plurality of image data captured at predetermined imaging intervals Δt by the imaging device;
Feature point extraction means (S20) for extracting feature points in each of the plurality of image data;
Optical flow deriving means (S30) for deriving a plurality of optical flows by tracking the positions of feature points in temporally continuous image data;
Classification means (S50) for classifying optical flows having the same direction and size in a plurality of derived optical flows into one group;
Selecting means (S60) for selecting the optical flow of the group including the most optical flows as the own vehicle optical flow corresponding to the own vehicle;
First detection means (S70) for detecting the state of the host vehicle based on the selected host vehicle optical flow.

  As described above, the plurality of optical flows derived based on the feature points in temporally continuous image data include optical flows of feature points extracted from stationary objects that are stationary with respect to the road surface (that is, feature points). And the optical flow of the feature point extracted from the moving object moving with respect to the road surface (that is, the optical flow of the part of the moving object extracted as the feature point) May be included. The moving objects include oncoming vehicles (front vehicles traveling in the opposite lane of the lane in which the host vehicle is traveling) and preceding vehicles (front vehicles traveling in the same lane as the host vehicle). It is. On the other hand, the stationary object includes a white line drawn on the road surface, a sign fixed to the ground, a guardrail, and the like.

  The optical flow of the moving object changes in accordance with the relative relationship between the moving direction of the moving object and the traveling direction of the own vehicle, and the relative relationship between the speed of the moving object and the speed of the own vehicle. . Also, oncoming vehicles and preceding vehicles as moving objects often have different traveling directions and speeds. Accordingly, the optical flow of each moving object is often different.

  On the other hand, the optical flow of a stationary object changes with the relationship between the relative traveling direction of the host vehicle relative to the stationary object and the relative speed of the host vehicle relative to the stationary object. Therefore, the optical flow of each stationary object often has the same direction and size. In addition, the optical flow of each stationary object is often different from the optical flow of each moving object. Therefore, in the present invention, an optical flow having the largest number of optical flows having the same direction and size is regarded as an optical flow of a stationary object.

  Therefore, in the vehicle state detection device of the present invention, the optical flow of the group including the most optical flows is selected as the own vehicle optical flow corresponding to the own vehicle. That is, the optical flow of the group (cluster) including the most optical flows is selected as the vehicle speed VW (speed V (vector) + angular speed W (vector)) of the host vehicle. Thus, for convenience, the vehicle speed VW (speed V + angular speed W) of the host vehicle is referred to as the host vehicle optical flow. And since the state of the own vehicle is detected based on this own vehicle optical flow, even if both moving and stationary objects are included in the image data, it is affected by the optical flow of the moving object. And the detection accuracy can be improved.

  Further, as shown in claim 2, the first detection means (S70) determines the vehicle's own vehicle based on the amount of change in the traveling direction of the feature point included in the vehicle's optical flow and the imaging interval Δt of the image data. You may make it include the speed detection means (S71) which detects the speed V in the advancing direction.

  That is, the speed V in the traveling direction of the host vehicle may be detected as the state of the host vehicle. As described above, since the speed V in the traveling direction of the host vehicle can be detected by processing the image data, it is not necessary to use a vehicle speed sensor.

  The own vehicle optical flow is a combination of the speed V and the angular velocity W of the own vehicle. Therefore, as shown in claim 3, the first detection means (S70) may include angular speed detection means (S72) for detecting the angular speed W of the host vehicle by subtracting the speed V from the host vehicle optical flow. Good.

  That is, the angular velocity W of the host vehicle may be detected as the state of the host vehicle. As described above, since the angular velocity W of the host vehicle can be detected by processing the image data, it is not necessary to use a gyro sensor.

  The angular velocity W is a combination of a parallel component indicating the yaw angle and pitching angle of the host vehicle and a rotation component indicating the roll angle of the host vehicle.

  Therefore, as shown in claim 4, the angular velocity detection means (S 72) has two points symmetrical with respect to the center position of the imaging screen coordinates in the imaging device from the difference optical flow obtained by subtracting the velocity V from the own vehicle optical flow. Each difference optical flow at one feature point is selected, and each selected difference optical flow is added together and divided by 2 to obtain a parallel component obtained by removing the rotation component from the difference optical flow. You may make it include the yaw / pitching detection means (S721) which detects the yaw angle and pitching angle of a vehicle. That is, the yaw angle and pitching angle of the host vehicle may be detected as the state of the host vehicle.

  Further, as shown in claim 5, the angular velocity detection means (S72) includes two points symmetrical with respect to the center position of the imaging screen coordinates in the imaging device from the difference optical flow obtained by subtracting the velocity V from the own vehicle optical flow. Each difference optical flow at the feature point is selected, and a value obtained by dividing the difference between the selected difference optical flows by 2 is defined as a rotation component obtained by removing the parallel component from the difference optical flow. You may make it include the roll detection means (S722) detected as a roll angle. That is, the roll angle of the host vehicle may be detected as the state of the host vehicle.

  Further, as shown in claim 6, an optical flow larger than the own vehicle optical flow is selected from the plurality of optical flows derived by the optical flow deriving means (S30), and the own vehicle optical flow is subtracted from this optical flow. Thus, the second detection means (S60) for detecting the oncoming vehicle optical flow corresponding to the oncoming vehicle, and the first determination means (S80) for judging whether the direction of the oncoming vehicle optical flow is directed to the host vehicle. , S81).

  By doing so, it is possible to determine whether or not there is an oncoming vehicle traveling toward the host vehicle by processing the image data.

  Further, as shown in claim 7, the roll angle detected by the roll detection means (S722) is compared with a threshold value, and when the roll angle reaches the threshold value, the traveling state of the host vehicle is unstable. The second determination means (S90, S91) for determining that the traveling state of the host vehicle is stable when the roll angle has not reached the threshold value may be provided.

  By doing so, it is possible to detect whether or not the traveling state of the host vehicle is stable by processing the image data.

  Further, as shown in claim 8, the yaw angle detected by the yaw / pitching detection means (S721) is compared with a threshold value, and when the yaw angle reaches the threshold value, the traveling state of the host vehicle is unstable. If the yaw angle has not reached the threshold value, third determination means (S90, S91) for determining that the traveling state of the host vehicle is stable may be provided.

  By doing so, it is possible to detect whether or not the traveling state of the host vehicle is stable by processing the image data.

It is a block diagram which shows schematic structure of the vehicle state detection apparatus in embodiment of this invention. It is an algorithm which shows the processing operation of the vehicle state detection apparatus in embodiment of this invention. (A)-(e) is an image figure which shows the processing operation of the vehicle state detection apparatus in embodiment of this invention. (A)-(c) is an image figure which shows the state detection process operation | movement of the vehicle state detection apparatus in embodiment of this invention. It is an image figure which shows angular velocity. (A)-(e) is an image figure which shows the detection process operation | movement which detects a yaw angle and a pitching angle from angular velocity. (A)-(d) is an image figure which shows the detection process operation | movement which detects a roll angle from angular velocity. It is drawing for demonstrating the coordinate transformation from an imaging screen coordinate system to a world coordinate system. 10 is an algorithm showing a collision determination processing operation of the vehicle state detection device in Modification 1. (A)-(c) is an image figure which shows the collision determination processing operation | movement of the vehicle state detection apparatus in the modification 1. FIG. 10 is an algorithm showing a rollover determination processing operation and a skid determination processing operation of the vehicle state detection device in Modification 2.

  Hereinafter, embodiments of the present invention will be described with reference to FIGS.

  A control device 10 shown in FIG. 1 corresponds to the vehicle state detection device of the present invention. This control device 10 (vehicle state detection device) is installed in a vehicle and electrically connected to itself (control device 10) 20 (imaging device) 20 (in the following description and drawings, simply referred to as a camera) The image data is acquired from the image data, and image processing is performed using the acquired image data. That is, the camera 20 is installed in the vehicle and is electrically connected to the control device 10. And the control apparatus 10 detects the state (speed V, angular velocity W, etc.) of the own vehicle (vehicle in which it is mounted) by detecting the state (speed, angular velocity, etc.) of the camera by this image processing. To do. Thus, the vehicle state detection device detects the state (speed V, angular speed W, etc.) of the host vehicle without using a plurality of sensors (vehicle speed sensor, gyro sensor, etc.).

  In addition, the control device 10 can be configured to be connected to an in-vehicle LAN (Local Area Network) and to be able to communicate with a control device such as a first ECU (Electronic Control Unit) 30, a second ECU 40, and a third ECU 50, for example. . Then, the control device 10 may perform various vehicle controls using the first ECU 30, the second ECU 40, and the third ECU 50.

  As shown in FIG. 1, the control device 10 is mainly composed of a microcomputer including a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, an I / F (interface) 14, and the like. It is configured. In the control device 10, the CPU 11 reads out a program stored in advance in the ROM 13 or the like to the RAM 12 and executes it, thereby executing a predetermined processing operation such as acquisition of image data captured by the camera 20.

  Here, based on FIGS. 2-8, the processing operation of the control apparatus 10 (vehicle state detection apparatus) is demonstrated. The control device 10 executes the algorithm shown in FIG. 2 every predetermined time while power is supplied from a power source (not shown, for example, a battery) mounted on the host vehicle.

  First, in step S10, an image data acquisition process is executed. The control device 10 (CPU 11) acquires a plurality of image data captured by the camera 20 at a predetermined imaging interval Δt (image data acquisition means). In other words, the image data (image signal) from the camera 20 is taken in every fixed time Δt. The CPU 11 stores the image data in the RAM 12 or the like in time series. The camera 20 is mounted on the vehicle together with the control device 10 and captures a predetermined range (predetermined imaging range) such as the front of the host vehicle. That is, the camera 20 captures the front of the host vehicle at predetermined imaging intervals Δt and outputs the image data (captured image information) to the control device 10. Therefore, the image data acquired by the control device 10 is an image of a frame every certain time.

  FIG. 3A shows an example of image data. As shown in FIG. 3A, the image data includes a stationary object 60 (an object that is stationary relative to the road surface by being fixed or drawn on the road surface) and a moving object 70 (the object that moves relative to the road surface). May be mixed). That is, not only a stationary object but also a moving object may exist in the imaging region of the camera 20. In this case, the camera 20 images a moving object together with a stationary object.

  The stationary object 60 includes white lines drawn on the road surface, signs fixed to the ground, guardrails (not shown), buildings (not shown), and the like. On the other hand, the moving object 70 includes an oncoming vehicle (a forward vehicle traveling in a lane opposite to the lane in which the host vehicle is traveling) and a preceding vehicle (not shown, traveling in the same lane as the host vehicle). Vehicle).

  Next, in step S20, feature point extraction processing is executed. The control device 10 (CPU 11) extracts a feature point in each of a plurality of image data captured by the camera 20 (feature point extraction means). In other words, the control device 10 (CPU 11) extracts a feature point from each image data captured by the camera 20.

  When the stationary object 60 and the moving object 70 coexist in the image data, feature points are extracted from the stationary object 60 and the moving object 70. For example, in the case of the image data shown in FIG. 3A, feature points 81 are extracted from the stationary object 60 and feature points 82 are extracted from the moving object 70. That is, the feature point 81 is a feature point extracted from the stationary object 60, and the feature point 82 is a feature point extracted from the moving object 70.

  When extracting the feature points, a known SIFT (Scale-Invariant Feature Transform) feature amount is used for the feature points, and the feature points are featured within a specific image range so that the density of the feature points is uniform. Decide the number of quantities as one. In other words, a known SIFT is used for extracting the feature points. In addition, the image data is divided into a plurality of regions having the same area so that the density of the feature points in the image data is uniform, and one feature point is extracted from each region (that is, all of the image data is extracted). Extract feature points evenly from the range). Specifically, first, key points are detected and the scale is determined. At this time, the smoothing parameter maximum is used as a scale (scale invariant). Next, feature amount description (feature vector) is described. At this time, the feature amount is robust against a change in illumination by describing with a multidimensional feature vector. Therefore, it is possible to extract feature points that are invariant to scale changes and robust to illumination changes.

  Next, in step S30, an optical flow derivation process is executed. The control device 10 (CPU 11) derives a plurality of optical flows by tracking the positions of the feature points 81 and 82 in temporally continuous image data (optical flow deriving means). That is, the control device 10 (CPU 11) derives an optical flow from the entire image between, for example, image data at the current time (t) and image data at the previous time (t-1). With this optical flow, the motion of each point (feature points 81 and 82) in the entire image data can be acquired as a vector. As described above, when the stationary object 60 and the moving object 70 are mixed in the image data, the optical flow is obtained from the entire image data, and is extracted from the stationary object 60 as shown in FIG. The optical flow OP1 of the feature point 81 (hereinafter also referred to as the optical flow OP1 of the stationary object 60) and the optical flow OP2 of the feature point 82 extracted from the moving object 70 (hereinafter also referred to as the optical flow OP2 of the moving object 70). Can be derived respectively. In addition, when deriving an optical flow, past image data may be used further than the previous time (t−1). The optical flow is a vector representing the motion of an object (feature point) in a temporally continuous digital image.

  The optical flow OP <b> 1 of the stationary object 60 changes according to the relationship between the relative traveling direction of the host vehicle relative to the stationary object 60 and the relative speed of the host vehicle relative to the stationary object 60. On the other hand, the optical flow OP2 of the moving object 70 is a relative relationship between the traveling direction of the moving object 70 and the traveling direction of the host vehicle, and a relative relationship between the speed of the moving object 70 and the speed of the host vehicle. It will change with.

  Next, in step S40, an optical flow normalization process is executed. The control device 10 (CPU 11) obtains the distance from the own vehicle (camera) to each feature point by stereo matching (normalization means). Then, the optical flow is normalized by the distance to the feature point and the direction of the vanishing point (normalization means). For example, as shown in FIG. 3C, description will be made by adopting a1 which is one of the feature points 81 and b1 which is one of the feature points 81. The obtained distance to a1 is 2 m, and the size of the optical flow at a1 is 10. On the other hand, the obtained distance to b1 is 10 m, and the magnitude of the optical flow at b1 is 2. When the optical flow is normalized by the distance to the feature point and the direction of the vanishing point with respect to a1 and b1, the optical flow of a1 and the optical flow of b1 are both perpendicular to each other and have a magnitude of 20 It becomes. By normalizing in this way, as shown in FIG. 3D, the optical flows OP1 of feature points extracted from the stationary object 60 all have the same direction and size.

  Similarly, the normalization process is also performed on the optical flow OP2 of the feature points extracted from the moving object 70 (here, the oncoming vehicle). As described above, the optical flow OP2 of the moving object 70 is the relative relationship between the traveling direction of the moving object 70 and the traveling direction of the host vehicle, and the relative speed between the moving object 70 and the speed of the host vehicle. Changes with the relationship. Therefore, as is clear from FIG. 3D, when the optical flow OP2 of the feature points extracted from the moving object 70 (here, the oncoming vehicle) is normalized, the size becomes larger than a1 and b2. Note that the direction of the normalized optical flow OP2 may be the same as or different from a1 and b2 depending on the relative relationship between the traveling direction of the moving object 70 and the traveling direction of the host vehicle. The optical flow described in the claims of the present invention indicates the optical flow after normalization.

  Next, in step S50, clustering processing is executed. That is, the control device 10 (CPU 11) classifies optical flows having the same direction and size in a plurality of derived optical flows (after normalization) into one group (classification means). In other words, clustering is performed depending on whether both the direction and the magnitude are the same in the derived optical flows (after normalization). Therefore, the optical flows (after normalization) classified into the same group (cluster) have the same direction and size.

  Next, in step S60, a separation process between a stationary object and a moving object is executed based on the clustering result. As described above, the plurality of optical flows derived based on the feature points in temporally continuous image data include the optical flow OP1 of the stationary object 60 and the optical flow OP2 of the moving object 70. There is.

  The optical flow OP2 of the moving object 70 changes in accordance with the relative relationship between the traveling direction of the moving object 70 and the traveling direction of the host vehicle, and the relative relationship between the speed of the moving object 70 and the speed of the host vehicle. To do. Further, the oncoming vehicle and the preceding vehicle as the moving object 70 often have different traveling directions and speeds. Therefore, the optical flow OP2 of each moving object 70 is often different.

  On the other hand, the optical flow of the stationary object 60 changes with the relationship between the relative traveling direction of the host vehicle relative to the stationary object 60 and the relative speed of the host vehicle relative to the stationary object. Accordingly, the optical flow of each stationary object 60 often has the same direction and size. In addition, the optical flow of each stationary object 60 is often different from the optical flow of each moving object 70. Therefore, the optical flow with the largest number of optical flows having the same direction and size is regarded as the optical flow of the stationary object 60.

  Therefore, the control device 10 (CPU 11) selects the optical flow of the group (cluster) including the most optical flows as the own vehicle optical flow corresponding to the own vehicle (selection means). The own vehicle optical flow indicates the vehicle speed VW (speed V + angular speed W) of the own vehicle. For convenience, the vehicle speed VW (speed V + angular speed W) of the host vehicle is referred to as the host vehicle optical flow. The vehicle speed VW (speed V + angular speed W) can also be referred to as the own vehicle state, the traveling state of the own vehicle, or the momentum of the own vehicle.

  For example, as shown in FIG. 3E, the group 91 includes nine optical flows (normalized optical flows), whereas the group 92 includes one optical flow (normal). Only the optical flow after conversion). Therefore, in this example, the optical flow included in the group 91 is selected as the own vehicle optical flow. That is, the optical flow OP1 of the stationary object is selected as the own vehicle optical flow (that is, the vehicle speed VW (speed V + angular velocity W) of the own vehicle). In this way, the maximum cluster when the normalized optical flows are clustered by direction and size is selected as a stationary object.

  The optical flow OP2 of the feature points extracted from the moving object 70 includes the vehicle speed (speed V + angular speed W) of the own vehicle and the vehicle speed of the moving object 70 (another vehicle speed (another vehicle speed + another vehicle angular velocity)). ) And are synthesized. Therefore, the optical flow OP2 of the feature points extracted from the oncoming vehicle as the moving object 70 is the vehicle speed (speed V + angular speed W) of the host vehicle and the oncoming vehicle speed MV (oncoming vehicle speed V1) of the moving object (oncoming vehicle) 70. (Vector) + opposite vehicle angular velocity W1 (vector)). The oncoming vehicle speed MV (speed V1 + angular speed W1) can also be referred to as an oncoming vehicle state, an oncoming vehicle running state, or an oncoming vehicle momentum.

  Therefore, an optical flow having a size larger than that of the own vehicle optical flow selected as described above can be regarded as an optical flow of feature points extracted from the oncoming vehicle. Therefore, the control device 10 (CPU 11) selects an optical flow having a size larger than that of the own vehicle optical flow as an optical flow of feature points extracted from the moving object (oncoming vehicle) 70. For example, as shown in FIG. 3E, the optical flow included in the group 92 (optical flow after normalization) is larger than the own vehicle optical flow. Therefore, the optical flow (after normalization) included in the group 92 is selected as the optical flow of the feature points extracted from the moving object (oncoming vehicle) 70. As described above, the optical flow OP2 of the feature points extracted from the moving object (oncoming vehicle) 70 selected here is a combination of the vehicle speed of the host vehicle and the oncoming vehicle speed of the oncoming vehicle. Therefore, it is described as a composite vehicle speed VV.

  An optical flow (after normalization) having a smaller size than the own vehicle optical flow can be regarded as an optical flow of feature points extracted from the preceding vehicle. Therefore, the control device 10 (CPU 11) can also select an optical flow having a size smaller than that of the own vehicle optical flow as an optical flow of feature points extracted from the moving object (preceding vehicle) 70. In other words, the control device 10 (CPU 11) can be paraphrased as selecting an optical flow having a size different from that of the own vehicle optical flow as an optical flow of feature points extracted from the moving object (oncoming vehicle or preceding vehicle) 70.

  In this way, the optical flow of the stationary object and the optical flow of the moving object are separated from the derived optical flow.

  Next, in step S70, vehicle state detection processing is executed. The control device 10 (CPU 11) detects the state of the host vehicle based on the host vehicle optical flow (vehicle speed (speed V + angular speed W)) selected as described above (first detection means). Since the state of the host vehicle is detected based on the host vehicle optical flow, even if both moving and stationary objects are included in the image data, it can be suppressed from being affected by the optical flow of the moving object. As the state of the host vehicle, the speed V in the traveling direction of the host vehicle, the angular velocity W of the host vehicle, and the acceleration A (vector) of the host vehicle can be detected. .

  Here, the vehicle state detection process will be described with reference to FIGS. 4 to 8 in addition to FIG. 3. First, in step S71, speed V detection processing is executed. The control device 10 (CPU 11) determines the velocity V in the traveling direction of the host vehicle based on the amount of change in the traveling direction (vanishing point direction) of the feature point included in the host vehicle optical flow and the imaging interval Δt of the image data. Detect (calculate) (speed detection means). This vanishing point can be calculated by a known technique.

More specifically, the speed V in the traveling direction of the host vehicle is calculated in the world coordinate system. At this time, the speed V in the traveling direction of the host vehicle is assumed as in Expression 1 (moving only in the wheel direction).

  Since Vz = time change of the distance (Zs) from the object to be imaged, Vz = dZs / dt can be obtained. The distance (Zs) from the object to be imaged can be obtained using known stereo matching. That is, it can be obtained by triangulation using a triangle composed of the center of two cameras and a measurement point. Therefore, three-dimensional measurement is performed based on determination of corresponding points in the left and right images.

  In this way, the speed V in the traveling direction of the host vehicle can be detected by processing the image data, and therefore it is not necessary to use a vehicle speed sensor. That is, it is not necessary to use a sensor that sequentially detects the speed of the host vehicle by detecting a signal corresponding to the rotational speed of the wheel of the host vehicle.

Next, in step S72, an angular velocity W detection process is executed. Specifically, as shown in Formula 2, the angular velocity W is detected (calculated) in the world coordinate system.

  As described above, the own vehicle optical flow is a combination of the velocity V and the angular velocity W in the traveling direction of the own vehicle. Therefore, the control device 10 (CPU 11) determines the vehicle's optical flow from the own vehicle's optical flow (that is, the vehicle's vehicle speed VW (speed V + angular velocity W), the stationary object's optical flow (normalized optical flow)). By subtracting the speed V (in other words, the speed component) in the traveling direction, the angular speed W (in other words, the angular speed component) of the host vehicle is detected (angular speed detecting means). In other words, the angular velocity W (in other words, the angular velocity component) of the host vehicle is detected by taking the difference of the speed V in the traveling direction of the host vehicle from the host vehicle optical flow. For example, by subtracting the speed V in the traveling direction of the host vehicle from the vehicle speed VW (speed V + angular speed W) of the host vehicle shown in FIG. 4A, as shown in FIG. W can be detected.

  In this way, the angular velocity W of the host vehicle can be detected by processing the image data, so there is no need to use a gyro sensor. That is, it is not necessary to use a gyro sensor using a vibrator or the like.

  The angular velocity W includes a parallel component and a rotation component as shown in FIG. FIG. 5 is an image diagram of the angular velocity W. Therefore, the parallel component (the yaw angle and pitching angle of the host vehicle) detection process (S721) and the rotation component (roll angle of the host vehicle) detection process (S722) may be executed using the angular velocity W. Good. The detection processing of the parallel component (the yaw angle and the pitching angle of the host vehicle) and the rotation component (the roll angle of the host vehicle) will be described later. In the following description and drawings, the yaw angle is also referred to as Yaw, the pitching angle as Pitch, and the roll angle as Roll.

  In this way, the vehicle state detection device (control device 10) may be paraphrased as detecting (estimating) the momentum (speed V, angular velocity W (pitching angle, roll angle, yaw angle)) of the host vehicle from the host vehicle optical flow. it can. Incidentally, the momentum to be detected (estimated) is strictly the momentum of the camera 20 that has captured the image data. However, since the camera 20 is mounted (installed) on the own vehicle and the camera 20 also moves together with the own vehicle. This corresponds to the momentum of the host vehicle.

  Next, in step S72, acceleration A detection processing is executed. As shown in FIG. 4C, the control device 10 (CPU 11) obtains the acceleration by subtracting the time series data of the speed V in the traveling direction of the host vehicle. FIG. 4C shows the temporal change in the velocity V at the same feature point.

Here, the detection processing of the parallel component (the yaw angle and the pitching angle of the host vehicle) and the rotation component (the roll angle of the host vehicle) in the angular velocity W will be described with reference to FIGS. In other words, each component (Yaw, Pitch, Roll) of the angular velocity W is obtained. Note that Equation 2 described above can be expanded as Equation 3. Further, the mathematical expression 3 is coordinate-converted from the imaging screen coordinate system to the world coordinate system based on the mathematical expressions 4 and 8.

  First, in step S721, parallel component (yaw angle and pitching angle of own vehicle) detection processing is executed.

  FIG. 6A is an image view of an optical flow in which the yaw angle, pitching angle, and roll angle of the host vehicle are combined. That is, the difference optical flow is obtained by subtracting the speed V from the own vehicle optical flow.

  As shown in FIG. 6B, the control device 10 (CPU 11) sets the center position of the imaging screen coordinates in the camera 20 from the difference optical flow obtained by subtracting the speed V from the own vehicle optical flow shown in FIG. Each differential optical flow at two feature points that are point-symmetric with respect to each other is selected. Note that the two differential optical flows shown in FIG. 6B are different in direction and size.

  Then, as shown in FIG. 6C, the control device 10 (CPU 11) adds the selected differential optical flows to each other and divides them by 2, and then removes the rotational component from the differential optical flow. Ingredients. In other words, by adding the selected difference optical flows to each other and dividing by 2, the rotational component is canceled from the angular velocity W, and the parallel component of the angular velocity W is obtained. Furthermore, the control device 10 (CPU 11) detects the yaw angle of the host vehicle shown in FIG. 6 (d) and the pitching angle shown in FIG. 6 (e) by decomposing the parallel component into a horizontal component and a vertical component. (Calculate) (yaw / pitching detection means). The yaw angle and pitching angle are detected as the yaw angle and pitching angle of the world coordinate system by the above-described coordinate conversion. In this way, the yaw angle and pitching angle of the host vehicle may be detected as the state of the host vehicle.

  If there are no two feature points that are point-symmetric with respect to the center position of the imaging screen coordinates, the feature point is searched by changing the angle and radius R of the candidate point. Further, when there is no feature point that is point-symmetric even if the candidate point is changed, the angle of view and orientation of the camera 20 may be changed to start the algorithm shown in FIG. In this case, a changing means for changing the angle of view and orientation of the camera 20 is required.

  Next, in step S722, detection processing of a rotation component (roll angle of the host vehicle) is executed.

  FIG. 7A is an image view of an optical flow in which the yaw angle, pitching angle, and roll angle of the host vehicle are combined. That is, the difference optical flow is obtained by subtracting the speed V from the own vehicle optical flow.

  As shown in FIG. 7B, the control device 10 (CPU 11) sets the center position of the imaging screen coordinates in the camera 20 from the difference optical flow obtained by subtracting the speed V from the own vehicle optical flow shown in FIG. Each differential optical flow at two feature points that are point-symmetric with respect to each other is selected. Note that the two differential optical flows shown in FIG. 7B are different in direction and size.

  Then, as shown in FIG. 7C, the control device 10 (CPU 11) obtains a rotation component obtained by removing the parallel component from the difference optical flow by dividing the difference between the selected difference optical flows by 2. And In other words, by subtracting each selected difference optical flow and dividing by 2, the parallel component is canceled from the angular velocity W and the rotational component of the angular velocity W is obtained as shown in FIG. That is, the control device 10 (CPU 11) detects (calculates) this rotational component as the roll angle of the host vehicle (roll detection means). This roll angle is detected as a roll angle in the world coordinate system by the above-described coordinate transformation. Thus, the roll angle of the host vehicle may be detected as the state of the host vehicle.

  If there are no two feature points that are point-symmetric with respect to the center position of the imaging screen coordinates, the feature point is searched by changing the angle and radius R of the candidate point. Further, when there is no feature point that is point-symmetric even if the candidate point is changed, the angle of view and orientation of the camera 20 may be changed to start the algorithm shown in FIG. In this case, a changing means for changing the angle of view and orientation of the camera 20 is required.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not restrict | limited to the embodiment mentioned above at all, and various deformation | transformation are possible in the range which does not deviate from the meaning of this invention.

(Modification 1)
In the above-described embodiment, the example in which the state of the host vehicle is detected based on the host vehicle optical flow (vehicle speed (speed V + angular speed W)) is adopted, but the present invention is not limited to this. As described above, using the optical flow OP2 (synthetic vehicle speed VV) of the feature points extracted from the oncoming vehicle, it is determined whether or not there is an oncoming vehicle traveling toward the host vehicle. Alternatively, the control device 10 (CPU 11) according to the first modification executes steps S10 to S60 shown in Fig. 2 and then executes steps S80 to S82 shown in Fig. 9. Then, step S70 shown in Fig. 2 is executed, so that detailed description of the processing of steps S10 to S70 is omitted here.

  First, in step S60 of FIG. 9 (similar to step S60 of FIG. 2), as described above, the control device 10 (CPU 11) determines the optical flow of the host vehicle (after normalization) from a plurality of derived optical flows (after normalization). An optical flow (after normalization) having a larger size than (after normalization) is selected (second detection means). The optical flow OP2 (synthetic vehicle speed VV) of the feature points extracted from the oncoming vehicle as the moving object 70 is the vehicle speed (speed V + angular speed W) of the host vehicle and the oncoming vehicle speed MV ( The oncoming vehicle speed V1 (vector) + the oncoming vehicle angular speed W1 (vector)) is combined. Therefore, here, this optical flow OP2 (synthetic vehicle speed VV) is selected. And the control apparatus 10 (CPU11) respond | corresponds to an oncoming vehicle by subtracting the own vehicle optical flow (vehicle speed (speed V + angular velocity W) of the own vehicle) from the selected optical flow OP2 (synthetic vehicle speed VV). An oncoming vehicle optical flow is detected (second detection means). That is, as shown in FIG. 10A, the oncoming vehicle speed MV (oncoming vehicle speed V1 + oncoming vehicle angular speed W1) is detected as the oncoming vehicle optical flow. This oncoming vehicle optical flow indicates the oncoming vehicle speed MV (oncoming vehicle speed V1 + oncoming vehicle angular speed W1). For convenience, the oncoming vehicle speed MV (oncoming vehicle speed V1 + oncoming vehicle angular speed W1) is referred to as an oncoming vehicle optical flow.

  Next, in step S80 of FIG. 9, oncoming vehicle state monitoring processing is executed. At this time, the control device 10 (CPU 11) monitors the oncoming vehicle optical flow (oncoming vehicle speed MV). Next, in step S81 of FIG. 9, it is determined whether or not the moving object 70 (oncoming vehicle) is traveling in the direction of the host vehicle (that is, the camera 20 mounted on the host vehicle) (first determination unit). At this time, the control device 10 (CPU 11) determines whether or not the moving object 70 is traveling in the direction of the camera 20 depending on whether or not the direction of the oncoming vehicle optical flow is facing the camera 20.

  And when it determines with the direction of an oncoming vehicle optical flow not facing the camera 20, it considers that the moving body 70 is not advancing to the camera 20 direction, and progresses to step S70. For example, as shown in FIG. 10B, the direction of the oncoming vehicle optical flow is monitored in the order of the oldest timing T1, the next timing T2, and the latest timing T3. When it is determined that the moving object 70 is not suitable for the camera 20, it is considered that the moving object 70 does not travel in the direction of the camera 20.

  On the other hand, if it is determined that the direction of the oncoming vehicle optical flow is facing the camera 20, the moving object 70 is regarded as traveling in the direction of the camera 20, and the process proceeds to step S82. For example, as shown in FIG. 10C, the direction of the oncoming vehicle optical flow is monitored in the order of the oldest timing T1, the next timing T2, and the latest timing T3. If it is determined that the moving object 70 is moving in the direction of the camera 20, the moving object 70 is considered to be traveling in the direction of the camera 20. By doing so, it is possible to determine whether or not there is an oncoming vehicle traveling toward the host vehicle by processing the image data.

  In step S82 of FIG. 9, a collision determination process is executed. At this time, the control device 10 (CPU 11) performs a collision determination based on the direction and magnitude of the oncoming vehicle optical flow. Here, when the direction of the oncoming vehicle optical flow is in the direction of the camera 20 and the size of the oncoming vehicle optical flow is larger than a predetermined value, there is a risk of a collision with the oncoming vehicle that is traveling toward the host vehicle. Is considered high. In addition, when the direction of the oncoming vehicle optical flow is directed toward the camera 20 and the size of the oncoming vehicle optical flow is smaller than a predetermined value, there is a risk of a collision with the oncoming vehicle traveling toward the host vehicle. Consider low.

(Modification 2)
As a second modification, it is possible to detect whether or not the traveling state of the host vehicle is stable using the optical flow OP2 (synthetic vehicle speed VV) of the feature points extracted from the oncoming vehicle. Good. Note that the control device 10 (CPU 11) in the second modification executes steps S90 to S93 shown in FIG. 11 after executing the processes of steps S10 to S70 shown in FIG. Therefore, description of the processing in steps S10 to S70 is omitted here.

  First, when the vehicle state detection process of step S70 shown in FIG. 2 is completed, the monitoring process of the angular velocity W is executed as shown in step S90 of FIG. At this time, the control device 10 (CPU 11) monitors the roll angle and yaw angle detected in step S70.

  In step S91 in FIG. 11, threshold determination is executed. At this time, the control device 10 (CPU 11) determines whether or not the roll angle has reached a threshold value (roll threshold value) and whether or not the yaw angle has reached a threshold value (yaw threshold value). Then, the control device 10 (CPU 11) compares the roll angle with the roll threshold value, determines that the traveling state of the host vehicle is unstable when the roll angle reaches the roll threshold value, and proceeds to step S92 ( Second determination means). Similarly, control device 10 (CPU 11) compares the yaw angle with the yaw threshold, determines that the traveling state of the host vehicle is unstable when the yaw angle reaches the yaw threshold, and proceeds to step S93. (Third determination means). Note that the control device 10 (CPU 11) determines that the roll angle has not reached the roll threshold and returns to step S10 if it is determined that the yaw angle has not reached the yaw threshold. By doing so, it is possible to detect (determine) whether or not the traveling state of the host vehicle is stable by processing the image data.

  In step S92, rollover determination processing may be executed. That is, when the roll angle reaches the roll threshold, the control device 10 (CPU 11) may consider that the traveling state of the host vehicle is unstable and may roll over. In this case, the control device 10 (CPU 11) may alert the user via other ECUs (first ECU 30, second ECU 40, third ECU 50).

  In step S93, VSC (Vehicle Stability Control) control determination may be executed. That is, when the yaw angle reaches the yaw threshold, the control device 10 (CPU 11) determines that the traveling state of the host vehicle is unstable and may skid, and determines that the execution of the VSC control is started. May be. That is, in this case, the control device 10 (CPU 11) may instruct the other ECUs (first ECU 30, second ECU 40, third ECU 50) to start VSC control. Note that VSC control automatically controls the brakes and engine output of each wheel when it detects a sudden steering to avoid an obstacle or a side slip of the host vehicle that occurs during cornering on a slippery road surface. It is a system that suppresses unstable behavior.

  DESCRIPTION OF SYMBOLS 10 Control apparatus, 11 CPU, 12 RAM, 13 ROM, 14 I / F, 20 Camera, 30 1st ECU, 40 2ECU, 50 3ECU

Claims (8)

A vehicle state detection device that detects the state of the host vehicle based on image data captured by an imaging device installed in the vehicle,
Image data acquisition means (S10) for acquiring a plurality of the image data captured at predetermined imaging intervals Δt by the imaging device;
Feature point extraction means (S20) for extracting feature points in each of the plurality of image data;
Optical flow deriving means (S30) for deriving a plurality of optical flows by tracking the positions of the feature points in the temporally continuous image data;
In the plurality of derived optical flows, classification means (S50) for classifying optical flows having the same direction and size into one group;
Selecting means (S60) for selecting the optical flow of the group including the most optical flows as the own vehicle optical flow corresponding to the own vehicle;
First detection means (S70) for detecting the state of the host vehicle based on the selected host vehicle optical flow;
A vehicle state detection device comprising:   The first detection means (S70) detects the speed in the traveling direction of the host vehicle based on the amount of change in the traveling direction of the feature point included in the host vehicle optical flow and the imaging interval Δt of the image data. The vehicle state detection device according to claim 1, further comprising speed detection means (S71).   The said 1st detection means (S70) includes the angular velocity detection means (S72) which detects the angular velocity of the own vehicle by subtracting the said speed from the said own vehicle optical flow. Vehicle state detection device. The angular velocity detection means (S72)
From the difference optical flow obtained by subtracting the speed from the own vehicle optical flow, select each difference optical flow at two feature points that are point-symmetric with respect to the center position of the imaging screen coordinates in the imaging device,
A value obtained by adding the respective differential optical flows to each other and dividing by 2 is defined as a parallel component obtained by removing the rotation component from the differential optical flow, and the yaw angle and pitching angle of the host vehicle are detected from the parallel component The vehicle state detection device according to claim 3, further comprising pitching detection means (S721). The angular velocity detection means (S72)
From the difference optical flow obtained by subtracting the speed from the own vehicle optical flow, select each difference optical flow at two feature points that are point-symmetric with respect to the center position of the imaging screen coordinates in the imaging device,
Roll detection means for detecting a value obtained by dividing the difference between the selected difference optical flows by 2 as a rotation component obtained by removing a parallel component from the difference optical flow, and detecting the rotation component as a roll angle of the host vehicle (S722). The vehicle state detection device according to claim 3, comprising: By selecting an optical flow larger than the own vehicle optical flow from the plurality of optical flows derived by the optical flow deriving means (S30), and subtracting the own vehicle optical flow from the optical flow, Second detection means (S60) for detecting a corresponding oncoming vehicle optical flow;
First determination means (S80, S81) for determining whether the direction of the oncoming vehicle optical flow is toward the host vehicle;
The vehicle state detection device according to any one of claims 2 to 5, wherein the vehicle state detection device includes:   The roll angle detected by the roll detection means (S722) is compared with a threshold, and when the roll angle reaches the threshold, it is determined that the traveling state of the host vehicle is unstable, and the roll angle is The vehicle state detection device according to claim 5, further comprising second determination means (S90, S91) for determining that the traveling state of the host vehicle is stable when the threshold value is not reached.   The yaw angle detected by the yaw / pitching detection means (S721) is compared with a threshold value, and when the yaw angle reaches the threshold value, it is determined that the traveling state of the host vehicle is unstable, and the yaw angle The vehicle state detection device according to claim 4, further comprising third determination means (S90, S91) for determining that the traveling state of the host vehicle is stable when the angle does not reach the threshold value.

Images