JP2013097390A - Own vehicle travel estimation method and own vehicle travel estimation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an own vehicle travel estimation method which can accurately estimate an own vehicle travel parameter even when a large object exists in an image, and an own vehicle travel estimation program.SOLUTION: An own vehicle travel estimation method concerning one embodiment of this invention is a method having the steps of: photographing the surroundings of a vehicle with a camera equipped on the vehicle in a prescribed frame period; detecting motion vectors among a plurality of images taken by the camera; obtaining travel parameters of an own vehicle from the detected motion vectors; newly detecting motion vectors by using images taken by the camera after taking the plurality of images used for obtaining the travel parameters of the own vehicle; detecting motion vectors which are within a prescribed threshold value when compared with travel vectors of the own vehicle corresponding to the travel parameters of the own vehicle, as background vectors from among the newly detected motion vectors; and newly obtaining travel parameters of the own vehicle by using the background vectors.

Description

  Embodiments described herein relate generally to a host vehicle movement estimation method and a host vehicle movement estimation program.

  Recently, as a technique for supporting driving of a vehicle such as an automobile, for example, there is a technique for determining the possibility of collision between a vehicle and an object based on an image captured by an in-vehicle camera. In this type of technology, in order to extract an object from an image captured by an in-vehicle camera, information on a translation component and a rotation component of the own vehicle (hereinafter referred to as an own vehicle movement parameter) is required.

  As a method for estimating the own vehicle movement parameter, for example, a method is known in which a motion vector is detected from an image captured by an in-vehicle camera and is estimated from the detected motion vector. In this method, the subject in the image captured by the in-vehicle camera is predominantly a stationary object, and the motion vector detected from the image is the motion vector of the stationary object. The vehicle movement parameter is estimated.

Japanese Patent Application No. 2008-203992

  However, when a large object crosses the image captured by the in-vehicle camera, the detected motion vector has many directions that are unrelated to the direction of movement of the vehicle, and the vehicle movement parameter is estimated. Becomes difficult. In addition, if there is a large object near the vehicle in the image captured by the in-vehicle camera, the detection error of the position of this object greatly affects the accuracy of the own vehicle movement parameter, and it is difficult to estimate the own vehicle movement parameter. turn into.

  In order to solve the above-described problem, a method for estimating movement of a host vehicle according to an embodiment of the present invention includes a step in which a camera provided in a vehicle images the surroundings of the vehicle at a predetermined frame period, and an image is captured by the camera. Detecting a motion vector between the plurality of images obtained, obtaining a movement parameter of the own vehicle from the detected motion vector, and capturing the plurality of images used when obtaining the movement parameter of the own vehicle A step of detecting a new motion vector using an image captured by the camera after the comparison with a movement vector of the own vehicle corresponding to the movement parameter of the own vehicle among the newly detected motion vectors The step of detecting a vector within the predetermined threshold as a background vector, and using this background vector, a new movement parameter of the host vehicle is obtained. A method is a method having.

The block diagram which shows an example of the own vehicle movement estimation apparatus for enforcing the own vehicle movement estimation method which concerns on one Embodiment of this invention. The flowchart which shows the procedure at the time of calculating | requiring the initial own vehicle movement parameter by CPU of the main-control part shown in FIG. Explanatory drawing which shows an example of the relationship between a coordinate system and a rotation direction. Explanatory drawing which shows an example of the relationship between the position z of the feature point in space, and a road surface. The flowchart which shows the procedure at the time of estimating the own vehicle movement parameter correctly, even if a big object exists in an image by CPU of the main-control part shown in FIG. FIG. 6 is a subroutine flowchart illustrating an example of a background vector determination process executed by a background vector detection unit in S15 of FIG. (A) is explanatory drawing which shows the example which each projects the left 1/4 visual field and right 1/4 visual field of a fish-eye camera on the left screen and right screen which are arrange | positioned in parallel with yz plane, (b) is fish FIG. 5 is an explanatory diagram showing an example in which the left 1/4 field of view and the right 1/4 field of view of an eye camera are respectively projected onto a left screen and a right screen arranged in parallel with the y axis and having a predetermined inclination with respect to the yz plane; (c) is explanatory drawing which shows the example which each projects the left 1/4 visual field and right 1/4 visual field of a fish-eye camera on the left screen and right screen respectively arrange | positioned in parallel with xz plane. The figure for demonstrating the calculation method of the motion vector in the case shown to Fig.7 (a).

  Embodiments of a host vehicle movement estimation method and a host vehicle movement estimation program according to the present invention will be described with reference to the accompanying drawings.

  FIG. 1 is a block diagram showing an example of a host vehicle movement estimation apparatus 10 for implementing the host vehicle movement estimation method according to an embodiment of the present invention.

  The own vehicle movement estimation device 10 includes a camera 11, a main control unit 12, a storage unit 13, a lighting device 16, a horn 17, a speaker 18, and a display device 19.

  The camera 11 is composed of a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The camera 11 captures a video around a vehicle such as a private car and generates image data to be supplied to the main controller 12. .

  For example, when monitoring the rear, the camera 11 is disposed slightly downward from a line parallel to the road surface in the vicinity of the license plate at the rear of the vehicle. A wide-angle lens or a fisheye lens may be attached to the camera 11 so that a wider range of outside-vehicle images can be acquired. When monitoring the side of the vehicle, the camera 11 is disposed near the side mirror. In addition, a wide range of vehicle surrounding images may be captured by using a plurality of cameras 11.

  The main control unit 12 is configured by a microcontroller including a CPU, a RAM, and a ROM, for example. The CPU of the main control unit 12 loads the own vehicle movement estimation program stored in a storage medium such as a ROM and data necessary for executing the program into the RAM, and a large object exists in the image according to the program. However, the processing for accurately estimating the own vehicle movement parameter is executed. The RAM of the main control unit 12 provides a work area for temporarily storing programs and data executed by the CPU. The storage medium such as the ROM of the main control unit 12 stores the own vehicle movement estimation program and various data necessary for executing these programs.

  A storage medium such as a ROM has a configuration including a recording medium readable by a CPU, such as a magnetic or optical recording medium or a semiconductor memory, and a part of programs and data in the storage medium. Or you may comprise so that all may be downloaded via an electronic network via the network connection part which is not shown in figure.

  In this case, the network connection unit implements various information communication protocols according to the network form, and the main control unit 12 and electric devices such as ECUs of other vehicles are connected via the electronic network according to the various protocols. Connect. For this connection, an electrical connection via an electronic network can be applied. Here, an electronic network means an information communication network using telecommunications technology in general, in addition to a wireless / wired LAN (Local Area Network) and the Internet network, a telephone communication line network, an optical fiber communication network, a cable communication network, Includes satellite communications networks.

  The storage unit 13 is a non-volatile memory in which data can be read and written by the main control unit 12, and stores in advance the vehicle movement parameters up to the past frame, the angle threshold value ThA, the difference threshold value ThS, and the ratio threshold value ThR. . A newly obtained background vector is also stored here. These pieces of information may be updated via an electronic network or a portable storage medium such as an optical disk.

  The lighting device 16 is configured by a general headlight, and is controlled by the main control unit 12 to blink (so-called passing), for example, to warn the outside of the host vehicle.

  The horn 17 is controlled by the main control unit 12 and outputs a warning sound to the outside of the host vehicle.

  The speaker 18 is provided in the host vehicle and is controlled by the main control unit 12 to respond to various information such as a beep and information for notifying the driver of the host vehicle that a danger is imminent. Is output.

  The display device 19 is provided at a position visible to the driver, and a display output device such as a general vehicle-mounted display, a car navigation system, or a HUD (head-up display) can be used. In accordance with the control, various information such as a captured image of the camera 11 and information for notifying that the danger is imminent are displayed.

  Next, the configuration of the function realization unit by the CPU of the main control unit 12 will be described.

  As shown in FIG. 1, the CPU of the main control unit 12 executes at least a plane image generation unit 21, a motion vector detection unit 22, a background vector detection unit 23, a host vehicle movement calculation unit 24, and an object vector according to the host vehicle movement estimation program. It functions as the processing unit 25. Each unit 21-25 uses a required work area in the RAM as a temporary storage location for data. In addition, you may comprise these function implementation parts by hardware logics, such as a circuit, without using CPU.

  The planar image generation unit 21 generates a planar image based on the image captured by the camera 11. For example, when a fisheye lens is attached to the camera 11, the planar image generation unit 21 normalizes the fisheye image captured by the camera 11 into a normal two-dimensional image, and based on the normalized image in advance. A plane image of a predetermined viewpoint is generated.

  The motion vector detection unit 22 calculates a motion vector for each pixel from a plurality of plane images generated by the plane image generation unit 21 using a block matching method, a gradient method, or the like.

  The background vector detection unit 23 calculates the movement vector of the own vehicle based on the movement parameters (the translation component R and the rotation component T of the own vehicle) calculated by the own vehicle movement calculation unit 24. Then, the movement vector of the own vehicle is compared with the motion vector detected by the motion vector detecting unit 22 from the frame image taken after the frame used for calculating the movement parameter of the own vehicle. A motion vector within the threshold is detected as a background vector.

  The background vector detection unit 23 can use the angle threshold ThA, the difference threshold ThS, and the ratio threshold ThR stored in the storage unit 13 as the predetermined threshold.

  The own vehicle movement calculation unit 24 first calculates an initial own vehicle movement parameter based on the motion vector. Alternatively, it is determined that the vehicle is stopped in the initial state, and the own vehicle movement parameter is set to 0 or a small parameter. Next, when a background vector is detected based on the initial vehicle movement parameter by the background vector detection unit 23, a new vehicle movement parameter is calculated based on the detected background vector. Then, this newly calculated parameter is given to the background vector detection unit 23.

  The object vector processing unit 25 treats a motion vector that has not been detected as a background vector by the background vector detection unit 23 as an object vector, and the lighting device 16, the horn 17, the speaker according to the approaching state of the object obtained from the object vector A warning is given to the driver via at least one of 18 and the display device 19.

  Next, an example of operation | movement of the own vehicle movement estimation apparatus 10 which concerns on this embodiment is demonstrated.

  FIG. 2 is a flowchart showing a procedure for obtaining an initial own vehicle movement parameter by the CPU of the main control unit 12 shown in FIG. In FIG. 2, a symbol with a number added to S indicates each step of the flowchart.

  First, in step S <b> 1, the camera 11 captures an image around the vehicle including at least a road surface, generates an image signal, and supplies the image signal to the planar image generation unit 21.

  Next, in step S <b> 2, the planar image generation unit 21 generates a planar image based on the image captured by the camera 11.

  Next, in step S <b> 3, the motion vector detection unit 22 calculates a motion vector for each pixel from the plurality of planar images generated by the planar image generation unit 21.

  Next, in step S4, the own vehicle movement calculation unit 24 calculates initial own vehicle movement parameters based on the motion vector calculated in step S2.

  The initial own vehicle movement parameter can be obtained by the above procedure.

  Here, a method of calculating the automatic movement vector based on the own vehicle movement parameter by the background vector detection unit 23 will be described.

  FIG. 3 is an explanatory diagram illustrating an example of the relationship between the coordinate system and the rotation direction. In the following description, as shown in FIG. 3, an example of using a coordinate system in which the traveling direction is the z axis, the road surface normal direction is the y axis, and the direction orthogonal to the traveling direction and the road surface normal direction is the x axis. Show about. At this time, the advancing direction may be shifted from the z axis so that the horizontal line can be seen right beside. Also, the x, y, and z components of the motion vector are represented as u, v, and w, respectively. In FIG. 3, the rotations about the x-axis, y-axis, and z-axis are shown as Rx, Ry, and Rz, respectively.

  The image captured by the camera 11 is developed on the camera projection plane 31 ideally represented by z = f by the planar image generation unit 21. Here, f is the focal length of the camera 11.

  The rotation direction and translation direction of the screen when the vehicle is moving are assumed to be R (Rx, Ry, Rz) and T (Tx, Ty, Tz), respectively. At this time, the rotation direction and translation direction of the own vehicle movement are (−Rx, −Ry, −Rz) and (−Tx, −Ty, −Tz).

According to a known equation, the own vehicle movement vector (x, y, cu, cv) = (x coordinate position, y coordinate position, x direction vector, y direction vector) can be obtained from the own vehicle movement parameter as follows. it can.
wu =-(x ・ y / f) ・ Rx + ((f ・ f + x ・ x) / f) ・ Ry-y ・ Rz
wv =-((f ・ f + y ・ y) / f) ・ Rx + (x ・ y / f) ・ Ry + x ・ Rz
tu = (f ・ Tx-x ・ Tz) / z
tv = (f ・ Ty-y ・ Tz) / z
cu = tu + wu
cv = tv + wv

  Here, tu and tv represent translation components, and wu and wv represent rotation components, respectively. At this time, z is limited to a value (Zmax) outside the ground (for example, assuming that all feature points are at Zmax), and the ground is made smaller than that value. When the convergence point FOE (FOEx, FOEy) is used, there is no inclination of the horizontal line for simplifying the calculation, FOEx = 0, and FOEy is calculated by allowing the shift to increase the resolution in the ground direction to some extent. Therefore, it is preferable that the planar image generation unit 21 generates a planar image so that the horizontal line is directly beside and the convergence point is at the position of x = 0.

  FIG. 4 is an explanatory diagram showing an example of the relationship between the position z of the feature point in space and the road surface.

  If the background is too far away, the motion vector is very small even with translation. However, in other cases, the motion vector of the feature point has a certain size. As shown in FIG. 4, when the distance from the camera to the road surface is Dgnd, for example, when 10 times or more of the distance to Dgnd is regarded as the background, Dgnd = Zmax / 10.

The z coordinate of the motion vector detected from the image can be calculated as follows.
z = (Dgnd ・ f / cos (θgnd)) / (y-FOEy) (z is limited by Zmax)
θgnd = atan (FOEy / f)

  FIG. 5 is a flowchart showing a procedure when the CPU of the main control unit 12 shown in FIG. 1 accurately estimates the own vehicle movement parameter even when a large object exists in the image. In FIG. 5, reference numerals with numbers added to S indicate steps in the flowchart.

  In step S11, the background vector detection unit 23 calculates the own vehicle movement vector based on the own vehicle movement parameter. As the own vehicle movement parameter, for example, the initial own vehicle movement parameter calculated in step S4 of FIG. 2 can be used.

  Next, in step S <b> 12, the camera 11 captures an image around the vehicle including at least the road surface, generates an image signal, and supplies the image signal to the planar image generation unit 21. The image captured at this time is taken after the image used for calculating the initial movement parameter. In other words, the movement parameter of the own vehicle used in the subsequent step is calculated based on a frame that is earlier than the frame captured in step S12.

  Next, in step S13, the planar image generation unit 21 generates a planar image based on the image captured by the camera 11 in step S12.

  Next, in step S <b> 14, the motion vector detection unit 22 calculates a motion vector for each pixel from the plurality of planar images generated by the planar image generation unit 21.

  Next, in step S15, the background vector detection unit 23 calculates the movement vector of the own vehicle from the own vehicle movement parameter calculated by the own vehicle movement calculation unit 24 based on the past frame. Then, the background vector detection unit 23 detects the movement vector of the own vehicle and the motion vector detected by the motion vector detection unit 22 from the frame image taken after the frame used for calculating the movement parameter of the own vehicle. , And a motion vector within a predetermined threshold is detected as a background vector.

  More specifically, the own vehicle movement vector (cu, cv) = (x direction vector, y direction vector) calculated in step S11 and the motion vector (pu, pv) detected from the image in step S14 = ( x direction vector and y direction vector) and determine whether or not the motion vector is within a predetermined threshold, that is, whether or not to detect as a background vector.

  Note that (pu-wu, pv-wv) and (tu, tv), which are vectors in a state where the rotation component is canceled, may be compared.

  The motion vector determined to be the background vector in step S15 is stored in the storage unit 13 by the background vector detection unit 23 (step S16).

  On the other hand, the motion vector determined not to be the background vector in step S15 is given to the object vector processing unit 25 as an object vector. The object vector processing unit 25 performs a process of detecting an object from the object vector (step S17), and at least one of the lighting device 16, the horn 17, the speaker 18, and the display device 19 according to the approaching state of the obtained object. A warning is given to the driver via (step S18). Note that the vicinity of the threshold value of the background vector and the object vector may be discarded instead of either.

  Next, in step S19, the background vector detection unit 23 determines whether or not all the motion vectors have been detected. If there is an undetected motion vector, the process returns to step S14, and if all have been detected, the process proceeds to step S20.

  In step S <b> 20, the own vehicle movement calculation unit 24 calculates the own vehicle movement parameter using the background vector stored in the storage unit 13. Details of the calculation method of the own vehicle movement parameter using the background vector will be described later.

  According to the above procedure, the own vehicle movement parameter can be accurately estimated even when a large object exists in the image.

  The own vehicle movement estimation apparatus 10 according to the present embodiment calculates the own vehicle movement parameter using only the feature points that are located at some distance away or the background vectors of the feature points on the ground. For this reason, for example, even when there are large objects on the screen or when large objects cross the screen, the motion vectors of these objects are not detected as background vectors, and the own vehicle movement parameter can be calculated accurately. it can. In addition, since only the background vector is used for the calculation, the calculation is very light and input information other than the image is not required.

  Next, a method for determining whether or not to detect a motion vector detected from an image as a background vector will be described in more detail.

  FIG. 6 is a subroutine flowchart showing an example of the procedure of the background vector determination process executed by the background vector detection unit 23 in S15 of FIG.

  By comparing the motion vector detected from the image with the own vehicle movement vector, the background vector detection unit 23 sets the background vector for calculating the own vehicle movement parameter used in the next frame if it is a close motion vector. As the order of determination, for example, comparison is made by angle, comparison by difference, and comparison by ratio. It should be noted that when processing starts, it usually starts from where it stopped, but it may not be the case. Therefore, for a short time, the condition may be changed so that the background vector is likely to be obtained, or an object may not be detected.

  In step S <b> 31, the background vector detection unit 23 determines whether the angle difference between the motion vector detected from the image and the own vehicle movement vector is equal to or smaller than the angle threshold ThA stored in the storage unit 13 in advance.

The angle comparison method performs angle comparison using the detected motion vector as it is and angle comparison using a vector in which the rotation component is canceled. That is, the following two vector angle comparisons are performed.
(pu, pv) and (tu + wu, tv + wv)
(pu-wu, pv-wv) and (tu, tv)

  In the vicinity of the FOE, the direction of the vector is deviated due to an error in the own vehicle movement parameter. Therefore, when the angle is deviated in either angle comparison, the vector is determined as the object vector instead of the background vector (NO in step S31). As a result, it is possible to prevent a motion vector that is not suitable as a background vector from being detected. Further, when the motion vector detection accuracy is high, it may be detected as an object vector only when one of the angles is shifted.

  Note that if the motion vector is small, the influence of noise increases and the direction of the motion vector varies. On the other hand, if it is large to some extent, the direction is less susceptible to noise. Therefore, the angle comparison is preferably performed only when the magnitude of the motion vector is equal to or larger than a predetermined magnitude.

  If there is an angle difference larger than the angle threshold ThA, the process proceeds to step S32, and the object vector is determined and the process proceeds to step S17 in FIG. On the other hand, when the angle difference is equal to or smaller than the angle threshold value ThA, the process proceeds to step S33.

  Next, in step S33, the background vector detection unit 23 determines whether or not the difference between the magnitude of the motion vector detected from the image and the own vehicle movement vector is equal to or less than the difference threshold ThS stored in the storage unit 13 in advance. Determine.

  Also in the size comparison, the comparison may be performed in a state where the rotation component is canceled. In order to simplify the calculation of the difference in size, the calculation may be performed using only the larger one of the vehicle movement vectors of u (vector in the x direction) or v (vector in the y direction).

  If the magnitude difference is equal to or smaller than the threshold ThS, the process proceeds to step S34, and the motion vector is determined as the background vector, and the process proceeds to step S16 in FIG. On the other hand, if the difference in size is larger than the difference threshold ThS, the process proceeds to step S35 to determine the ratio.

  In order to prevent erroneous detection of the background vector, the difference threshold ThS may be weighted as follows.

First, when the own vehicle movement parameter is large, the amount of change in the own vehicle movement parameter increases due to vibration or the like. For this reason, when the own vehicle movement parameter is large, the difference threshold value is increased. At this time, the maximum value may be limited.
s1 = ThS ・ | R | ・ | T | / (| Rs | ・ | Ts |)
(If s1 is smaller than ThS, s1 = ThS)
| R |: Rotation component size of own vehicle movement parameter
| T |: Size of translation component of own vehicle movement parameter
| Rs |: The magnitude of the rotation component of the own vehicle movement parameter
| Ts |: Magnitude of translation component of own vehicle movement parameter

In addition, the distance to the detected motion vector becomes longer as it goes outside the center of the screen. For this reason, the influence of noise is more likely to be amplified, and the motion vector detection accuracy is deteriorated. Therefore, the difference threshold is increased as the distance increases.
s2 = s1 ・ (√ (f ・ f + x ・ x + y ・ y)) / f

Furthermore, it becomes a road surface as it goes to the bottom of the screen, and approaches the own vehicle. For this reason, the value of z becomes small and the influence of the own vehicle movement vector becomes large. Therefore, the degree of influence when the own vehicle movement parameter is shifted increases. Therefore, the difference threshold is increased as z is closer.
s3 = s2 ・ Zmax / z

  Further, this multiplication may be further multiplied to make it easier to become a background vector even if there is an error.

  For a motion vector that does not become an object vector in the angle comparison and does not become a background vector in the size difference comparison, in step S35, the background vector detection unit 23 moves the motion vector detected from the image and the own vehicle movement. It is determined whether or not the vector ratio is less than or equal to a ratio threshold ThR stored in the storage unit 13 in advance. If the difference from 1 in the ratio is equal to or less than the ratio threshold ThR, the process proceeds to step S34, the motion vector is determined as the background vector, and the process proceeds to step S16 in FIG. On the other hand, when the difference from 1 is larger than the ratio threshold ThR, the process proceeds to step S32, is determined as an object vector, and the process proceeds to step S17 in FIG.

  In comparison of the ratio, the comparison may be performed in a state where the rotation component is canceled. For example, if the ratio is close to 1, it is regarded as the background, otherwise it is not regarded as the background. In order to simplify the calculation, the calculation may be performed using only the larger one of the own vehicle movement vectors of u (vector in the x direction) or v (vector in the y direction).

  Next, a case where the camera 11 is configured to be able to capture the left and right areas will be described.

  When the camera 11 can capture the left and right areas, such as when using a fisheye lens, images may be prepared not only on the front but also on the left and right. At this time, the image is placed directly beside so that the own vehicle movement parameter can be developed in the left and right directions, or the vector acquired by the oblique image or the like is projectively transformed to the beside plane.

For example, in the case of the left image, the rotation parameter Rl (Rlx, Rly, Rlz) can be written as follows using the front rotation parameter.
Rlx = Rz
Rly = Ry
Rlz = -Rx

Similarly, the translation parameter Tl (Tlx, Tly, Tlz) of the left image can be written using the front translation parameter as follows:
Tlx = Tz
Tly = Ty
Tlz = -Tx

It becomes. In the case of the right image, assuming that the rotation parameter is Rr (Rrx, Rry, Rrz) and the translation parameter is Tr (Trx, Try, Trz), the following parameters are obtained from the front parameters.
Rrx = -Rz
Rry = Ry
Rrz = Rx
Trx = -Tz
Try = Ty
Trz = Tx

The left side z to be obtained can be expressed as follows from the position of FOEy in the front image, assuming that the right side of the screen is + x.
z = (Dgnd ・ f / cos (θgnd)) / (y-FOEy ・ x / f) (z is limited by Zmax)
θgnd = atan (FOEy / f)

Similarly, if the right side of the screen is + x, z on the right side can be expressed as follows.
z = (Dgnd ・ f / cos (θgnd)) / (y + FOEy ・ x / f) (z is limited by Zmax)
θgnd = atan (FOEy / f)

  Using the above parameters, the background vector can be detected by performing the same processing as the front image. Further, the background vector detected from the left and right images may be used for calculating the own vehicle movement parameter. In addition, the calculation of the rotation component may be performed only on the front surface that can be easily distinguished from the calculation of the translation component. The vehicle movement parameter obtained for each direction is finally set as one parameter for the front direction. In that case, it may be average or weighted by the number of vectors.

  Next, details of the method for calculating the own vehicle movement parameter using the background vector, which is executed by the own vehicle movement calculation unit 24 in step S20 of FIG. 5, will be described.

  First, the rotation component R of the own vehicle movement parameter is calculated. When the obtained background vector is (x1, y1, u1, v1) (xn, yn, un, vn) = (x coordinate position, y coordinate position, x direction vector, y direction vector), What is necessary is just to obtain | require the least squares solution of these calculation formulas. However, n represents an integer of 1 or more.

  Next, the translation component T is calculated using R obtained by the above-mentioned known rotation component calculation. The obtained background vector is (x1, y1, z1, u1, v1) (xn, yn, zn, un, vn) = (x coordinate position, y coordinate position, z coordinate position, x direction vector, y (Direction vector), a least squares solution of the following formula may be obtained. However, n represents an integer of 1 or more.

  Next, the convergence point FOE (FOEx, FOEy) is calculated using R obtained by the known rotation component calculation. When the obtained background vector is (x1, y1, u1, v1) (xn, yn, un, vn) = (x coordinate position, y coordinate position, x direction vector, y direction vector), What is necessary is just to obtain | require the least squares solution of these calculation formulas. However, n represents an integer of 1 or more.

Furthermore, the above known calculation can be simplified by adding the following conditions. The following conditions may be used when there are few obtained background vectors.
FOEx = 0
FOEy = constant (calibration required)
Rx = 0
Ry = variable (object to be calculated)
Rz = (-FOEy / f) ・ Ry
Tx = 0
Ty = (FOEy / f) ・ Tz
Tz = variable (object to be calculated)

  The reason why FOEy is given a value is to make it easier to detect an obstacle by making the ground direction to some extent and increasing the resolution on the ground side. In order to simplify the calculation, FOEy = −f / 2 or FOEy = −f may be used.

When FOEx is not 0, the calculation may be performed as follows.
Rx = (-FOEx / f) ・ Ry
Tx = (FOEx / f) ・ Tz

The results obtained with the above formulas vary greatly. For this reason, it may be determined that the amount of change in the own vehicle parameter increases as the own vehicle moves, and the average of the obtained vectors (u, v) may be multiplied by a constant to limit the vector. At this time, f / (√ (f · f + x · x + y · y)) may be applied in consideration of the fact that the vector becomes larger because it is farther from the edge of the screen. Also, considering that the vector becomes large on the ground, z / Zmax may be applied to the magnitude of the vector. Moreover, you may use (u-wu, v-wv) which canceled the rotation component. The average of the vector sizes when all of the above are taken into consideration is as follows.
Ans = (1 / n) ・ Σ ((√ (X ・ X + Y ・ Y)) ・ (f / F) ・ (z / Zmax))
X = u-wu
Y = v-wv
F = √ (f ・ f + x ・ x + y ・ y)
wu =-(x ・ y / f) ・ Rx + ((f ・ f + x ・ x) / f) ・ Ry-y ・ Rz
wv =-((f ・ f + y ・ y) / f) ・ Rx + (x ・ y / f) ・ Ry + x ・ Rz
Depending on the result obtained by the above, there is a case where it hardly moves or moves too much. For this reason, a maximum and minimum limit is included. Next, each vehicle movement parameter is multiplied by a specific proportionality constant to limit it. Also, if the initial value can be trusted to some extent by performing calibration first, the restriction on the amount of change to the initial value side (stopped side) is relaxed by loosening twice, etc., to the initial value side. It may be easy to go.

  Next, a method for detecting a background vector by dividing a frame image into a plurality of regions will be described.

  FIG. 7A is an explanatory diagram showing an example in which the left quarter view and the right quarter view of the fisheye camera are projected onto the left screen 42 and the right screen 43 arranged in parallel to the yz plane, respectively. (B) Projects the left 1/4 field of view and the right 1/4 field of view of the fisheye camera on the left screen 42 and the right screen 43 arranged in parallel with the y axis and having a predetermined inclination with respect to the yz plane, respectively. It is explanatory drawing which shows the example to perform, (c) each projects the left 1/4 visual field and right 1/4 visual field of a fish-eye camera on the left screen 42 and the right screen 43 which are arrange | positioned in parallel with xz plane, respectively. It is explanatory drawing which shows an example. In any of FIGS. 7A to 7C, the front left 1/4 field of view and the front right 1/4 field of view of the fisheye camera are projected on the front surface 41.

  FIG. 8 is a diagram for explaining a motion vector calculation method in the case shown in FIG.

  As shown in FIG. 7, when using a camera such as a fish-eye camera that can look around 180 degrees, the image is divided into a front 41, a left screen 42, and a right screen 43 as shown in FIG. You may get it. At this time, it may be only around the horizontal line. If a vector on the road surface is used, it is normalized by multiplying by (z / Zmax). Further, a vector obtained from an image obtained by extending a plane of a different screen direction, a cylindrical image, or a spherical image may be converted into a vector for the front and right sides by projective transformation. At this time, the image or vector is projectively transformed so that the FOE is the center of the front image and the horizontal line is directly beside. The obtained vector is set only in the horizontal direction, the vector becomes larger as the width of the screen becomes smaller, and the vector becomes larger as the ground becomes smaller. The vector obtained as described above is divided into four regions every 45 degrees and averaged.

  For example, in the case shown in FIG. 7A, the obtained vectors are Ufl on the front left, Ufr on the front right, Usl on the left side, Usr on the right side, and the obtained angles are θfl on the front left and When θfr, the left side is θsl, and the right side is θsr, first, assuming that the right side of the screen is + x, the following equation holds for the front image.

Ufl = (1 / n) ・ Σ (u ・ (f / √ (f ・ f + x ・ x)) ・ (z / Zmax))
θfl = (1 / n) ・ Σacos (x / f) (= 90 ° to 135 °)
Ufr = (1 / n) ・ Σ (u ・ (f / √ (f ・ f + x ・ x)) ・ (z / Zmax))
θfr = (1 / n) ・ Σacos (x / f) (= 45 ° ~ 90 °)

If the right side of the screen is + x, the following equation holds for the left screen.
Usl = (1 / n) ・ Σ (u ・ (f / √ (f ・ f + x ・ x)) ・ (z / Zmax))
θsl = 180 °-(1 / n) ・ Σasin (x / f) (= 135 ° to 180 °)

If the right side of the screen is + x, the following equation holds for the right screen.
Usr = (1 / n) ・ Σ (u ・ (f / √ (f ・ f + x ・ x)) ・ (z / Zmax))
θsr =-(1 / n) ・ Σasin (x / f) (= 0 ° ~ 45 °)

  Here, the calculation is performed for the case shown in FIG. 7A, that is, the case where the image plane is the front side and the side side. However, even if the images are acquired in different directions such as FIGS. 7B and 7C. The length of the vector on the semicircle may be calculated. In addition, when a vector is acquired from an image obtained by extending a cylindrical shape to a plane, the x direction becomes an angle. Therefore, normalization with distance (f / √ (f · f + x · x)) is not performed, and the angle is Can be averaged as is.

Since the obtained result is a rough vector, only the y-axis rotation component Ry and the z-direction component Tz are calculated. The four vectors are vectors on a circle. Since it is assumed that the vector is composed only of Ry and Tz, the following equation is obtained when the rotation component is Uro.
f ・ Tz / Zmax =-(Ufl-Uro) / cos (θfl)
f ・ Tz / Zmax =-(Ufr-Uro) / cos (θfr)
f ・ Tz / Zmax =-(Usl-Uro) / cos (θsl)
f ・ Tz / Zmax =-(Usr-Uro) / cos (θsr)
Ry = Uro / f

  If two points are selected from the above, Ry and Tz are derived by simultaneous equations.

  In order to improve the above calculation accuracy, Ry may be derived from Ufl and Ufr in which the translation component appears to be small, and Tz derived from Usl and Usr in which the translation component becomes large may be averaged to be Tz.

  In consideration of the possibility that an object vector close to the background vector is a background vector, the calculation may be performed using all four vectors. In this case, Ry is obtained by averaging Ry derived from Ufl and Ufr and Ry derived from Usl and Usr. Then, Ry is substituted into each calculation formula to derive each Tz, and the average is Tz. Alternatively, a least square solution may be derived.

  In addition, six solutions can be derived from two points, but an intermediate value of the derived solutions (for example, the average of two middle points) may be Ry and Tz. In this case, even when an object vector close to the background vector is used as the background vector, it can be ignored to some extent.

In order to simplify the calculation, the position of the vector average may be an intermediate position divided into four as described below, and may be a position used in the calculation.
θsr = 45 ° / 2
θfr = θsr + 45 °
θfl = θfr + 45 °
θsl = θfl + 45 °

  Assuming that the angle is simplified as described above, it is assumed that the vector is composed only of Ry and Tz. Therefore, if the vectors at the symmetrical positions are averaged, only the Ry component is obtained. When the Ry component is canceled, the vectors at the left and right symmetrical positions have the opposite polarity and the same magnitude.

In order to calculate Ry, an average ((Ufl + Ufr) / 2) of two front vectors which are not easily affected by the translation component may be used. Also, the average of all four vectors ((Ufl + Ufr + Usl + Usr) / 4) may be used to prevent the result from being dragged by the object vector. For example, when a vector is not obtained in the front, an average of two horizontal vectors ((Usl + Usr) / 2) may be used. The average value Uro obtained is considered to be only the rotational component,
Ry = Uro / f

It becomes. Next, Tz is calculated. If you cancel the rotational components of the four vectors,
Ufl = Ufl-Uro
Ufr = Ufr-Uro
Usl = Usl-Uro
Usr = Usr-Uro

Is obtained. The obtained result ideally has only a translational component, and when θ = 22.5 °, the result is as follows.
Ufl ≒ -Ufr
Usl ≒ -Usr
Usl / cos (θ) ≒ Ufl / cos (90 ° -θ)
→ Usl ≒ Ufl / tan (θ)
Usl / cos (θ) = f ・ Tz / Zmax
→ Tz = Zmax ・ Usl / (f ・ cos (θ))

Therefore, the translation component uses Usl and Usr, which tend to be large,
Tz = Zmax ・ ((Usl-Usr) / 2) / (f ・ cos (θ))

Or use all four vectors,
Tz = Zmax ・ ((Usl-Usr + (Ufl-Ufr) / tan (θ)) / 4) / (f ・ cos (θ))

Or just use Usl
Tz = Zmax ・ Usl / (f ・ cos (θ))

Or just use Usr
Tz = -Zmax ・ Usr / (f ・ cos (θ))

It is also good. If the vector can only be obtained from two places, and the rotation component is Uro,
Ufl-Uro ≒-(Ufr-Uro)
Usl-Uro ≒-(Usr-Uro)
Usl-Uro ≒ (Ufl-Uro) / tan (θ)

  Thus, Uro may be calculated and derived by simultaneous equations, and Ry and Tz may be calculated.

The own vehicle movement parameter is limited so as to be in the vicinity of Ry and Tz obtained as described above. Rx, Rz, Tx, and Ty are values derived from the following equations, and the own vehicle movement parameters are limited so that they are in the vicinity thereof.
Rx = (-FOEx / f) ・ Ry
Rz = (-FOEy / f) ・ Ry
Tx = (FOEx / f) ・ Tz
Ty = (FOEy / f) ・ Tz

  In addition, if the horizontal line is converted from a state that is not directly beside, to the beside, the reverse rotation is performed accordingly.

  In addition, you may use the parameter obtained here as an own vehicle movement parameter as it is, without using it as a restriction | limiting. Further, although divided into four, it may be finer, such as eight, or may be roughly divided into two or three. Moreover, it is good also as another angle, such as 30 degree, instead of the area | region for every 45 degree | times, and you may change an angle for every area | region. In this case, it is desirable that the angle region is the same on the left and right.

  Next, limitations due to restrictions on vehicle behavior will be described.

  Since there is a restriction on the behavior of the car, a restriction corresponding to this restriction may be added. For example, when there is no slip, there is a restriction that it is necessary to translate to some extent because the tire is attached to the front and rear in order to rotate. These restrictions 1-9 are exemplified below. Further, since there is a possibility that the parameter is an abnormal value, a limit change amount may be set.

Restriction 1 is a restriction that there is Rz if there is a shift amount Ry of FOEy.
Rz = (-FOEy / f) ・ Ry

Similarly, as described below, a restriction may be placed on a parameter whose position of a variable is changed.
Ry = Rz / (-FOEy / f)
FOEy = -f ・ Rz / Ry

The restriction 2 is a restriction that there is Ty if there is a shift amount Tz of FOEy.
Ty = (FOEy / f) ・ Tz

  Similarly, a restriction may be placed on a parameter whose position of a variable is changed.

Restriction 3 is a restriction that there is Rx if there is a shift amount Ry of FOEx.
Rx = (-FOEx / f) ・ Ry

  Similarly, a restriction may be placed on a parameter whose position of a variable is changed.

Limit 4 is a limit that Tx exists if there is a shift Tz of FOEx.
Tx = (FOEx / f) ・ Tz

  Similarly, a restriction may be placed on a parameter whose position of a variable is changed.

The restriction 5 is a restriction that if there is Ry, there is Tz.
Moving forward in screen direction Tz = -A ・ | Ry |
Moving backward from screen direction Tz = A ・ | Ry |

  A is a proportionality constant when the maximum depression angle is given. This is a limitation when the magnitude of Tz is smaller than the value here. Whether the vehicle is moving forward or backward may be in a direction in which information by CAN (Communication Area Network) and the sign of Tz do not change.

Limit 6 is a limit that Tx requires Tz.
B = A ・ | Tz |

  A is set to a small value to some extent, and Tx is limited to be between (FOEx / f) · Tz ± B.

The restriction 7 is a restriction that Ty requires Tz.
B = A ・ | Tz |

  Ty is limited to a value between AFO and (FOEy / f) · Tz ± B with a small value.

The limit 8 is a limit that advancing in the reverse direction is allowed a little but not a large value. For example, even when the captured image of the back camera is not displayed on the display device 19 only when the host vehicle is moving backward, if the brake is not stepped on a hill or the like, the polarity Tz of only the forward direction to the screen is not obtained. There is. However, it is not a large value. For this reason, a limit is placed on the size when the polarity is reversed.
Moving forward in screen direction Tz <A
Moving backward from the screen direction Tz> -A
A is a small positive value.

The restriction 9 is a restriction that it is determined that the vehicle is stopped when Ry and Tz are smaller than a certain threshold value.
Rx = 0
Ry = 0
Rz = 0
Tz = A
Tx = (FOEx / f) ・ Tz
Ty = (FOEy / f) ・ Tz

  A is set to a small value, but Tz is set to a negative value when moving forward in the screen direction, and set to a positive value when moving backward from the screen direction. Whether the vehicle is moving forward or backward may be in the direction in which the information by CAN and the sign of Tz do not change. In consideration of the case of vector angle comparison, the parameter is not set to 0 and a small value is entered.

  Next, the restriction by the road surface vector will be described.

  Even when the vehicle is moving forward, if the background is too far, the background hardly moves. On the other hand, since the z coordinate limit is set, the parameter is greatly shifted. For this reason, a line drawn on the road surface may be erroneously determined as an object, or an object on the road surface may be erroneously determined as a background vector that is a road surface pattern.

  In order to avoid this erroneous determination, it is first assumed that the object is in contact with the road surface and a vector is obtained at the contact position. Then, the motion vector is detected from the bottom of the screen, and if the number of vectors that have a certain size below the specified angle below the horizontal line and become the background vector reaches the specified number, the vehicle movement parameter is calculated by the calculation described later. to correct. In consideration of erroneous detection of the background vector, the amount of change in the own vehicle movement parameter is limited. When the correction is completed within the limit of the amount of change, the standard for determining the motion vector detected after the correction as a background vector is strict so that it can be easily detected as an object. Specifically, for example, the process of heavily weighting the difference threshold ThS to S1-S3 may not be executed, or the degree of influence thereof may be reduced.

  The image may be corrected using vectors obtained from the left and right images. In that case, the vector is projectively transformed. By using the left and right images, it is possible to use a side image that is not visible in the front image. Conversely, the following equation may be calculated using the own vehicle movement parameters in the left and right images and converted into the own vehicle movement parameters in the front image.

As a premise of the correction process, the parameters to be corrected are only Ry and Tz on the behavior of the own vehicle, and Rx, Rz, Tx, and Ty are
Rx = (-FOEx / f) ・ Ry
Rz = (-FOEy / f) ・ Ry
Tx = (FOEx / f) ・ Tz
Ty = (FOEy / f) ・ Tz

  It will be derived in. Therefore, the parameters to be derived are ΔRy and ΔTz.

When these parameters ΔRy and ΔTz are solved by simultaneous equations, they are known equations.
wu =-(x ・ y / f) ・ Rx + ((f ・ f + x ・ x) / f) ・ Ry-y ・ Rz
wv =-((f ・ f + y ・ y) / f) ・ Rx + (x ・ y / f) ・ Ry + x ・ Rz
tu = (f ・ Tx-x ・ Tz) / z
tv = (f ・ Ty-y ・ Tz) / z

Than,
Axr = -FOEy / f
Axt = FOEy / f
ΔRx = Axr ・ ΔRy
ΔTx = Axt ・ ΔTz
Ayr = -FOEy / f
Ayt = FOEy / f
ΔRz = Ayr ・ ΔRy
ΔTy = Ayt ・ ΔTz

Substituting
Δwu = (-(x ・ y / f) ・ Axr + (f ・ f + x ・ x) / f-y ・ Ayr) ・ ΔRy
Δwv = (-((f ・ f + y ・ y) / f) ・ Axr + x ・ y / f + x ・ Ayr) ・ ΔRy
Δtu = (f ・ Axt-x) ・ ΔTz / z
Δtv = (f ・ Ayt-y) ・ ΔTz / z

The following conditions are satisfied because the background is correct.
u-(wu + Δwu) = tu + Δtu
v-(wv + Δwv) = tv + Δtv

  From the above conditions, ΔRy and ΔTz are derived by simultaneous equations. The same processing is performed with a specified number of vectors, and the obtained ΔRy and ΔTz are averaged. Based on the obtained ΔRy and ΔTz, ΔRx, ΔRz, ΔTx, and ΔTy are also created and corrected.

The calculation by the above simultaneous equations is complicated and heavy. Therefore, for easier processing, the parameters ΔRy and ΔTz may be derived by regarding the vertical component as a translation component and the horizontal component as a rotation component. In other words, since the vertical component of the detected vector on the ground near the host vehicle is almost the translation component (Ty, Tz), the translation component change amount ΔRy is ignored and the correction amount is as follows.
v-wv = tv
→ v-(-((f ・ f + y ・ y) / f) ・ Axr + x ・ y / f + x ・ Ayr) ・ Ry = (f ・ Ayt-y) ・ (Tz + ΔTz) / z
→ ΔTz = (v-(-((f ・ f + y ・ y) / f) ・ Axr + x ・ y / f + x ・ Ayr) ・ Ry) ・ z / (f ・ Ayt-y)-Tz

When ΔRy is derived by calculating the lateral component from ΔTz obtained as described above,
u-wu = tu
→ u-(-(x ・ y / f) ・ Axr + (f ・ f + x ・ x) / f-y ・ Ayr) ・ (Ry + ΔRy) = (f ・ Axt-x) ・ (Tz + ΔTz ) / z
→ ΔRy = (u-(f ・ Axt-x) ・ (Tz + ΔTz) / z) / (-(x ・ y / f) ・ Axr + (f ・ f + x ・ x) / f-y ・ Ayr )-Ry

  It becomes. The same processing is performed with a specified number of vectors, and the obtained ΔRy and ΔTz are averaged. Based on the obtained ΔRy and ΔTz, ΔRx, ΔRz, ΔTx, and ΔTy are also created and corrected.

  In addition, although some embodiment of this invention was described, these embodiment is shown as an example and is not intending limiting the range of invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  For example, the processing may be performed in parallel using a plurality of images obtained by a plurality of cameras attached to different locations. In this case, even when the object is surrounded, the possibility that the own vehicle movement parameter can be specified by some camera is increased.

  Further, in the embodiment of the present invention, each step of the flowchart shows an example of processing that is performed in time series in the order described. The process to be executed is also included.

DESCRIPTION OF SYMBOLS 10 Own vehicle movement estimation apparatus 11 Camera 12 Main control part 13 Storage part 21 Plane image generation part 22 Motion vector detection part 23 Background vector detection part 24 Own vehicle movement calculation part 25 Object vector processing part

Claims (10)

A camera provided on the vehicle images the surroundings of the vehicle at a predetermined frame period;
Detecting a motion vector between a plurality of images captured by the camera;
Obtaining a moving parameter of the own vehicle from the detected motion vector;
Detecting a new motion vector using an image captured by the camera after capturing the plurality of images used in determining the movement parameter of the host vehicle;
Detecting, as a background vector, the newly detected motion vector that is within a predetermined threshold in comparison with the movement vector of the own vehicle corresponding to the movement parameter of the own vehicle;
Using this background vector to obtain a new movement parameter of the vehicle,
The own vehicle movement estimation method comprising: The step of detecting a new motion vector comprises:
Detecting a motion vector of a feature point on a road surface or a feature point located at a position more than a predetermined distance in the traveling direction from the vehicle,
The own vehicle movement estimation method according to claim 1. Imaging the surroundings of the vehicle,
A fisheye camera provided in the vehicle captures a fisheye image including a front image, a left image, and a right image of the vehicle at a predetermined frame period;
The step of obtaining the movement parameter of the own vehicle from the detected motion vector includes:
Obtaining a moving parameter of the own vehicle from each of the front image, the left image and the right image included in the fisheye image;
Detecting as the background vector comprises:
Detecting the background vector from each of the front image, left image and right image;
The step of newly obtaining the movement parameter of the own vehicle includes:
The movement parameter of the own vehicle is obtained from the background vector detected from each of the front image, the left image, and the right image, and the movement parameter of the own vehicle is obtained from each of the front image, the left image, and the right image. It is a step to newly determine the movement parameter of the own vehicle based on
The own vehicle movement estimation method according to claim 1 or 2. The step of newly obtaining the movement parameter of the own vehicle includes:
Among the background vectors, a step of newly obtaining a movement parameter of the own vehicle using a background vector within a predetermined range from an average of all the background vectors.
The own vehicle movement estimation method according to any one of claims 1 to 3. Dividing the image around the vehicle into a plurality of areas, and detecting the background vector for each divided area;
A provisional parameter calculating step for obtaining a traveling direction component of the vehicle of the translation component and a component around a road surface normal direction axis of the rotation component based on the background vector detected for each of the divided regions;
The newly calculated value is such that the translational component of the translational component of the vehicle and the component of the rotational component around the road surface normal axis are within predetermined values from the values obtained in the temporary parameter calculating step. Limiting the movement parameters of the own vehicle,
Further having
The own vehicle movement estimation method according to any one of claims 1 to 4. Limiting the newly determined movement parameter of the host vehicle using a restriction on the behavior of the vehicle;
Further having
The own vehicle movement estimation method according to any one of claims 1 to 5. Obtaining a motion vector of a vehicle vicinity road surface from an image of the vicinity of the vehicle included in a lower image region of images around the vehicle;
Correcting the newly determined movement parameter of the host vehicle using a motion vector of the road surface near the vehicle;
Further having
The own vehicle movement estimation method according to any one of claims 1 to 6. Corresponding to the corrected movement parameter of the own vehicle among the newly detected motion vectors after executing the step of correcting the newly obtained movement parameter of the own vehicle using the motion vector of the road surface near the vehicle Detecting a background vector that is within a difference smaller than the predetermined threshold in comparison with the movement vector of the own vehicle,
Further having
The own vehicle movement estimation method according to claim 7. Steps in which different images around the vehicle are captured by each of a plurality of cameras provided at a plurality of locations of the vehicle;
Further comprising
One of the plurality of own vehicle movement parameters obtained based on the images captured by each of the plurality of cameras is the newly obtained own vehicle movement parameter,
The own vehicle movement estimation method according to any one of claims 1 to 8. On the computer,
Acquiring a surrounding image of the vehicle captured by a camera provided in the vehicle at a predetermined frame period;
Detecting a motion vector between a plurality of images captured by the camera;
Obtaining a moving parameter of the own vehicle from the detected motion vector;
A step of newly detecting a motion vector using an image captured by the camera after capturing the plurality of images used when determining the movement parameter of the own vehicle;
Detecting, as a background vector, the newly detected motion vector that is within a predetermined threshold in comparison with the movement vector of the own vehicle corresponding to the movement parameter of the own vehicle;
Using this background vector to obtain a new movement parameter of the vehicle,
A vehicle movement estimation program for executing

Images