JP5588332B2 - Image processing apparatus for vehicle and image processing method for vehicle - Google Patents

Description

  Embodiments described herein relate generally to a vehicle image processing apparatus and a vehicle image processing method.

  Recently, as a technique for supporting driving of a vehicle such as an automobile, development of a technique for determining the possibility of collision between the vehicle and an object is desired. As this type of technology, for example, there is a technology for determining the possibility of collision between a vehicle and an object based on an image captured by a camera mounted on the vehicle. When using an image captured by a camera mounted on a vehicle, it is possible to use digitized image data as compared with the case of using a radar, so that it is possible to make a complicated determination such as an approach angle of an object. .

Japanese Patent Laid-Open No. 2004-56763

  By the way, the conventional technique for determining the possibility of collision between the vehicle and the object based on the image captured by the camera is based on the premise that the road surface imaged by the camera is parallel to the vehicle bottom surface (road surface plane model). Therefore, the case where the road surface imaged by the camera and the vehicle bottom surface are not parallel is not considered. For this reason, when the road surface imaged by the camera and the vehicle bottom surface are not parallel, it is difficult to obtain the distance from the object and specify the moving body from the image.

  For example, the road surface imaged by the camera and the vehicle bottom surface are not parallel to each other. For example, the road surface gradient changes, and the road surface directly under the vehicle and the road surface imaged by the camera have a gradient change, or the road surface is flat. Even so, there are situations in which there is a vehicle tilt due to a large amount of vehicle load and fluctuations in the number of passengers. These situations are likely to be encountered in actual vehicle driving.

  In order to solve the above-described problems, a vehicle image processing apparatus according to an embodiment of the present invention includes a camera, a translation component calculation unit with a scale factor, a storage unit, a road surface detection unit, and an approximate plane creation unit. And a road surface data updating unit. The camera is provided in the vehicle and images the periphery of the vehicle. The translation component with a scale factor calculation unit calculates a relative translation component with a scale factor based on a motion vector corresponding to an object included in an image captured by the camera and a rotation component of the vehicle. A memory | storage part memorize | stores the road surface data which are the data which linked | related the coordinate value on the image in a camera coordinate system, and the optical axis direction coordinate value about the point on a predetermined setting road surface. The road surface detection unit calculates the optical axis direction coordinate value of the object in the camera coordinate system using the relative translation component with the scale factor and the translation component of the own vehicle, and the coordinate value on the image of the object in the road surface data The associated optical axis direction coordinate values are acquired from the storage unit, and if the difference between these optical axis direction coordinate values is equal to or less than a predetermined road surface threshold, the object is determined to be a road surface. The approximate plane creation unit uses the eigenvector of the covariance matrix of the three-dimensional coordinate values in the camera coordinate system of the plurality of objects determined as the road surface to establish this expression representing the approximate plane formed by the plurality of objects. Approximate plane data in which the coordinate value on the image in the camera coordinate system and the coordinate value in the optical axis direction are associated with each other on the approximate plane is obtained. The road surface data update unit updates the road surface data in the storage unit with the approximate plane data so that the predetermined set road surface is an approximate plane.

1 is an overall configuration diagram illustrating an example of a vehicle image processing device according to a first embodiment of the present invention. Explanatory drawing which shows the relationship between the camera projection surface in a camera coordinate system, and a road surface. The flowchart which shows the procedure at the time of CPU of image processing ECU which concerns on 1st Embodiment calculates | requires the position of a road surface in real time from the image acquired with the camera. The whole block diagram which shows an example of the image processing apparatus for vehicles which concerns on 2nd Embodiment of this invention. The flowchart which shows the procedure at the time of CPU of image processing ECU which concerns on 2nd Embodiment calculates | requires the position of a road surface in real time from the image acquired with the camera. The whole block diagram which shows an example of the image processing apparatus for vehicles which concerns on 3rd Embodiment of this invention. The flowchart which shows the procedure at the time of CPU of image processing ECU which concerns on 3rd Embodiment calculates | requires the position of a road surface in real time from the image acquired with the camera.

Embodiments of a vehicle image processing apparatus and a vehicle image processing method according to the present invention will be described with reference to the accompanying drawings.
(First embodiment)

  FIG. 1 is an overall configuration diagram showing an example of a vehicle image processing apparatus 10 according to the first embodiment of the present invention.

  The vehicle image processing apparatus 10 includes a camera 11, an image processing ECU (Electronic Control Unit) 12, and a storage unit 13.

  The camera 11 is constituted by a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The camera 11 captures an image around a vehicle such as a private car and generates an image signal and supplies the image signal to the image processing ECU 12.

  For example, when monitoring the rear, the camera 11 is disposed slightly downward from a line parallel to the road surface in the vicinity of the license plate at the rear of the vehicle. A wide-angle lens or a fisheye lens may be attached to the camera 11 so that a wider range of outside-vehicle images can be acquired. When monitoring the side of the vehicle, the camera 11 is disposed near the side mirror. Of course, a wide range of outside-vehicle surrounding images may be captured by using a plurality of cameras 11.

  The image processing ECU 12 includes a storage medium such as a CPU, RAM, and ROM. The CPU of the image processing ECU 12 loads an image processing program stored in a storage medium such as a ROM and data necessary for executing the program into the RAM, and based on the image acquired by the camera 11 according to the program. The position of the road surface is obtained in real time from the image acquired by the camera 11.

  The RAM of the image processing ECU 12 provides a work area for temporarily storing programs and data executed by the CPU. The storage medium such as the ROM of the image processing ECU 12 stores an image processing program and various data necessary for executing these programs.

  A storage medium such as a ROM has a configuration including a recording medium readable by a CPU, such as a magnetic or optical recording medium or a semiconductor memory, and a part of programs and data in the storage medium. Or you may comprise so that all may be downloaded via an electronic network via the network connection part which is not shown in figure.

  In this case, the network connection unit implements various information communication protocols according to the form of the network, and the image processing ECU 12 and electric devices such as ECUs of other vehicles are connected via the electronic network according to the various protocols. Connecting. For this connection, an electrical connection via an electronic network can be applied. Here, an electronic network means an information communication network using telecommunications technology in general, in addition to a wireless / wired LAN (Local Area Network) and the Internet network, a telephone communication line network, an optical fiber communication network, a cable communication network, Includes satellite communications networks.

  The storage unit 13 is a non-volatile memory in which data can be read and written by the image processing ECU 12, and information FOE (x0, y0) of vanishing point (FOE: Focus of Expansion) and road surface data z2 = z (x, x, y), road surface threshold zth, first eigenvalue threshold λth1, second eigenvalue threshold λth2, camera height H, and angle threshold ψth are stored. These pieces of information may be updated via an electronic network or a portable storage medium such as an optical disk.

  Road surface data z2 = z (x, y) refers to data in which a coordinate value on an image and an optical axis direction coordinate value are associated with each other on a predetermined set road surface. That is, the image processing ECU 12 refers to the road surface data stored in the storage unit 13 so that the object is displayed on the predetermined set road surface for the object displayed at the coordinate values (x, y) on the image. The optical axis direction coordinate value z2 in the case where it is assumed to be present can be acquired.

As shown in FIG. 1, the CPU of the image processing ECU 12 executes at least a swing angle correction unit 21, a motion vector calculation unit 22, a rotation component calculation unit 23, a moving body detection unit 24, and a translation component with a scale factor according to an image processing program. Unit 25, own vehicle translation component calculation unit 26, road surface detection unit 27, road surface point coordinate calculation unit 28, eigenvalue / eigenvector calculation unit 29, fixed point setting unit 30, approximate plane creation unit 31, road surface data update unit 32, and swing angle It functions as the calculation unit 33. Each of the units 21 to 33 uses a required work area of the RAM as a temporary storage location for data. In addition, you may comprise these function implementation parts by hardware logics, such as a circuit, without using CPU.
Here, the camera coordinate system will be briefly described.

  FIG. 2 is an explanatory diagram showing the relationship between the camera projection surface 41 and the road surface 42 in the camera coordinate system.

  Since the vehicular image processing apparatus 10 according to the present embodiment uses only the camera coordinate system and does not use the world coordinate system, processing such as mutual conversion of the coordinate system is unnecessary.

  The camera coordinate system (x, y, z) is a left-handed coordinate system with the center of the camera 11 as the origin. The z axis of the camera coordinate system is set to coincide with the optical axis of the camera 11.

  The camera projection plane 41 is a plane with z = f in the camera coordinate system. f is the focal length of the camera 11. The camera 11 captures an image around the vehicle projected on the camera projection plane 41.

  The three-dimensional coordinate of the camera coordinate system of the point P with the object is P (X, Y, Z), and the coordinate of the point p projected onto the camera projection plane 41 of the point P is p (x, y, f). To do. If the z coordinate (optical axis direction coordinate) of the point P is defined as z (Z = z), X = (z / f) · x, Y = (z / f) · y, and Z = z. Note that x and y are xy coordinate values on the projection plane 41, and z is a z coordinate value of the object. Since the xy coordinate values on the projection surface 41 correspond to the pixels of the camera 11, they can be easily obtained from the image.

  The camera height H refers to the height of the center of the camera 11 when viewed from the road surface (hereinafter referred to as a plane directly below the vehicle) 43 directly below the vehicle where the tire contacts the ground.

  In the present embodiment, the intersection of the vertical line drawn from the center of the camera 11 and the plane 43 directly below the vehicle is a fixed point P0 (α, β, γ), and the camera height H that is the distance from P0 to the origin is It is assumed to be a constant. In the present embodiment, the road surface 42 imaged by the camera 11 passes through the fixed point P0.

  The swing angle correction unit 21 rotationally corrects the image captured by the camera 11 around the z axis so as to correct the angle deviation ψ received from the swing angle calculation unit 33. As a result, the image captured by the camera 11 is corrected for the mounting angle error around the z-axis of the camera 11 and the horizon is displayed at a horizontal position, for example. The rotation correction by the swing angle correction unit 21 may be used, for example, in an initial setting when the camera 11 is attached.

  The motion vector calculation unit 22 calculates a motion vector (u, v) for each pixel from a plurality of images captured by the camera 11 using a block matching method, a gradient method, or the like.

The rotation component calculation unit 23 calculates rotation components R _x , R _y , R _z based on the motion vector (u, v) and the vanishing point information FOE (x0, y0). More specific description will be given below.

It is known that the motion vector (u, v) on the camera projection plane 41 can be written as follows.
u = xy (1 / f) R _x − (x ² + f ² ) (1 / f) R _y + yR _z −f (T _x / z) + x (T _z / z) (1)
v = (y ² + f ² ) (1 / f) R _x −xy (1 / f) R _y −xR _z −f (T _y / z) + y (T _z / z) (2)

Here, x and y are xy coordinate values on the projection plane 41, and z is a z coordinate value of the object. Also, R _x , R _y , R _z , T _x , T _y , T _z are motion parameters, R _x , R _y , R _z represent the rotational components of the vehicle, and T _x , T _y , T _z are Represents the relative translation component between the vehicle and the object.

The rotation component _{Rx of the} vehicle is a rotation component about the x-axis, and represents the pitching component of the own vehicle. R _y is a rotational component about the y-axis and represents the yawing component of the host vehicle. Yawing occurs mainly at the steering angle (when the vehicle is turning). R _z is a rotation component about the z axis and represents a rolling component of the vehicle.

When the object is a stationary object, it is known that the extension line of the motion vector of the object corrected by the rotation component converges to FOE (x0, y0) (Takashi Sakimoto et al. Symposium 2003 Proceedings, "Study on Moving Vehicle Detection and Tracking Using Extended Focus" (2003)). Therefore, the following formula (3) is derived.
[u−xy (1 / f) R _x + (x ² + f ² ) (1 / f) R _y −yR _z ] / [v− (y ² + f ² ) (1 / f) R _x + xy (1 / f) R _y + xR _z ]
= [x-x0] / [y-y0] (3)

Accordingly, if R _x , R _y , and R _z are considered as three unknowns, R _x , R _y , and R _z can be calculated if motion vectors of at least three pixels corresponding to the stationary object can be obtained.

Therefore, the rotation component calculation unit 23 acquires the vanishing point information FOE (x0, y0) from the storage unit 13, and formula (3) for the motion vectors (u, v) of three or more pixels corresponding to the stationary object. Based on this, rotation components R _x , R _y , R _z are calculated. That is, the rotation component calculation unit 23 can calculate the rotation component from the image acquired by the camera 11.

  The rotation component can also be obtained from vehicle information such as vehicle speed data and steering angle data, a gyro sensor, and the like.

The moving object detection unit 24 determines that an object that does not converge at the vanishing point FOE is a moving object that moves differently from the own vehicle, and determines that an object that converges at the vanishing point FOE is a stationary object. The moving body detection unit 24 gives information about the object determined as a stationary object to the rotation component calculation unit 23 and the own vehicle translation component calculation unit 26. For this reason, the rotation component calculation unit 23 determines the rotation components R _x and R _y based on the equation (3) for the motion vectors (u, v) of three or more pixels corresponding to the object determined to be a stationary object. , R _z can be calculated.

The translation factor calculation unit 25 with the scale factor includes the scale factor of the object based on the motion vector (u, v) corresponding to the object included in the image and the vehicle rotation components R _x , R _y , R _z. Relative translational components (hereinafter referred to as scaled translational components) T _x / z, T _y / z, and T _z / z are calculated.

The translation components T _x / z, T _y / z, and T _z / z with scale factors are obtained by dividing the relative translation components T _x , T _y , and T _z of the object by the z coordinate value of the object. For this reason, if the relative translation components T _x , T _y , and T _z are the same, the farther away (the larger the z coordinate value), the smaller the translation component with a scale factor. Regardless of whether the point P of the object is stationary or moving, the point P is observed on the image as moving with the translation components T _x / z, T _y / z, and T _z / z with scale factors. Is done. More specific description will be given below.

Equations (1) and (2) can be modified as follows.
u−xy (1 / f) R _x + (x ² + f ² ) (1 / f) R _y −yR _z = −f (T _x / z) + x (T _z / z) (4)
v− (y ² + f ² ) (1 / f) R _x + xy (1 / f) R _y + xR _z = −f (T _y / z) + y (T _z / z) (5)

  In Expressions (4) and (5), the left side is obtained by correcting the motion vector (u, v) with a rotation component.

From the equations (4) and (5), the scale factor of the object is determined based on the motion vector (u, v) corresponding to the object included in the image and the rotation components R _x , R _y , R _{z of the} vehicle. The attached translational components T _x / z, T _y / z, and T _z / z can be calculated.

In addition, the inclination of the motion vector of a stationary object (including a moving body that moves in parallel with the own vehicle) corrected by the own vehicle rotation component is equal to the inclination of the vector from the pixel (x, y) toward the FOE. From this relationship, the following expressions (6) to (9) are derived.
(x−x0) / (y−y0) = {−f (T _x / z) + x (T _z / z)} / {−f (T _y / z) + y (T _z / z)}
= (x−fT _x / T _z ) / (y−fT _y / T _z ) (6)
x0 / f = T _x / T _z (7)
yo / f = T _y / T _z (8)

By substituting T _x and T _y with T _z using Equation (7) and Equation (8), Equation (4) and Equation (5) can be modified as follows.
{u−xy (1 / f) R _x + (x ² + f ² ) (1 / f) R _y −yR _z } = (− x0 + x) (T _z / z) (9)
{v− (y ² + f ² ) (1 / f) R _x + xy (1 / f) R _y + xR _z } = (− y 0 + y) (T _z / z) (10)

Therefore, the translation component with scale factor calculation unit 25 can calculate the translation component with scale factor T _z / z based on the equations (9) and (10).

The own vehicle translation component calculation unit 26 uses the road surface data stored in the storage unit 13 to display the object n (n is a subscript for identifying the object) displayed at the coordinate values (x, y) on the image. Is obtained on the predetermined set road surface, the coordinate value z2n in the optical axis direction is obtained, and z2n with respect to the translation component with a scale factor (T _z / z) n of the object n The vehicle translation component T _z is calculated by multiplying by.

When the object n is a stationary object, if (T _z / z) n · z2n is calculated, it becomes an apparent translation component of the own vehicle. If the average of m vehicle translation components is T _z ave, T _z ave can be obtained from the following equation.
T _z ave = (1 / m) Σ (T _z / z) n · z2n (n = 1,2, ... m-1, m) (11)

The road surface detection unit 27 calculates the coordinate value z1n of the object n in the optical axis direction from the average T _z ave of the own vehicle translation component and the translation component with a scale factor (T _z / z) n of the object n using the following equation. Is calculated.
z1n = (z / T _z ) n ・ T _z ave (12)

  Further, the road surface detection unit 27 uses the optical axis direction coordinate value z1n of the object n obtained using the equation (12) and the coordinate value (xn) on the image of the object n in the road surface data stored in the storage unit 13. , Yn) is compared with the optical axis direction coordinate value z2n = z (xn, yn), and if the absolute value of the difference between the two is equal to or less than a predetermined road surface threshold zth, the object n is determined to be a road surface. Then, the information on the object n is given to the road surface point coordinate calculation unit 28.

  The road surface point coordinate calculation unit 28 uses the coordinate value (xn, yn) on the image of the object n determined as the road surface by the road surface detection unit 27 and the optical axis direction coordinate value z1n, and the camera of the object n. Find the three-dimensional coordinate value in the coordinate system.

The three-dimensional coordinate value Pn (Xn, Yn, Zn) of the point Pn corresponding to the object n satisfies the following relationship with xn, yn, z1n.
Xn = (z1n / f) .xn (13)
Yn = (z1n / f) · yn (14)
Z = z1n (15)

  The point coordinate calculation unit 28 on the road surface obtains a three-dimensional coordinate value of the point Pn in the camera coordinate system using the equations (13) to (15).

Here, Z1n is determined from the mean _T z ave of the vehicle translation component estimated from translation components _T z / z and the image with the scale _factor, T z ave translational component _T z / z and the last with the scale factor Since it is obtained from the road surface data z2 = z (x, y), there is some error. However, this error has the same effect on Xn, Yn, and Zn as is apparent from equations (13)-(15). Therefore, it does not affect the inclination of the plane derived as a plane on which a plurality of points P are placed.

The eigenvalue / eigenvector calculation unit 29 creates the following covariance matrix S from Pn (Xn, Yn, Zn).
S = (1 / N) Σ (Xn, Yn, Zn) ^T (Xn, Yn, Zn) (16)
However, N represents the total number of objects, and Σ represents the total sum for N pixels.

  Further, the eigenvalue / eigenvector calculation unit 29 obtains eigenvalues λ1, λ2, λ3, and eigenvectors η1, η2, and η3 of the covariance matrix S. However, λ1, λ2, and λ3 are set in descending order of eigenvalues, and eigenvectors corresponding to the eigenvalues are η1, η2, and η3, respectively.

  When the distribution of a plurality of points is approximated by a plane, it is known that a plane extending by η1 and η2 gives a least square optimal solution. In addition, it is considered that as the eigenvalues λ1 and λ2 of the covariance matrix S are larger, a larger number of points are dispersed in the plane direction spanned by η1 and η2. Further, it is considered that the smaller the λ3 is, the smaller the dispersion of the points is in the direction orthogonal to the plane stretched by η1 and η2.

  Therefore, the eigenvalue / eigenvector calculation unit 29, when the eigenvalues λ1 and λ2 are equal to or greater than the first eigenvalue threshold λth1 and the eigenvalue λ3 is equal to or less than the second eigenvalue threshold λth2, 42 is determined to be approximated by a plane spanned by η1 and η2, and information on the eigenvalues λ1, λ2, λ3 and eigenvectors η1, η2, and η3 is given to the approximate plane creating unit 31. On the other hand, the eigenvalue / eigenvector calculation unit 29 approximates the road surface 42 when the eigenvalues λ1 and λ2 are smaller than the first eigenvalue threshold λth1 or the eigenvalue λ3 is larger than the second eigenvalue threshold λth2. It is determined that it is insufficient for obtaining a flat surface and discarded.

The fixed point setting unit 30 sets the fixed point P0 as follows using the information of the camera height H and the vanishing point FOE. In the present embodiment, the camera height H and the vanishing point FOE are known constants stored in the storage unit 13 in advance.
α = 0 (17)
β = −Hf / √ (f ² + y0 ² ) (18)
γ = Hy0 / √ (f ² + y0 ² ) (19)

  The approximate plane creation unit 31 uses the eigenvector of the covariance matrix S to establish an expression representing the approximate plane formed by the object n determined as the road surface by the road surface detection unit 27.

  Specifically, the expression representing the approximate plane can be established by the following two methods. In the following description, it is assumed that η1 = (η11, η12, η13), η2 = (η21, η22, η23), and η1 = (η31, η32, η33).

The first method is a method of creating an expression representing an approximate plane using the eigenvector η3. The eigenvector η3 is a normal vector of the approximate plane (the approximate plane is orthogonal to the eigenvector η3). In this case, if the approximate plane passes through the fixed point P0 (α, β, γ), an expression representing the approximate plane can be written as follows.
η31 (X−α) + η32 (Y−β) + η33 (Z−γ) = 0 (20)

By substituting equations (17)-(19) into equation (20), the following equation is obtained.
η31 (xz−αf) + η32 (yz−βf) + η33 (fz−γf) = 0 (21)

Here, (x, y) is a coordinate value on the image of the point P (X, Y, Z) on the approximate plane, and z is the optical axis direction of the point P (X, Y, Z) on the approximate plane. It is a coordinate value. For this reason, by transforming Equation (21) as follows, for the object displayed at the coordinate value (x, y) on the image, the coordinates in the optical axis direction when this object is on the approximate plane The value z and the coordinate value (x, y) on the image can be associated with each other.
z = f (η31α + η32β + η33γ) / (η31x + η32y + η33f)
= Hf (−η32f + η33y0) / {(η31x + η32y + η33f) √ (f2 + yo2)} (22)

The second method is a method of creating an expression representing an approximate plane using the eigenvector η1 × η2. In this case, if the approximate plane passes through the fixed point P0 (α, β, γ), an expression representing the approximate plane can be written as follows.
(η13η22−η23η12) (X−α) + (η11η23−η13η21) (Y−β)
+ (η21, η12−η11, η22) (Z−γ) = 0 (23)

Therefore, as in the first method, the object is displayed on the approximate plane for the object displayed at the coordinate values (x, y) on the image by transforming Equation (23) as follows. The optical axis direction coordinate value z in a certain case and the coordinate value (x, y) on the image can be associated with each other.
z = f {(η13η22−η23η12) α + (η11η23−η13η21) β + η33γ}
/ {(η13η22−η23η12) x + (η11η23−η13η21) y + (η21η12−η11η22) f} (24)

  The data of the optical axis direction coordinate value z and the coordinate value (x, y) on the image (hereinafter referred to as approximate plane data) associated by the equations (22) and (24) are the road surfaces stored in the storage unit 13. It can be said that the data is closer to the road surface included in the image captured by the camera 11 than the data.

At this time, the distance r from the origin to the point P on the approximate plane is given by the following equation.
r ² = X ² + Y ² + Z ² = (x ² + y ² + f ² ) (z ² / f ² ) (25)

  Therefore, if the coordinate value (x, y) on the image is obtained, the three-dimensional coordinate value in the case where it is assumed that the object displayed at the coordinate value (x, y) is on the approximate plane is expressed by the formula ( 22) and Equation (24), and the distance r can be uniquely obtained from Equation (25).

  The road surface data update unit 32 receives the approximate plane data from the approximate plane creation unit 31 and updates the road surface data in the storage unit 13 with the approximate plane data.

  The swing angle calculation unit 33 obtains an angle shift ψ with respect to the xy axis on the image (on the projection plane 41) with reference to the normal vector η3 of the approximate plane. Specifically, the swing angle calculation unit 33 calculates an angle between the direction of the normal vector η3 projected onto the projection plane 41 and the x axis (or y axis), and from this angle 0 degree (or 90 degrees). The shift ψ is obtained. Further, the swing angle calculation unit 33 gives information of the angle shift ψ to the swing angle correction unit 21 when the angle shift ψ is equal to or less than the angle threshold ψth.

  This angular deviation ψ does not directly affect the determination of the three-dimensional object and the road surface in the present embodiment. However, since it takes time to attach the camera 11 to the vehicle so that the horizon of the captured image is horizontal, the image of the camera 11 may include a rotation error around the z-axis at the time of attachment. According to the swing angle calculation unit 33, information on the angle shift ψ can be easily obtained, and the rotation error of the captured image can be easily corrected by software by the swing angle correction unit 21.

  Next, an example of the operation of the vehicle image processing device 10 according to the present embodiment will be described.

  FIG. 3 is a flowchart illustrating a procedure when the CPU of the image processing ECU 12 according to the first embodiment obtains the position of the road surface in real time from the image acquired by the camera 11. In FIG. 3, reference numerals with numbers added to S indicate steps in the flowchart.

  First, in step ST <b> 1, the swing angle correction unit 21 acquires image data of a captured image from the camera 11.

  Next, in step ST <b> 2, the swing angle correction unit 21 rotationally corrects the image captured by the camera 11 about the z axis so as to correct the angle shift ψ received from the swing angle calculation unit 33.

  Next, in step ST <b> 3, the motion vector calculation unit 22 obtains a motion vector (u, v) between a plurality of images captured by the camera 11 for each pixel based on the acquired image data.

Next, in step ST4, the rotation component calculation unit 23 acquires the vanishing point information FOE (x0, y0) from the storage unit 13, and formulas for three or more motion vectors (u, v) corresponding to the stationary object. Based on (3), rotation components R _x , R _y , R _z are calculated.

Next, in step ST5, the moving object detection unit 24 determines that an object that does not converge to the vanishing point FOE is a moving object that moves differently from the own vehicle, and an object that converges to the vanishing point FOE. Judge as a stationary object. Note that steps ST4 and ST5 may be repeated a predetermined number of times. In this case, based on the information of the stationary object extracted by the moving object detection unit 24, the rotation component calculation unit 23 calculates the rotation components R _x , R _y , R _z in step ST4.

Next, in step ST6, the translation component with scale factor calculation unit 25 calculates Equation (4) based on the rotation components R _x , R _y , R _z of the vehicle and the motion vector (u, v) of the object. Then, the translation components T _x / z, T _y / z, and T _z / z with scale factors of the object are calculated from the equation (5). Moreover, the translation component with scale factor calculation unit 25 calculates the translation component with scale factor T _z / z based on the equations (9) and (10).

  Next, in step ST <b> 7, the vehicle translation component calculation unit 26 determines that the object n displayed at the coordinate value (x, y) on the image from the road surface data stored in the storage unit 13 is on a predetermined set road surface. The optical axis direction coordinate value z2n when it is assumed to be present is acquired.

Next, in step ST8, the host vehicle translation component calculation unit 26 multiplies the plurality of target vehicle translation components T _z by multiplying z2 by the scale factor translation component (T _z / z) for a plurality of objects. And the average T _z ave of these is calculated.

Next, in step ST9, the road surface detection unit 27 uses Equation (12) from the average T _z ave of the own vehicle translation component and the translation component with a scale factor (T _z / z) n of the object n. An optical axis direction coordinate value z1n of the object n is calculated.

  Next, in step ST10, the road surface detection unit 27 converts the optical axis direction coordinate value z1n of the object n and the coordinate value (xn, yn) on the image of the object n in the road surface data stored in the storage unit 13. It is determined whether or not the absolute value | z1n−z2n | of the difference from the associated optical axis direction coordinate value z2n = z (xn, yn) is equal to or less than a predetermined road surface threshold zth. If | z1n−z2n | is equal to or less than zth, it is determined that the object n is a road surface, and information on the object n is given to the road surface point coordinate calculation unit 28, and the process proceeds to step S11. On the other hand, when | z1n−z2n | is larger than zth, it is determined that the object n is not a road surface but a solid object, and this step ST10 is repeated for other objects.

  Next, in step ST11, the road surface point coordinate calculation unit 28 uses the coordinate value (xn, yn) on the image of the object n determined as the road surface by the road surface detection unit 27 and the optical axis direction coordinate value z1n. Thus, the three-dimensional coordinate value Pn (Xn, Yn, Zn) of the object n in the camera coordinate system is obtained based on the equations (13)-(15).

  Next, in step ST12, the eigenvalue / eigenvector calculation unit 29 creates a covariance matrix S from Pn (Xn, Yn, Zn) (see Expression (16)).

  Next, in step ST13, the eigenvalue / eigenvector calculation unit 29 obtains eigenvalues λ1, λ2, λ3, and eigenvectors η1, η2, η3 of the covariance matrix S. However, λ1, λ2, and λ3 are set in descending order of eigenvalues, and eigenvectors corresponding to the eigenvalues are η1, η2, and η3, respectively.

  Next, in step S14, the eigenvalue / eigenvector calculation unit 29 exists in the current image when the eigenvalues λ1 and λ2 are equal to or greater than the first eigenvalue threshold λth1 and the eigenvalue λ3 is equal to or less than the second eigenvalue threshold λth2. It is determined that the road surface 42 that can be approximated by the plane spanned by η1 and η2, and information on the eigenvalues λ1, λ2, λ3, and eigenvectors η1, η2, and η3 is provided to the approximate plane creating unit 31 and the process proceeds to step ST15. On the other hand, when the eigenvalues λ1 and λ2 are smaller than the first eigenvalue threshold λth1 or the eigenvalue λ3 is larger than the second eigenvalue threshold λth2, the eigenvalue / eigenvector calculator 29 calculates a plane on which the obtained data approximates the road surface 42. It is determined that it is insufficient for the request and discarded, and the series of procedures is completed.

  Next, in step ST15, the fixed point setting unit 30 sets the fixed point P0 (α, β, γ) using equations (17) to (19) using information on the camera height H and the vanishing point FOE. In the present embodiment, the camera height H and the vanishing point FOE are known constants stored in the storage unit 13 in advance.

  Next, in step ST16, the approximate plane creation unit 31 uses the eigenvector of the covariance matrix S to express the approximate plane formed by the object n determined by the road surface detection unit 27 as the road surface (21) or (23 ) To derive the formula (22) or the formula (24).

  Next, in step ST17, the road surface data updating unit 32 receives the approximate plane data defined by the equation (22) or the equation (24) from the approximate plane creation unit 31, and the road surface data in the storage unit 13 is used as the approximate plane data. Update with.

  Next, in step ST18, the swing angle calculation unit 33 obtains an angle shift ψ with respect to the xy axis on the image (on the projection plane 41) with reference to the normal vector η3 of the approximate plane, and the angle shift ψ is the angle threshold ψth. The information of the angle deviation ψ is given to the swing angle correction unit 21 as follows.

  With the above procedure, the position of the road surface can be obtained from the image acquired by the camera 11 in real time.

  The vehicular image processing apparatus 10 according to the present embodiment can obtain the position of the road surface in real time from the image acquired by the camera 11. For this reason, even when the road surface imaged by the camera 11 and the vehicle bottom surface are not parallel, the distance from the object can be easily obtained from the image, and the moving body can be specified.

  Therefore, according to the vehicular image processing apparatus 10 according to the present embodiment, for example, when the road surface gradient changes, the road surface directly under the vehicle and the road surface imaged by the camera have a change in gradient, or the road surface is flat. However, even when there is a vehicle inclination due to a large amount of vehicle load or a change in the number of passengers, the distance from the road surface can be easily obtained from the image, and the moving body can be specified.

  Further, the vehicle image processing apparatus 10 according to the present embodiment easily corrects the mounting angle error of the camera 11 around the z-axis in software by using information on the road surface normal vector acquired in real time. be able to.

Further, the vehicle image processing apparatus 10 according to the present embodiment can update the road surface data in the storage unit 13 using only the image data captured by the camera 11.
(Second Embodiment)

  Next, a second embodiment of the vehicle image processing device and the vehicle image processing method according to the present invention will be described.

  FIG. 4 is an overall configuration diagram showing an example of a vehicle image processing apparatus 10A according to the second embodiment of the present invention.

  The vehicle image processing apparatus 10A shown in the second embodiment is for calibrating the vehicle speed detection unit 51 that detects the speed of the vehicle and the camera height H by acquiring the output of the vehicle speed pulse and the acceleration sensor. It differs from the vehicle image processing apparatus 10 shown in the first embodiment in that the camera height calibration unit 52 is provided and the camera height H does not have to be stored in the storage unit 13 in advance. Since other configurations and operations are not substantially different from the vehicle image processing apparatus 10 shown in FIG. 1, the same components are denoted by the same reference numerals and description thereof is omitted.

The host vehicle speed detection unit 51 detects the speed of the host vehicle by acquiring the vehicle speed pulse and the output of the acceleration sensor, and acquires the host vehicle translation component T _z . This own vehicle translation speed T _z is information obtained independently of the image. The own vehicle translation component T _z is used in the road surface detection unit 27 instead of the average T _z ave of the own vehicle translation component in the first embodiment. Note that the host vehicle speed detection unit 51 may calculate the host vehicle translation component T _z from the average of the host vehicle speed over a predetermined period.

The camera height calibration unit 52 uses the relative translation components T _x / z, T _y / z, T _z / z with scale factors and the own vehicle translation component T _z detected by the own vehicle speed detection unit 51 to use the road surface. The optical axis direction coordinate value z1n of the object n in the camera coordinate system calculated by the detection unit 27 is acquired. In addition, the camera height calibration unit 52 obtains an expression obtained by substituting the expressions (17) to (19) of the fixed point P0 (α, β, γ) with respect to the expression (22) from the approximate plane creation unit 31. Here, in the present embodiment, the camera height H in the equations (17) to (19) of the fixed point P0 (α, β, γ) is treated as an unknown.

  Then, the camera height calibration unit 52 substitutes the optical axes of a plurality of objects with respect to the equation obtained by substituting the equations (17) to (19) of the fixed point P0 (α, β, γ) into the equation (22). By substituting the direction coordinate value z1, a plurality of H values are obtained, and an average of the obtained plurality of H values is set as a height H at which the camera is attached, and is stored in the storage unit 13.

  Next, an example of the operation of the vehicle image processing device 10A according to the present embodiment will be described.

  FIG. 5 is a flowchart illustrating a procedure when the CPU of the image processing ECU 12 according to the second embodiment obtains the position of the road surface in real time from the image acquired by the camera 11. In FIG. 5, reference numerals with numbers added to S indicate steps in the flowchart. Steps equivalent to those in FIG. 3 are denoted by the same reference numerals, and redundant description is omitted.

In step S <b> 21, the host vehicle speed detection unit 51 detects the speed of the host vehicle by acquiring the vehicle speed pulse and the output of the acceleration sensor, and acquires the host vehicle translation component T _z .

  In step S22, the camera height calibration unit 52 obtains the optical axis direction coordinate value z1n of the object n from the road surface detection unit 27 and the fixed plane P0 (α, β, γ) from the approximate plane creation unit 31 with respect to Expression (22). ) Are substituted for the expressions (17) to (19). The camera height calibration unit 52 uses the coordinates of the plurality of objects in the optical axis direction for the formula obtained by assigning the formulas (17) to (19) of the fixed point P0 (α, β, γ) to the formula (22). A plurality of values of H are obtained by substituting the value z1. Then, the camera height calibration unit 52 stores the average of the obtained plurality of H values as the height H at which the camera is attached, and stores it in the storage unit 13.

  The vehicle image processing apparatus 10 </ b> A according to the present embodiment can also obtain the position of the road surface in real time from the image acquired by the camera 11. For this reason, even when the road surface imaged by the camera 11 and the vehicle bottom surface are not parallel, the distance from the object can be easily obtained from the image, and the moving body can be specified.

  Therefore, according to the vehicular image processing device 10A according to the present embodiment, for example, when the road surface gradient changes and there is a change in the gradient between the road surface directly under the vehicle and the road surface imaged by the camera, or the road surface is flat. However, even when there is a vehicle inclination due to a large amount of vehicle load or a change in the number of passengers, the distance from the road surface can be easily obtained from the image, and the moving body can be specified.

  Further, the vehicle image processing apparatus 10A according to the present embodiment easily corrects the mounting angle error around the z axis of the camera 11 by software by using the information on the normal vector of the road surface acquired in real time. be able to.

Further, the vehicular image processing apparatus 10A according to the present embodiment includes the own vehicle speed detection unit 51 and can use an accurate value as the own vehicle translation speed T _z . For this reason, the mounting height H of the camera 11 can be calibrated using this accurate own vehicle translation speed T _z .
(Third embodiment)

  Next, a third embodiment of the vehicle image processing device and the vehicle image processing method according to the present invention will be described.

  FIG. 6 is an overall configuration diagram illustrating an example of a vehicle image processing device 10B according to the third embodiment of the present invention.

  The vehicular image processing apparatus 10B shown in the second embodiment includes a vehicle speed detection unit 51 that detects the speed of the vehicle by acquiring a vehicle speed pulse and an output of an acceleration sensor, and a road surface detection unit 27 on the road surface. A point on road surface reliability evaluation unit 61 that determines whether a point determined as a point is a point on the road surface using another condition is provided, and a reliability threshold Rth is further stored in the storage unit 13 in advance. The difference from the vehicular image processing apparatus 10 according to the first embodiment is that the approximate plane does not need to pass through the fixed point P0. Since other configurations and operations are not substantially different from the vehicle image processing apparatus 10 shown in FIG. 1, the same components are denoted by the same reference numerals and description thereof is omitted. Moreover, since the own vehicle speed detection unit 51 is not substantially different from the vehicle image processing apparatus 10A shown in the second embodiment, the same reference numerals are given and description thereof is omitted.

  The road surface point reliability evaluation unit 61 extracts an object whose inner product of a vector between a plurality of objects determined to be a road surface by the road surface detection unit 27 and an eigenvector η3 is a predetermined reliability threshold Rth or less. Information about the target object is given to the approximate plane creation unit 31.

  Here, calculating the inner product of the vector between the plurality of objects and the eigenvector η3 has the same meaning as the calculation of the coefficient of η3 of the KL conversion. That is, this inner product represents the magnitude of the η3 component of the vector between a plurality of objects, and can be said to indicate a difference from the plane spanned by η1 and η2. Therefore, it can be determined that a product having a large inner product has a large difference from the plane. If it is ideally located on a plane, this inner product is zero.

The approximate plane creation unit 31 receives the normal vector η3 (or η1 × η2) from the eigenvalue / eigenvector calculation unit 29 and defines the next approximate plane equation orthogonal to η3 using the unknown constant d.
η31X + η32Y + η33Z = d (26)

Further, the approximate plane creation unit 31 substitutes the three-dimensional coordinate value Pn (Xn, Yn, Zn) of the object received from the road surface point reliability evaluation unit 61 into the equation (26), thereby obtaining the unknown constant d. Find the value. At this time, the average value dave of d may be calculated using as many objects as possible. By using this dave, the equation (26) can be written as follows.
η31X + η32Y + η33Z = dave (27)

Then, the approximate plane creation unit 31 organizes the expression (27) using the expressions (13) to (15), and obtains the following expression (28) representing the approximate plane data as z = z2.
z2 = fdave / (η31x + η32y + η33f) (28)

  The road surface data update unit 32 receives the approximate plane data represented by the equation (28) from the approximate plane creation unit 31 and updates the road surface data in the storage unit 13 with the approximate plane data.

  Next, an example of the operation of the vehicle image processing device 10B according to the present embodiment will be described.

  FIG. 7 is a flowchart showing a procedure when the CPU of the image processing ECU 12 according to the third embodiment obtains the position of the road surface in real time from the image acquired by the camera 11. In FIG. 7, reference numerals with numbers added to S indicate steps in the flowchart. Steps equivalent to those in FIGS. 3 and 5 are denoted by the same reference numerals, and redundant description is omitted.

  In step ST31, the approximate plane creating unit 31 uses Formula (26) representing an approximate plane orthogonal to η3 using the unknown constant d.

  Next, in step ST32, the road surface point reliability evaluation unit 61 has an inner product of a vector between a plurality of objects determined as a road surface by the road surface detection unit 27 and an eigenvector η3 is equal to or less than a predetermined reliability threshold Rth. An object is extracted, and information on the extracted object is given to the approximate plane creation unit 31.

  Next, in step ST33, the approximate plane creating unit 31 substitutes the three-dimensional coordinate value Pn (Xn, Yn, Zn) of the object received from the road surface point reliability evaluation unit 61 into the equation (26). The value of the unknown constant d is obtained. Then, the approximate plane creation unit 31 organizes the expression (27) using the expressions (13) to (15), and establishes an expression (28) representing the approximate plane data.

  Next, in step ST34, the road surface data update unit 32 receives the approximate plane data represented by the equation (28) from the approximate plane creation unit 31, and updates the road surface data in the storage unit 13 with the approximate plane data.

  The vehicular image processing apparatus 10B according to the present embodiment can also obtain the position of the road surface in real time from the image acquired by the camera 11. For this reason, even when the road surface imaged by the camera 11 and the vehicle bottom surface are not parallel, the distance from the object can be easily obtained from the image, and the moving body can be specified.

  Therefore, according to the vehicular image processing device 10B according to the present embodiment, for example, when the road surface gradient is changed and there is a change in the gradient between the road surface directly under the vehicle and the road surface imaged by the camera, or the road surface is flat. Even in the case where there is a vehicle inclination due to a large amount of vehicle load or a change in the number of passengers, the distance from the object can be easily obtained from the image, and the moving body can be specified.

  Further, the vehicle image processing apparatus 10B according to the present embodiment easily corrects the mounting angle error of the camera 11 around the z-axis in software by using information on the road surface normal vector acquired in real time. be able to.

  Further, the vehicular image processing apparatus 10B according to the present embodiment includes a road surface point reliability evaluation unit 61, and further selects an object that is determined as a road surface by the road surface detection unit 27. For this reason, an expression representing an approximate plane closer to the actual road surface can be established.

  Further, the formula representing the approximate plane set up by the approximate plane creation unit 31 does not include information on the fixed point P0. For this reason, even when the road surface imaged by the camera 11 is a plane that does not pass through the fixed point P0 regardless of the starting position of the slope as viewed from the camera 11, an expression that accurately represents the plane that approximates the road surface can be established. . In addition, the road surface data stored in the storage unit 13 can be updated with road surface data (approximate plane data) corresponding to the road surface. Therefore, the distance from the road surface can be determined more accurately than in the vehicular image devices 10 and 10A according to the first embodiment and the second embodiment, and the mobile object, the three-dimensional object, and the road surface can be specified more accurately. It can be carried out.

  In addition, although some embodiment of this invention was described, these embodiment is shown as an example and is not intending limiting the range of invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  Further, in the embodiment of the present invention, each step of the flowchart shows an example of processing that is performed in time series in the order described. The process to be executed is also included.

DESCRIPTION OF SYMBOLS 10 Vehicle image processing apparatus 11 Camera 12 Image processing ECU
DESCRIPTION OF SYMBOLS 13 Memory | storage part 21 Swing angle correction | amendment part 22 Motion vector calculation part 23 Rotation component calculation part 24 Moving body detection part 25 Translation component calculation part 26 with a scale factor Own vehicle translation component calculation part 27 Road surface detection part 28 On-surface point coordinate calculation part 29 Eigenvalue / eigenvector calculation unit 30 Fixed point setting unit 31 Approximate plane creation unit 32 Road surface data update unit 33 Swing angle calculation unit 41 Camera projection plane 42 Road surface 43 Plane directly under the vehicle 51 Own vehicle speed detection unit 52 Camera height calibration unit 61 Point on road surface Reliability Evaluation Department

Claims (11)

A camera provided in the vehicle for imaging the periphery of the vehicle;
A translation component with a scale factor that calculates a relative translation component with a scale factor based on a motion vector corresponding to an object included in an image captured by the camera and a rotation component of the vehicle;
A storage unit that stores road surface data that is data in which a coordinate value on the image in the camera coordinate system and a coordinate value in the optical axis direction are associated with each other on a predetermined set road surface;
While calculating the optical axis direction coordinate value of the object in the camera coordinate system using the relative translation component with the scale factor and the translation component of the own vehicle, the coordinate value on the image of the object in the road surface data And a road surface detection unit that determines that the object is a road surface when a difference between the optical axis direction coordinate values is equal to or less than a predetermined road surface threshold value,
A vehicle image processing apparatus comprising: This approximate plane is established by formulating an approximate plane formed by the plurality of objects using eigenvectors of a covariance matrix of three-dimensional coordinate values in the camera coordinate system of the plurality of objects determined as the road surface. An approximate plane creation unit for obtaining approximate plane data in which the coordinate value on the image in the camera coordinate system and the optical axis direction coordinate value are associated with each other on the upper point;
A road surface data update unit that updates the road surface data of the storage unit with the approximate plane data so that the predetermined set road surface is the approximate plane;
The vehicular image processing apparatus according to claim 1, further comprising: The covariance matrix is
Having first, second, and third eigenvalues in descending order of values, and having first, second, and third eigenvectors corresponding to each eigenvalue,
The approximate plane creation unit
When the first and second eigenvalues of the covariance matrix are greater than or equal to a first eigenvalue threshold and the third eigenvalue is less than or equal to a second eigenvalue threshold, Formulating a plane representing a plane orthogonal to the third eigenvector corresponding to the eigenvalue of 3;
The image processing apparatus for vehicles according to claim 2. The approximate plane creation unit
The approximate plane using the expression representing the approximate plane, the height at which the camera is mounted as viewed from the road surface directly below the vehicle, and the intersection of the vertical line drawn from the center of the camera and the road surface directly below the vehicle Ask for data,
The vehicular image processing apparatus according to claim 3. An optical axis direction coordinate value associated with a coordinate value on the image of the object is acquired from the road surface data stored in the storage unit, and the optical axis direction coordinate value is multiplied by the relative translation component with the scale factor. Accordingly, the vehicle translation component calculation unit that calculates the translation component of the vehicle based on the image and supplies the translation component to the road surface detection unit,
The vehicle image processing device according to claim 4, further comprising: A vehicle speed detection unit that detects the translation component of the vehicle and gives it to the road surface detection unit;
An optical axis direction coordinate value of the object in the camera coordinate system calculated using the relative translation component with the scale factor and the translation component of the host vehicle detected by the host vehicle speed detection unit is the road surface detection unit. And a three-dimensional coordinate value in the camera coordinate system that is an intersection of a vertical line drawn from the center of the camera with respect to an expression representing a plane orthogonal to the third eigenvector and the road surface directly under the vehicle. Is obtained from the approximate plane creation unit by substituting the three-dimensional coordinate value of the intersection point, which is expressed as an unknown, the height at which the camera is mounted as viewed from the road surface directly under the vehicle, A plurality of unknown values are obtained by substituting the optical axis direction coordinate values of the target object, and an average of the obtained plural values is a height at which the camera is attached And Tadashibu,
Further comprising
The approximate plane creation unit
Using the expression representing the approximate plane, the height at which the camera calculated by the height calibration unit is attached, and the intersection of the vertical line drawn from the center of the camera and the plane directly under the vehicle, Find approximate plane data,
The vehicular image processing apparatus according to claim 3. A vehicle speed detection unit that detects the translation component of the vehicle and gives it to the road surface detection unit;
A point reliability evaluation unit on the road surface for extracting the object whose inner product of a vector between the plurality of objects determined to be the road surface and the third eigenvector is equal to or less than a predetermined reliability threshold;
Further comprising
The approximate plane creation unit
As an equation representing the approximate plane, an equation representing a plane orthogonal to the third eigenvector corresponding to the third eigenvalue and having one unknown constant is established, and the road surface point reliability evaluation unit is established for this equation The approximate plane data is obtained by substituting the three-dimensional coordinate values of the plurality of objects extracted by obtaining the unknown constant value.
The vehicular image processing apparatus according to claim 3. A moving body detection unit that determines the object as a stationary object when the motion vector corresponding to the object corrected with the rotation component of the vehicle is directed to a vanishing point on the image;
Further comprising
The translation factor calculation unit with a scale factor is
Calculating the relative translation component with the scale factor for the object that is determined as the stationary object by the moving object detection unit;
The road surface detection unit is
Performing the determination on the object determined as the stationary object by the moving object detection unit;
The vehicle image processing device according to claim 1. A swing angle correction unit that rotationally corrects the image captured by the camera in accordance with the direction of the third eigenvector projected in the image captured by the camera;
The vehicular image processing apparatus according to claim 3, further comprising: A camera provided in the vehicle images the periphery of the vehicle;
Calculating a relative translation component with a scale factor based on a motion vector corresponding to an object included in an image captured by the camera and a rotation component of the vehicle;
Storing road surface data, which is data in which a coordinate value on the image in the camera coordinate system and a coordinate value in the optical axis direction are associated with each other on a predetermined set road surface, in a storage unit;
While calculating the optical axis direction coordinate value of the object in the camera coordinate system using the relative translation component with the scale factor and the translation component of the own vehicle, the coordinate value on the image of the object in the road surface data Obtaining the optical axis direction coordinate value associated with the storage unit from the storage unit, and determining that the object is a road surface when the difference between the optical axis direction coordinate values is equal to or less than a predetermined road surface threshold value;
This approximate plane is established by formulating an approximate plane formed by the plurality of objects using eigenvectors of a covariance matrix of three-dimensional coordinate values in the camera coordinate system of the plurality of objects determined as the road surface. Obtaining approximate plane data associating a coordinate value on the image and an optical axis direction coordinate value in the camera coordinate system for the upper point;
Updating the road surface data of the storage unit with the approximate plane data so that the predetermined set road surface is the approximate plane;
A vehicle image processing method comprising: The covariance matrix is
Having first, second, and third eigenvalues in descending order of values, and having first, second, and third eigenvectors corresponding to each eigenvalue,
The step of obtaining the approximate plane data includes:
When the first and second eigenvalues of the covariance matrix are greater than or equal to a first eigenvalue threshold and the third eigenvalue is less than or equal to a second eigenvalue threshold, Establishing a formula representing a plane orthogonal to the third eigenvector corresponding to the eigenvalue of 3;
The vehicle image processing method according to claim 10.

Images