JP2002034035A - Image converting method and on-vehicle circumference monitoring device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an on-vehicle circumference monitoring device using an image converting method, by which a bird's-eye view which is free of sense of incongruity can be produced, even by means of a single-lens camera. SOLUTION: This on-vehicle circumference monitoring device produces bird' s-eye view, by converting an actual image taken with an actual camera 2 oriented to look an object from a certain direction into a virtual image taken with a virtual camera 3, oriented virtually to view the object from another direction and displays the view. The actual camera 2 is a digital animation camera 20, which is mounted on a vehicle 8 and photographs the circumference of the vehicle 8, and the vehicle 8 is provided with a speed detecting device 21 which detects the speed of its own vehicle 8, an image information processor 22 which produces the bird's-eye view by converting the actual image taken with the actual camera 2 into the virtual image taken with the virtual camera 3, and a display device 24 which displays bird's-eye view. The image information processor 22 produces the bird's-eye view, by computing height information contained in the actual image taken with the camera 2, based on the speed information obtained by means of the speed detecting device 21.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention virtually arranges a real image photographed by a real camera arranged so as to look at a photographing object from one direction so as to look at the photographing object from another direction. The present invention relates to an image conversion method for converting an image into a virtual image captured by a virtual camera, and an in-vehicle surrounding monitoring apparatus using the method.

[0002]

2. Description of the Related Art Conventionally, as shown in FIG. 1, a real image taken by a real camera 2 arranged so as to look at a photographing target surface 1 from one direction is viewed from another direction. As described above, there is known an image conversion method of converting a virtual image taken by a virtual camera 3 arranged virtually. This image conversion method is used, for example, in an in-vehicle periphery monitoring device that captures a blind spot area at the rear of a vehicle and creates an overhead view.

[0003]

When creating a bird's-eye view by this type of image conversion, the problem is how much distortion is caused by the image conversion.

[0004] First, assuming that the imaging target surface 1 is a flat surface, the degree of distortion will be considered.

In FIG. 1, a photographing optical axis O1 of the real camera 2 is shown.
And the shooting optical axis O2 of the virtual camera 3 correspond to the shooting target plane (X-Y
Coincides with the origin O of the (plane) 1 and the imaging optical axis O1 and the imaging optical axis O
2 is located in a YZ plane orthogonal to the imaging target surface 1,
The depression angle between the photographing optical axis O1 and the Z axis is θr, and the photographing optical axis O2
Θi, the distance from the origin O to the center of the taking lens of the real camera 1 is Rr, and the distance from the origin O to the center of the taking lens of the virtual camera 3 is Ri. And
It is assumed that the two cameras 2 and 3 shoot images while focusing on the origin O. That is, it is assumed that the origin O is imaged by the cameras 2 and 3 so that the camera is not out of focus. Also,
fr is the focal length of the real camera 2, and fi is the focal length of the virtual camera 3.

A real image (Xr,
Yr) is converted to a virtual image (Xi, Y
The following method is used to convert to i).

First, the real image (Xr, Yr) is projected on the xy plane,
Assuming that the projected image projected on the xy plane is captured by the virtual camera 3, a mapping to the (Xi, Yi) plane is created.

Thus, the amount of image distortion when three-dimensional information is not included in the imaging information can be evaluated.

Under the conditions shown in FIG. 1, the following equation was obtained. x = Rr ・ Xr ・ sin (θr) / (-Yr ・ cos (θr) + frsin (θr)) (1) y = Rr ・ Yr / (-Yr ・ cos (θr) + frsin (θr)) Xi = fi ・ Rr ・ Xr ・ sin (θr) / (-Ri ・ Yr ・ cos (θr) + Rr ・ Yr ・ cos (θi) + fr ・ Ri ・ sin (θr)) (2) Yi = fi ・ Rr ・ Yr・ Sin (θi) / (− Ri ・ Yr ・ cos (θi) + Rr ・ Yr ・ cos (θi) + fr ・ Ri ・ sin (θr)) Equation (1) expresses the real image of the real camera 2 in the xy plane. Equation (2) is based on this, and the virtual camera 3
This is an expression for converting to a virtual image.

However, if this equation (2) is used as it is,
Since only the number of pixels of the real image of the real camera 2 is converted into the virtual image of the virtual camera 3, a gap is formed in a region where the area is enlarged in the image conversion. Therefore, the inverse transformation of the equation (2) is performed to derive the equation (3), and the pixel of the virtual camera 3 is made to correspond to the real image of the real camera 2 to obtain an image without gaps. Xr = fr ・ Ri ・ Xi ・ sin (θi) / (Ri ・ Yi ・ cos (θr) -Rr ・ Yi ・ cos (θi) + fi ・ Rr ・ sin (θi)) (3) Yr = fr ・ Ri ・Yi · sin (θr) / (Ri · Yi · cos (θr) −Rr · Yi · cos (θi) + fi · Rr · sin (θi)) In other words, the real camera 2 and the virtual camera 3 1, the depression angle θ between the real camera 2 and the virtual camera 3
Since r and θi are different, there is an enlarged area where the image area is enlarged when an image captured by the real camera 2 is converted into a virtual image captured by the virtual camera 3. Are generated.

Therefore, in the enlarged area where the image area is enlarged by converting the real image to the virtual image, the inverse conversion property that the image area is reduced when inversely converting the virtual image to the real image is used. Each pixel of the virtual camera 3 is made to correspond to each pixel of the real camera 2, and an image without gaps is created when the image is converted from a real image to a virtual image using the pixels.

FIG. 2 shows an example in which the real camera 2 uses the real camera (digital camera) 2 to set the depression angle θr at 45 degrees and the distance Rr.
Is set to about 150 cm, and the floor of the room is set as the photographing target surface 1 and photographed. Also, the focal length f of the real camera 2
For r, a catalog value of a digital camera was used. Real camera 2
Has a general value of 10 μm.

On the floor 4 ', there is a calendar 4, on the right side of which a trash can 5 is placed. By converting this image using an appropriate parameter, an overhead view shown in FIG. 3 is obtained. FIG. 3 is a view seen from directly above (θi = 90 °).
Has been converted to. The distance Ri of the virtual camera 3 was set so that the entirety could be seen.

The trapezoidal floor mesh as shown in FIG. 2 has been transformed into a quadrilateral as shown in FIG. 3 by image conversion, and has been appropriately converted using equations (2) and (3). I can understand.

Here, for example, when considering the use as a back-eye camera of a vehicle, there is a possibility that the trash can 5 on the side may obstruct the understanding of the surrounding situation obtained through the image. Since the lid of the trash can 5 is opened outside the actual position, it cannot be intuitively understood, and only the position of the grounding surface of the trash can 5 can be trusted.

It is difficult to obtain an easy-to-understand image only by simple conversion between planes. In order to properly reproduce a horizontal object, a distance map indicating the distance to the real camera 2 for each pixel is required. felt.

In other words, according to this image conversion method, an almost faithful bird's-eye view can be created without any discomfort for a two-dimensional object such as paint on the ground.

However, a three-dimensional object to be photographed becomes a distorted figure as a bird's-eye view when image conversion is performed.
It is difficult to create a bird's-eye view without discomfort by image conversion.

In particular, when there are obstacles on the left and right of the rear part of the vehicle, it is difficult to identify the position of the obstacle on the screen of a display device (monitor) that displays a bird's-eye view, and the practical use of an on-vehicle surrounding monitoring device is aimed at. This is a problem to be overcome.

This problem can be solved by knowing the three-dimensional positional relationship between the object to be photographed and the real camera. However, this requires a range finder device or the like. There is a problem that the method is mainly based on stereo vision, which is expensive, and is expensive.

The present invention has been made in view of the above circumstances, and provides an image conversion method capable of creating a bird's-eye view without a sense of incongruity even with a single-lens camera, and a vehicle surrounding monitoring apparatus using the image conversion method. It is in.

[0022]

According to a first aspect of the present invention, there is provided an image conversion method, comprising: converting an actual image taken by a real camera arranged so as to view an object to be photographed from a certain direction; In an image conversion method for converting a virtual image photographed by a virtual camera virtually arranged to look at an object, the photographing is performed based on installation data of the real camera on a vehicle and optical characteristic data of the real camera. An image conversion method characterized by analyzing moving image information obtained by moving an object relative to the real camera to obtain three-dimensional information of an image.

According to a second aspect of the present invention, there is provided an image conversion apparatus which converts a real image taken by a real camera arranged to look at a photographing object from a certain direction so as to look at the photographing object from another direction. In an in-vehicle surrounding monitoring device that converts a virtual image taken by a virtual camera arranged in a vehicle and creates and displays a bird's-eye view, a digital type in which the real camera is mounted on a vehicle and images the surroundings of the vehicle A video camera, wherein the vehicle is a vehicle speed detection device that detects a vehicle speed of the own vehicle, and image information that creates a bird's-eye view by converting a real image taken by the real camera into a virtual image taken by the virtual camera. A processing device and a display device for displaying the overhead view, wherein the image information processing device is photographed by the real camera based on vehicle speed information obtained by the vehicle speed detection device. And calculates the height information included in the image, characterized in that to create the overhead view.

According to a third aspect of the present invention, the vehicle is provided with a steering angle detecting device for detecting a steering angle of a steering wheel, wherein the vehicle obtains steering angle information obtained by the steering angle detecting device and the vehicle speed detecting device. The overhead view is created by calculating height information included in a real image captured by the real camera when the vehicle turns, based on the obtained vehicle speed information.

The vehicle surrounding monitoring device according to claim 4, wherein the vehicle temporarily stores an image processed by the image information processing device when the vehicle stops and displays the image on the display device. -It is characterized by having a selector device.

According to a fifth aspect of the present invention, in the on-vehicle surrounding monitoring apparatus, the image buffer selector detects an image change based on an object approaching into an image pickup area of the real camera while the vehicle is stationary, and detects the change in the real camera. The captured image is displayed on the display device.

According to a sixth aspect of the present invention, in the vehicle surroundings monitoring apparatus, the image information processing apparatus includes a distance data from the center of the vehicle to the center of the image pickup area, an installation data of the real camera with respect to the vehicle, and the real data. The pixel size data and the focal length data of the camera are prepared in advance, and the image information processing apparatus performs optical flow detection based on a moving image captured by the digital moving image camera to generate an optical flow map of the entire screen. Flow map generation means, height information map generation means for calculating height information in real space for each pixel based on the optical flow map, the data and the vehicle speed, to generate a height information map, A creating unit for creating the overhead view from the height information map and the digital data of the moving image; And wherein the Rukoto.

According to a seventh aspect of the present invention, in the vehicle surrounding monitoring apparatus, the image information processing apparatus prepares, in advance, installation data of the real camera with respect to the vehicle, pixel size data of the real camera, and focal length data, The image information processing apparatus performs an optical flow detection based on a moving image captured by the digital video camera to generate an optical flow map of the entire screen, an optical flow map generating unit, the optical map, the vehicle speed, and each of the above. A turning radius calculating means for obtaining a turning radius and a turning direction from the data, a turning radius information orthogonal to the turning radius of the vehicle based on the turning radius information obtained by the turning radius calculating means, the optical flow map and the data; Angle calculating means for calculating a shift angle between a tangent line to be formed and the optical axis of the real camera Calculating height information in a real space for each pixel based on the optical flow map, the respective data, the shift angle obtained by the turning radius calculator and the shift angle calculator, and the vehicle speed. It is characterized by comprising height information map generating means for generating an information map, and generating means for generating the overhead view from the height information map and the digital data of the moving image.

The vehicle surroundings monitoring device according to claim 8, wherein the vehicle is provided with a steering angle detecting device for detecting a steering angle of a steering wheel, and the image information processing device includes a setting data of the actual camera with respect to the vehicle. Pixel size data of the actual camera, focal length data, and deviation angle relation data between a deviation angle and a steering angle between a tangent perpendicular to the turning radius of the vehicle and the optical axis of the real camera are prepared in advance. The image information processing apparatus performs an optical flow detection based on a moving image captured by the digital moving image camera to generate an optical flow map of the entire screen, and a steering device of the steering angle detection device. Based on the deviation angle data obtained based on the angle, the vehicle speed, the data, and the optical flow map, each pixel in real space is Height information map generating means for calculating height information to calculate height information map, and creating means for creating the overhead view from the height information map and the digital data of the moving image. Features.

The image monitoring apparatus according to claim 9, wherein the vehicle temporarily stores an image processed by the image information processing device when the vehicle stops and displays the image on the display device. A selector device, wherein the image information processing device compares the optical flow map when the vehicle is stationary with the optical flow map while the vehicle is running based on the optical flow map, detects a significant image change, and detects the image buffer. An image change detecting means for transmitting an image change detection signal to the selector device, wherein the image buffer selector device is configured to control the image while the vehicle is stationary and the virtual image obtained immediately before the stop based on the image change detection signal; And transmitting to the display device.

According to a tenth aspect of the present invention, there is provided
The installation data is data relating to a mounting height of the real camera to the vehicle, a depression angle with respect to the ground, and a distance from the ground to the real camera on the optical axis center line of the real camera.

[0032]

DESCRIPTION OF THE PREFERRED EMBODIMENTS (Principle of Acquiring Three-Dimensional Information Based on Real Image) (Linear Motion) FIG. 4 shows a case where the coordinate system is fixed to the own vehicle and the motion of the own vehicle is replaced with the relative motion of the surroundings. FIG. 3 is a model diagram showing a state in which a moving object is imaged by the real camera 2. As shown in FIG. 4, a rectangular object 6 having a certain height
Is moving in the z-axis direction on the xz surface, is at a position shown by a solid line at a certain time t1, and is at a position shown by a broken line at a time t2. Here, the symbols a and b indicate the corner points of the rectangular object.

At this time, it is assumed that the real image taken by the real camera 2 at the time t1 is as shown in FIG. In FIG. 5, reference numeral 7 denotes a real image of the rectangular object 6, and reference numeral 8 denotes a screen. At time t2, the rectangular object 6 approaches the real camera 2 and moves from the upper side to the lower side of the screen 8, so that the real image 7 captured by the real camera 2 at time t2 is as shown in FIG. It becomes something. As the rectangular object 6 approaches the real camera 2, the real image 7 also becomes larger.

The real image 7 moves from the position shown by the broken line to the position shown by the solid line between the time t1 and the time t2, whereby the corner points a and b on the screen 8 become the corner points a '. , B '
Go to Vectors A and B indicating the moving directions and the magnitudes of the corner points a and b are called optical flow (OF) vectors when handling moving images.

This OF vector depends not only on the position in the plane xz but also on the height. Therefore, the height of the object can be evaluated by analyzing the amount of the OF vector. Here, the amount when the real image 7 moves during one frame of the moving image is defined as an OF vector. That is, one frame time is considered as a unit time.

In FIG. 4, the height from the photographing target surface 1 to the center of the photographing lens of the real camera 2 is H. The depression angle is θ, and the distance from the origin O to the center of the taking lens is R.
r, the focal length of the real camera 2 is f, and the point of the rectangular object 6
The projection of the point (x, z, h) onto the point (x, y) of the real image is given by the following relational expression.

X = 2fxsin [θ] / (− h + 2H + hcos [2θ] + zsin [2θ]) (1.1) Y = f (hcos [θ] + zsin [θ]) / (cos [θ] ( z + Hcot [θ]) + (− h + H) sin [θ]) (1.2) Both of the expressions (1.1) and (1.2) include the height information h.

This inverse conversion equation is described below.

X = (hH) XX / (YYcos [θ] -fsin [θ]) (2.1) y = cos [θ] (fh-HYYcot [θ]) + (hH) YYsin [θ] / (YYcos [ θ] -fsin [θ]) (2.2) Consider a case where the point (x, z, h) of the object 6 is only translation (Δx, Δz, 0). At this time, the OF vector Q (ΔX, ΔY) in the screen
Is obtained by differentiating equations (1.1) and (1.2).

ΔX = (2fsin [θ] / (− h + 2H + hcos [2θ] + zsin [2θ])) Δx + (− 4fxcos [θ] sin [θ] ² / (− h + 2H + hcos [2θ ] + zsin [2θ]) ² ) Δz (3.1) ΔY = (-4f (hH) sin [θ] ² / (-h + 2H + hcos [2θ] + zsin [2θ]) ² ) Δz (3.2) However, in the case of the measurement of the OF vector using an actual moving image, since a discrete image is used, there may be a case where an error occurs with respect to the differential amount of (3.1) and (3.2), and correction may be necessary. It is assumed that the resolution is sufficiently high and high-speed shooting is performed without any error.

Under this condition, assuming that the object moves (Δx, Δz, 0) during one frame, equations (3.1) and (3.2)
Gives an OF vector Q (ΔX, ΔY).

In the back-eye camera that mounts the real camera 2 on the vehicle and captures an image of the rear part of the vehicle, the real camera 2 moves with respect to the rectangular object 6. Since the movement relation is relative, equations (3.1) and (3.2)
Can be basically used.

Focusing on a point (X, Y) on the screen assuming a linear motion, this point (x, y) is expressed in real space by the following equations (2.1) and (2.2).
(X, z, h). Now, assuming that the own vehicle moves linearly and moves (0, −Δz, 0) during one frame, the rectangular object 6 has moved (0, + Δz, 0). This Δz is a known value obtained from the vehicle speed sensor. At this time, the OF vector Q (ΔX, ΔY) is calculated by the equation (2.
From (1), (2.2), and equations (3.1) and (3.2), ΔX = (Xcos [θ] (-Ycos [θ] + fsin [θ]) / f (hH)) Δz (4.1) ΔY = (-( Ycos [θ] -fsin [θ]) ² / f (hH)) Δz (4.2).

Accordingly, the height h is obtained from the equations (4.1) and (4.2).

H = (ΔXfH−ΔzXYcos [θ] ² + ΔzfXcos [θ] sin [θ]) / ΔXf (5.1) h = (ΔYfH−ΔzY ² cos [θ] ² + 2ΔzfYcos [θ] sin [θ] -Δzf ² sin [θ] ² ) / ΔYf (5.2) Although the values of h in these two equations should originally match, they do not match because there is an error in the measurement of the OF vector, etc. It is necessary to make various measures such as creating a virtual plane for performing smoothing and high-speed operation using the value of, and viewing the difference from this virtual plane.

For example, the formulas (4.1) and (4.2) have a common term related to the variable Y. By eliminating this, it is possible to consider formulating the formulas (5.1) and (5.2) into one equation. Can be

When the equations (5.1) and (5.2) are solved by the equations, the height to be obtained is h = H + (ΔYX ² cos [θ] ² / ΔX ² f) Δz (6)

As described above, when the vehicle speed Δz is obtained in principle, it is possible to obtain the height information of the rectangular object 6 using a moving image obtained by monocular photographing. In general, it is possible in principle to obtain three-dimensional information of a photographing target. (Rotating Motion) As shown in FIG. 7, it is assumed that the own vehicle 8 is rotating at a constant angular velocity while keeping the steering wheel at a constant angle. The real camera 2 also rotates with this. The rotation radius (distance) from the rotation center OO of the vehicle 8 to the center (origin O) of the imaging area 9 is defined as r. As shown in an enlarged manner in FIG. 8, the z-axis direction of the imaging area 9 of the real camera 2 that is performing a circular motion does not always match the tangential direction of the rotation trajectory. That is, the optical axis of the real camera 2 does not always coincide with the tangent line orthogonal to the rotation radius, and there is an axis deviation between the optical axis and the tangent line, which is defined as an axis deviation angle θz. The axis deviation angle θz is uniquely determined by the rotation center position of the vehicle 8 and the installation conditions of the camera.

Consider how the point (x, z, h) in the real space moves due to rotation. It is assumed that the height h does not change.

Assuming that the vehicle is rotating at each speed ω (the unit time is a frame rate), the origin O of the xz plane moves by an amount r · ω. This amount r · ω is almost the same as the distance that the vehicle travels per unit time, but does not match. Also,
The distance ro of each point (x, z) on the plane xz from the rotation center oo is given by the following equation.

Ro = (r ² + x ² + z ² + 2r (xcos [θz] + zsin [θz])) ^1/2 (7) Here, the moving speed of the point (x, z) is ro.ω Becomes By projecting this velocity vector onto the XY plane, an OF vector can be obtained. Observed OF vectors are linear and slightly different from ro · ω, which indicates the length of an arc, since actual imaging results in a discrete image with respect to time. Correction may be required. In the following discussion, it is assumed that an image is captured sufficiently quickly and ideal analysis is possible, and a basic expression is constructed based on a velocity vector obtained by differentiation.

Assuming that the car is stationary and the ground rotates in the opposite direction, the drawing is made as shown in FIG. At this time, the displacement (Δx, Δz) due to the movement of the point (x, z) is obtained as follows. In FIG. 9, the symbol α is the rotation center O.
This is the angle formed by the line connecting O and the displaced point (x + Δx, z + Δz) in the real space, and the line connecting the rotation center OO and the origin O.

Δx = −ω (z + rsin [θz]) (8.1) Δz = ω (x + rcos [θz]) (8.2) This equation (8.1) (8.2), equation (2.1), (2.2), From (3.1) and (3.2), the OF vector Q ′ (ΔX, ΔY) of the point (X, Y) in the image can be obtained by the following equation.

ΔX = Xωcos [θ] ((− h + H) X + rcos [θz] (− Ycos [θ] + fsin [θ])) / f (h−H) + 1 / (hH) (ω (sin [θ] ((-h + H) Y + frsin [θz])-cos [θ] (fh-HYcot [θ] + rYsin [θz]))) (9.1) ΔY = -1 / f (hH ) (ω (Ycos [θ] -fsin [θ]) ((hH) X + rcos [θz] (Ycos [θ] -fsin [θ]))) (9.2) Vector Q '(ΔX, ΔY)
Is the basic equation. This equation is the linear motion equation (4.1),
It corresponds to (4.2).

Next, a case where the height information h is obtained will be considered.

(When steering wheel angle information and vehicle speed information are given) The radius of rotation r, angular speed ω, and deviation angle θz can be obtained in principle from the size of the vehicle and the characteristics of the steering wheel. Then, the only unknown variable is the height h, and the equation (9.1),
The height h is obtained independently from (9.2). The independently determined heights h should be equal to each other if there is no measurement error.

The height h is obtained by the following equations (10.1) and (10.2).

H = (fHΔX + ω (−rXYcos [θ] ² cos [θz] + fsin [θ] (HY + frsin [θz]) + cos [θ] (HX ² + f HYcot [θ] + frXcos [ θz] sin [θ] -frYsin [θz]))) / ((f ² + X ² ) ωcos [θ] + f (ΔX + Yωs in [θ])) (10.1) h = H-rωcos [θz] (Ycos [θ] -fsin [θ]) ² / (fΔY + XYωcos [θ] -YXωsin [θ]) (10.2) (When only vehicle speed information is given at the time of rotational movement) Vehicle speed corresponds to the product of angular speed and radius of gyration. In fact, whether the obtained vehicle speed is the speed of the center of gravity of the car, the speed of the inner wheel or the outer wheel, or the speed of the front wheel or the rear wheel depends on the design of the car, so it should be considered for each individual vehicle, but If the design is clear, it is possible to convert to the speed of the origin O of the imaging region 9. Assuming that the velocity after the conversion to the origin O is v, v = r · ω (11) The simultaneous equations are established by the equation (11) and the equations (9.1) and (9.2). On the other hand, there are four unknown variables, ω, r, θz, and h, and there is not enough formula.
Can not ask.

Here, considering the relationship between the radius r and the axis angle θz, as shown in FIG. 10, the distance Rr between the rotation center OO and the real camera 2, the height of the real camera 2 from the ground, and the depression angle θ are unique. Is determined. Therefore, if w shown in FIG. 10 is given by the design drawing of the vehicle 8, θz is given by w = rsin [θz] (12).

Here, w is the distance from the vehicle center OO 'to the origin O, in FIG. 10, ra is the distance from the rotation center OO to the vehicle center OO', and θz is the rotation center OO and the origin O.
, And a line connecting the origin OO and the vehicle center OO ′.

These equations (11) and (12), equations (10.1) and (10.2)
Is used to solve the equation for the height h, the height h can be obtained in principle.

This equation becomes a quartic equation, and the algorithm for solving the quaternary equation is known. Since the coefficients of this equation are very complicated and cannot be easily expressed, only the equations will be described in Appendix 1.1 to Appendix 1.6 below.

As another method, the height h can be obtained from the rotation radius r first, and then the height h can be obtained from the rotation radius r without directly obtaining the height h.

The radius of gyration r is obtained by the following equation.

Ar (r ² -w ² ) ^1/2 + (r ² -w ² ) ^1/2 + cr + d = 0 (13.1) a = fvY ² ΔXcos [θ] ² -fvXYΔYcos [θ] ^2- 2f ² vYΔXcos [θ] sin [θ] + f ² vXΔYcos [θ] sin [θ] + f ³ vΔXsin [θ] ² (13.2) b = f ² v ² Y ² cos [θ] ³ -2f ³ v ^{2 Ycos [θ] 2 sin [θ} ] + fv 2 Y 3 cos [θ] 2 sin [θ] + f 4 v 2 cos [θ] sin [θ] 2 - 2f 2 v 2 Y 2 cos [θ] sin [ θ] ² + f ³ v ² Ysin [θ] ³ (13.3) c = -f ³ HvΔYcos [θ] -f ² vwYΔYcos [θ] + f ² HvYΔYcos [θ] cot [θ] + f ³ vwΔYsin [θ] (13.4) d = -2f ² Hv ² XYcos [θ ] ² -fv ² wXY ² cos [θ] ² + fHv ² XY ² cos [θ] ² cot [θ] + f ³ Hv ² Xcos [θ] sin [θ] + 2f ² v ² wXYcos [θ] sin [ θ] -f ³ v ² wXsin [θ] ² (13.5) By solving this equation, if the radius of gyration r is obtained, the equation (1
Other values are also obtained from 1) and Equation (12), and Equation (10.1),
Three-dimensional information can be evaluated using (10.2) and the like. The general analytical solution of this nonlinear equation has already been clarified, and if the numerical values of a, b, c, and d are known, it can be obtained simply by substituting.

As described above, whether the height h is directly obtained or the height h is obtained by solving the equation in order from the turning radius r, it is possible to obtain three-dimensional information only from the vehicle speed by image information processing. Can be. Giving only the steering angle (when only the steering angle is given) corresponds in principle to the fact that the radius of gyration of a circularly moving vehicle is known. That is, the distance r and the deviation angle θz between the imaging center O and the rotation center OO are known variables.

On the other hand, the angular velocity ω is an unknown variable. Therefore, in equations (9.1) and (9.2), simultaneous equations are solved with the height h and the angular velocity ω as unknown variables. These two equations are both linear functions of the angular velocity ω, and there is no constant term that does not include the angular velocity ω. Therefore, ΔX / ΔY is a function of only the height h independent of the angular velocity ω.

When ΔX and ΔY are independent, the ratio is d
And the following equations are obtained respectively.

H = (− HXYΔXcos [θ] + HX ² ΔYcos [θ] + rY ² ΔXcos [θ] ² cos [θz] −rXYΔYcos [θ] ² cos [θz] − fHYΔYcos [θ] cot [θ] + fHXΔXsin [θ]-fHYΔYsin [θ] -2frYΔXcos [θ] cos [θz] sin [θ] + frXΔYcos [θ] cos [θz] sin [θ] + f ² rΔXcos [θz] sin [θ] ² + fHYΔYcos θ] sin [θz] -f ² rΔYsin [θ] sin [θz]) / (-XYΔXcos [θ] -f ² ΔYcos [θ] + X ² ΔYcos [θ] + fXΔXsin [θ] -fYΔYsin [θ]) (14.1) h = (-HX ² cos [θ] + dHXYcos [θ] + rXYcos [θ] ² cos [θz] -drY ² cos [θ] ² cos [θz] + fHYcos [θ] cot [θ]- dfHXsin [θ] + fHYsin [θ] -frXcos [θ] cos [θz] sin [θ] + 2dfrYcos [θ] cos [θz] sin [θ]-df ² rcos [θz] sin [θ] ² -frYcos [ θ] sin [θz] + f ² rsin [θ] sin [θz]) / (f ² cos [θ]-X ² cos [θ] + dXYcos [θ] -dfXsin [θ] + fYsin [θ]) ( However, d = ΔX / ΔY) (14.2) This equation does not include the angular velocity ω or the velocity v, and there are no unknown variables. Therefore, even if only the steering angle is given, it is possible to acquire three-dimensional information. Is shown. (Relationship between linear interlocking and rotational motion) When linear motion is assumed, the vehicle speed, that is, the distance that one point of an object moves in one frame is Δz.
It has been shown that the height h is represented by the formulas (5.1), (5.2), and (6).

It is considered that there are various types of solutions other than the above. The linear motion and the rotational motion are as follows: v = Δz = r · ω (15.1) r → ∞ (15.2) ω → 0 (15.3) θz It is thought that they are connected by the conversion relation of → 0 (15.4).

The basic equations (9.1) and (9.
When this transformation is applied to 2), equations (4.1) and (4.2) for linear movement can be derived.

The formula for calculating the height h is also given by the formula (1)
By applying this transformation to (0.1) and (10.2), equations (5.1) and (5.2) can be derived. Therefore, by carefully handling infinity, the calculation of the linear motion and the rotational motion can be performed seamlessly.

First, consider the case where only the speed is given.

If only the speed is given, no information on rotation is included in the equation, so it is only necessary to focus on finding a mathematically significant solution. In contrast, the turning radius r
Is obtained, the equation (13.1) gives the OF vector (Δ
(X, ΔY), the solution of r is ズ. Therefore, when solving the equation (13.1), it is necessary to take measures such as variable conversion so that the solution does not diverge. Also, (ΔX,
Considering that the measurement error is included in ΔY), the error included in the solution may increase near the divergence of the solution. Therefore, various contrivances are required.For example, in order to avoid this problem without dealing with it, it is necessary to judge rotation and rectilinear motion before performing calculations, and to use any one of the equations. Might be. However, because the formulas to be handled suddenly change, some means of avoiding discontinuous conversion in time series will be required. For this determination, statistical processing of the OF vector on the z-axis would be effective, or it would be possible to derive a determination equation by analyzing equation (13.1). For example, OF vector equation (4.1), (4.2) in linear motion
By substituting into Equation (13.2), a = 0, so this can be used in the determination equation.

On the other hand, when only the steering angle is given, the equation (14.
1) Even if (14.2) is similarly deformed, the height h during linear motion
Is not obtained. This is because there is no distance r in the denominator of the equations (14.1) and (14.2), and when the distance r is diverged, only the numerator term diverges and no solution exists. If this is traced back, there is a problem in the equation when ΔX / ΔY is calculated from Equations (9.1) and (9.2). When the distance r is diverged, ΔX / ΔY = (rXYcos [θ] ² + f (hH) Ysin [θ] + cos [θ] (f ² h + hx ² -HX ² -fHYcot [θ] -frXsin [θ])) / ((Ycos [θ] -fsin [θ]) (hX -HX + rYcos [θ] -frsin [θ])) (16.1) (ΔX / ΔY) _x → ∞ = Xcos [θ] / (Ycos [θ] -fsin [θ]) (16.2) ), The equation does not include the height h. Equations for linear motion (4.1), (4.2)
The same equation (16.2) can be obtained by applying the same processing to Conversely, the aspect ratio of the OF vector when performing linear motion is
It can be understood that it does not depend on h. That is, in such a series of calculations, it is suggested that the absolute value of the OF vector becomes important.

In summary, if all the information is available, it is possible to acquire three-dimensional information under any motion state by using equations (10.1) and (10.2). Even when only the speed information is known, the turning radius r can be obtained by evaluating the expressions (13.1) to (13.5), and finally, the expression (10.
It can be seen that three-dimensional information can be obtained by using (1) and (10.2).

However, in the linear motion state, the equation (13.1)
Since (13.5) seems to be sensitive, it is necessary to consider the use of equations for linear motion. On the other hand, under the condition that only the steering angle is given, the height h can be obtained using the equations (14.1) and (14.2) only when the vehicle is rotating, but the same method is used when the vehicle is performing a linear motion. Cannot evaluate 3D information. Even if linear motion is assumed, the height h cannot be obtained without vehicle speed information from the basic equations (4.1) and (4.2) for that. (Rotation direction) The vehicle 8 rotating to the left has been modeled and various evaluations have been made. Here, the case of clockwise rotation is considered.

Since the coordinate system of the real camera 2 is desired to be fixed, the sign of the expression is partially changed between the right rotation system and the left rotation system, and unfortunately the same expression cannot be used as it is and another expression must be prepared.

Note that the switching of formulas is performed by using OF
It can be determined from the vector and the vehicle speed information. That is, it is possible to determine the rotation direction based on whether the average OF vector of the entire screen is to the left or right, and information on the positive or negative of the vehicle speed.

When assembling the formula, various methods can be considered for obtaining the coordinate axes. Here, the method will be described with reference to FIG. The x-z axis is unified with the case of the left rotation, and the angles α and θz are positive values. The angular velocity ω of the rotation of the vehicle 8 is defined as positive in the rotation in the direction of the arrow in FIG. At this time, the equations (1) to (6) during the linear motion do not change.

On the other hand, the equations (7), (8.1) and (8.2)
Is ro = (r ² + x ² + z ² + -2r (xcos [θz] -zsin [θz])) ^1/2 (17) ΔX = ω (z + rsin [θz]) (18.1) Δz = -xω + rωcos [θz] (18.2)

As a result, the basic simultaneous equations during rotational motion
The equations corresponding to (9.1) and (9.2) are: ΔX = Xωcos [θ] ((hH) X + rcos [θz] (− Ycos [θ] + fsin [θ])) / f (hH) + 1 / (-h + H) (ω (sin [θ] ((-h + H) Y + frsin [θz])-cos [θ] (fh-HYcot [θ] + rYsin [θ]))) (19.1) ΔY = -1 / f (hH) (ω (Ycos [θ] -fsin [θ]) ((-h + H) X + rcos [θz] (Ycos [θ] -fsin [θ]))) (19.2 ).

Using this equation, a solution corresponding to each condition is obtained.

When expressions for obtaining the height h when all information is given are obtained independently, h = (− fHΔX + ω (rXYcos [θ] ² cos [θz] + fsin [θ] (HY + frsin [θz]) + cos [θ] (HX ² + fHYcot [θ]-fr (Xcos [θz] sin [θ] + Ysin [θz]))))) / (-fΔX + (f ² + X ² ) ωcos [θ] + fYωsin [θ]) (20.1) or h = H + rωcos [θz] (Ycos [θ] -fsin [θ]) ² / (XYωcos [θ] -f (ΔY + Xωsin [θ])) (20.2).

This corresponds to equations (10.1) and (10.2).

Given only the vehicle speed, the method for directly determining the height h, as discussed above, is not discussed here.

However, when a quartic equation of height h is obtained, the equation becomes the same as that described in Appendix 1 for both left and right rotations, and it is not necessary to judge left and right in actual calculations.

Therefore, if a method for directly obtaining the height h is used, no condition judgment is required, and it is advantageous if the expression itself is not complicated.

On the other hand, the radius of gyration r is obtained and the equations (20.1) and (2
To obtain three-dimensional information using 0.2), the following equation is solved.

Ar (r ² -w ² ) ^1/2 + b (r ² -w ² ) ^1/2 + cr + d = 0 (21.1) a = fvY ² ΔXcos [θ] ² -fvXYΔYcos [θ] ² -2f ² vYΔXcos [θ] sin [θ] + f ² vXΔYcos [θ] sin [θ] + f ³ vΔXsin [θ] ² (21.2) b = -f ² v ² Y ² cos [θ] ³ + 2f ³ v ² Ycos [θ] ² sin [θ]-fv ² Y ³ cos [θ] ² sin [θ] -f ⁴ v ² cos [θ] sin [θ] ² + 2f ² v ² Y ² cos [θ] sin [θ] ² -f ³ v ² Ysin [θ] ³ (21.3) c = f ³ HvΔYcos [θ] + f ² vwYΔYcos [θ]-f ² HvYΔYcos [θ] cot [θ] -f ³ vwΔYsin [θ ] (21.4) d = -2f ² Hv ² XYcos [θ] ² -fv ² wXY ² cos [θ] ² + fHv ² XY ² cos [θ] ² cot [θ] + f ³ Hv ² Xcos [θ] sin [θ] + 2f ² v ² wXYcos [θ] sin [θ] -f ³ v ² wXsin [θ] ² (21.5) If only the steering angle is given, the equations corresponding to equations (14.1) and (14.2) Is the following equation.

[0091] h = (- rY (YΔX- XΔY) cos [θ] 2 cos [θz] + cos [θ] (- HXYΔX + HX 2 ΔY + fHYΔYcot [θ] - frXΔYcos [θz] sin [θ] -frYΔYsin [θz]) + f (-frΔXcos [θz] sin [θ] ² + rYΔXcos [θz] sin [2θ] + sin [θ] (HXΔX + HYΔY + frΔYsin [θz]))) / ((-XYΔX + f ² ΔY + X ² ΔY) cos [θ] + f (XΔX + YΔY) sin [θ]) (22.1) = (rY (X-dY) cos [θ] ² cos [θz] + cos [θ] (HX ² -dHXY + fHYcot [θ] -frXcos [θz] sin [θ] -frYsin [θz]) + f (-dfrcos [θz] sin [θ] ² + drYcos [θz] sin [2θ] + sin [θ] ( dHX + HY + frsin [θz] ))) / ((f 2 + X (X-dY)) cos [θ] + f (dX + Y) sin [θ]) ( where d = ΔX / ΔY) (22.2 As described above, since the expression may slightly change between right rotation and left rotation, if there is no steering angle information, it is necessary to detect it from image information processing as described above.

From these theoretical considerations, the vehicle speed information, the height of the real camera 2, the depression angle of the real camera 2, the optical characteristic data (focal length data, pixel size data) of the real camera 2, and the center position O of the imaging area If the distance w to the center position OO 'of the vehicle 8 is known, three-dimensional information can be acquired in principle.

Further, a deviation angle θz between a tangent and the optical axis (line of sight) of the real camera 2 which has processed the steering angle information of the steering wheel,
Given the radius of gyration r, 3
It became clear that dimensional information could be obtained.

On the other hand, when the vehicle speed information is lost, the actual camera 2
Angle θz between the optical axis (line of sight) and the tangent, and the radius of gyration r
Is known, it is impossible to obtain three-dimensional information in the case of linear motion.

Hereinafter, a detailed embodiment of the present invention will be described. (Embodiment 1) FIG. 12 shows Embodiment 1 of the present invention.
It is a block diagram of. In FIG. 12, reference numeral 20 denotes a moving image camera as the real camera 2, 21 denotes a vehicle speed detection device, 22
Denotes an image information processing device, 23 denotes an image buffer / selector device, 24 denotes a display device (monitor device), and 25 denotes a steering angle detecting device. The moving image camera 20 is a digital type that is mounted on the vehicle 8 and captures an image around the vehicle 8. The vehicle speed detection sensor 21 detects the vehicle speed of the vehicle 8, and the image information processing device 22 has a function of converting a real image captured by the video camera 20 into a virtual image captured by the virtual camera to create an overhead view. . The image buffer selector device 23 has a function of temporarily storing an image processed by the image information processing device 22 when the vehicle stops, and displaying the image on the display device 24.

This is because the height information cannot be obtained by the image information processing device 22 when the vehicle is stationary. In order to display the bird's-eye view without discomfort even when the vehicle is stopped, The virtual image processed just before the stillness is stored in the image buffer / selector device 23 and transmitted to the display device 24.

While the vehicle is stopped, an image change due to the entry of an object into the image pickup area of the video camera 20 is detected, and the actual image captured by the video camera 20 is displayed on the display device 24.
To be displayed. In addition, even when the video camera 20 is activated while the vehicle is stopped, the actual image captured by the video camera 20 is displayed on the display device 24.

The image information processing device 22 calculates the height information included in the actual image captured by the moving image camera 20 according to the processing blocks shown in FIG. Here, installation data of the video camera 20 with respect to the vehicle 8 (height data attached to the vehicle, depression angle data with respect to the ground), pixel size data of the video camera 20,
It is assumed that focal length data, deviation angle data, rotation radius, and rotation direction data of the video camera 20 are prepared in advance. That is, the database group includes a correlation table between the rotation radius of the imaging region and the steering angle, a correlation table between the angle (deviation angle) θz made by the optical axis with respect to the rotation tangent and the steering angle, the camera height, the depression angle , Focal length, and pixel size information are stored.

The image information processing apparatus 22 has an optical flow map generating means 22a for detecting an optical flow based on a moving image and creating an optical flow map of the entire screen.

Further, the image information processing device 22 obtains the relative position, the turning radius, and the turning direction of the imaging area and the vehicle body from the own vehicle steering angle information based on the database group.

The image information processing device 22 calculates height information in an actual space for each pixel based on the data and the vehicle speed, and generates a height information map generating means 22b for generating a height information map. Having. For generating the height information map, for example, equations (10.1), (10.2), (20.1), and (20.2) are used.

The image information processing apparatus 22 has a creating means 22 c for creating an arbitrary overhead view from the height information map and the digital image data. The overhead view data is sent to the image buffer selector 23.

The image information processing device 22 also has image change detecting means 22d for detecting an image change while the vehicle is stationary. The image change detecting means 22d detects a significant image change based on the optical flow map, generates an image change detection signal, and transmits the image change detection signal to the image buffer selector 24.

The image buffer selector 23 switches between a real image while the vehicle is stationary and a virtual image obtained immediately before the vehicle is stationary based on the image change detection signal, and displays the image.
Send to (Embodiment 2) FIG. 14 is a block diagram showing Embodiment 2 of the invention. In the second embodiment of the present invention, the steering angle detecting device 25 is not provided.

In the second embodiment of the present invention, the image information processing device 22 calculates the turning radius r and the axis deviation angle θz of the vehicle.

The image information processing apparatus 22 performs optical flow detection and creates an optical flow map from the entire screen. Next, the image information processing device 22 calculates the relative radius w between the center of the imaging area and the center of the vehicle as a data group prepared in advance, the camera installation data, and the turning radius calculation means 2 based on the optical flow map.
The rotation radius and the rotation direction are calculated by 2e. This includes
For example, equations (13.1) to (13.5) and equations (21.1) and (21.2) are used.

Next, the image information processing device 22 calculates the shift angle θz by the shift angle calculating means 22f based on the data, the turning radius information and the vehicle speed. Then, the image information processing device 22 calculates height information in an actual space for each pixel from the optical flow map, the rotation radius information, the deviation angle, and each data, and generates a height information map. A desired overhead view is created from the height information map and the actual image. While the vehicle is stopped, an image change detection signal is generated based on the optical flow map.

The image buffer selector 24 switches between a real image while the vehicle is stationary and a virtual image obtained immediately before the vehicle is stationary based on the image change detection signal and transmits the switched image to the display device. (Embodiment 3) FIG. 15 is a block diagram of Embodiment 3 of the present invention. The image information processing device 22 performs optical flow detection and creates an optical flow map from the entire screen. Next, the image information processing device 22 determines the vehicle speed, the relative distance w between the center of the imaging area and the center of the vehicle body.
The height information is directly calculated based on the camera installation data and the optical flow map. For this, the following equations (Appendix 1.1) to (Appendix 1.6) are used. Subsequent processing is the same as in the first embodiment and the second embodiment of the invention, and a detailed description thereof will be omitted. Appendix 1 Equation for finding height when only speed is given ah ⁴ + bh ³ + ch ² + dh ¹ + e = 0 (Appendix 1.1) a = ((-XYΔX + f ² ΔY + X ² ΔY) cos [θ] + f (XΔX + YΔY) sin [θ]) ² (Appendix 1.2) b = -Csc [θ] ((-XYΔX + f ² ΔY + X ² ΔY) cos [θ] + f (XΔX + YΔY ) sin [θ]) (f (2HXΔX + fwΔY + 3HYΔY)-f (2HXΔX + fwΔY + HYΔY) cos [2θ] + (-2HXYΔX + f ² HΔY + 2HX ² ΔY-fwYΔY) sin [2θ]) 1.3) c = -f ² v ² Y ⁴ Xcos [θ] ⁶ + 2fv ² Y ³ (2f ² -Y ² ) cos [θ] ⁵ sin [θ] + 2f ³ cos [θ] sin [θ] ³ ( w ² ΔX (-2YΔX + XΔY) + v ² Y (-f ² + 2Y ² ) sin [θ] ² ) + Y ² cos [θ] ⁴ (w ² (YΔX-XΔY) ² -v ² (6f ⁴ -8f ² Y ² + Y ⁴ ) sin [θ] ² ) + 2fYcos [θ] ³ sin [θ] (-w ² (2Y ² ΔX ² -3XYΔXΔY + X ² ΔY ² ) + 2V ² (f ⁴ -3f ² Y ² + Y ⁴ ) sin [θ] ² ) + cos [θ] ² (6H ² X ² Y ² ΔX ² -12H ² X ³ YΔXΔY + 6fHwXY ² ΔXΔY + f ⁴ H ² ΔY ² + 6f ² H ² X ^{2 ΔY 2 + 6H 2 X 4} ΔY 2 -2f 3 HwYΔY 2 - 6fHwX 2 YΔY 2 + 6f 2 H 2 Y 2 ΔY 2 + f 2 w 2 Y 2 ΔY 2 + 2fHYΔY (-3HXYΔX + 2f 2 HΔY + 3HX 2 ΔY-fwYΔY) cot [θ] + f 2 H 2 Y 2 ΔY 2 cot [θ] 2 + 6f 2 w 2 Y 2 ΔX 2 sin [θ] 2 -6f 2 w 2 XYΔXΔYsin [ ^{] 2 + f 2 w 2 X} 2 ΔY 2 sin [θ] 2 -f 6 v 2 sin [θ] 4 + 8f 4 v 2 Y 2 sin [θ] 4 -6f 2 v 2 Y 4 sin [θ] 4 ) + f (f (6H ² X ² ΔX ² + 6fHwXΔXΔY + 12H ² XYΔXΔY + f ² w ² ΔY ² + 6fHwYΔY ² + 6H ² Y ² ΔY ² ) sin [θ] ² + f ³ w ² ΔX ² sin [θ ^{] 4 - f 3 v 2 Y} 2 sin [θ] 6 + (- 6H 2 X 2 YΔX 2 + 3f 2 H 2 XΔXΔY + 6H 2 X 3 ΔXΔY-6fHwXYΔXΔY-6H 2 XY 2 ΔXΔY + 2f 3 HwΔY 2 + 3fHwX 2 ΔY ^{2 + 3f 2 H 2 YΔY 2} - f 2 w 2 YΔY 2 + 6H 2 X 2 YΔY 2 -3fHwY 2 ΔY 2) sin [2θ]) ( with ^{1.4) d = -2 (v 2} Y 3 (-6f 3 H-5f ² wY + 3fHY ² + wY ³ ) cos [θ] ⁵ sin [θ] + v ² Y ⁴ cos [θ] ⁵ cot [θ] (-fHYcos [θ] + (4f ² H + fwY-HY ² ) sin [θ])-fYcos [θ] ³ sin [θ] (2Hw ² (2Y ² ΔX ² -3XYΔXΔY + X ² ΔY ² ) + V ² (f ⁴ H + 10f ³ wY + 2f ² HY ² -10fwY ³ -4HY ⁴ ) sin [θ] ² ) + f ² cos [θ] ⁴ (Hw ² (YΔX-XΔY) ² + v ² (4f ⁴ H + 10f ³ wY- 2f ² HY ² -5fwY ³ -HY ^{4 ) sin [θ] 2) +} cos [θ] 2 (2H 3 X 2 Y 2 ΔX 2 + 2f 2 H 3 XYΔXΔY- 4H 3 X 3 YΔXΔY + 3fH 2 wXY 2 ΔXΔY + f 2 H 3 X 2 ΔY 2 + 2H ³ X ⁴ ΔY ² + f ³ H ² wYΔY ² -3fH ² wX ² YΔY ² + 3f ² H ³ Y ² ΔY ² + f ² Hw ² Y ² Δ ^{Y 2 + fH 2 YΔY (-3HXYΔX} + f 2 HΔY + 3HX 2 ΔY-2fwYΔY) cot [θ] + f 2 H 3 Y 2 ΔY 2 cot [θ] 2 + 6f 2 Hw 2 Y 2 ΔX 2 sin [θ ^{] 2 - 6f 2 Hw 2 XYΔXΔYsin} [θ] 2 + f 2 Hw 2 X 2 ΔY 2 sin [θ] 2 + 5f 5 v 2 wYsin [θ] 4 + 3f 4 Hv 2 Y 2 sin [θ] 4 - 10f ³ v ² wY ³ sin [θ] ⁴ -6f ² Hv ² Y ⁴ sin [θ] ⁴ ) + f ² sin [θ] ² (H (2H ² X ² ΔX ² + 3fHwXΔXΔY + 4H ² XYΔXΔY + f ² w ² ΔY ² + 3fHwYΔY ² + 2H ² Y ² ΔY ² ) + f ² Hw ² ΔX ² sin [θ] ² -f ² v ² Y (fw + HY) sin [θ] ⁴ ) + fcos [θ] sin [θ ^{] (H (-4H 2 X 2} Y 2 ΔX 2 + f 2 H 2 XΔXΔY + 4H 2 X 3 ΔXΔY-6fHwXYΔXΔY-4H 2 XY 2 ΔXΔY + f 3 HwΔY 2 + 3fHwX 2 ΔY 2 + f 2 H 2 YΔY 2 - 2f ^{2 w 2 YΔY 2 + 4H 2} X 2 YΔY 2 -3fHwY 2 ΔY 2) - 2f 2 Hw 2 ΔX (2YΔX-XΔY) sin [θ] 2f 2 v 2 (f 3 w + f 2 HY-5fwY 2 -4HY ³ ) sin [θ] ⁴ )) (Appendix 1.5) e = -v ² Y ⁴ cos [θ] ⁶ (6f ² H ² + 10fHwY + 2H ² Y ² + w ² Y ² -2HY (2fH + wY ) cot [θ] + H ² Y ² cot [θ] ² ) + 2v ² Y ³ (2f ³ H ² + 10f ² HwY + f (4H ² + 3w ² ) Y ² + HwY ³ ) cos [θ] ^{5 sin [θ] + 2f 3 cos} [θ] sin [θ] 3 (H 2 w 2 ΔX (-2YΔX + XΔY) + v 2 Y (3f 2 w 2 + 5fHwY + 2H 2 Y 2) sin [θ] ^{2) + 2fY cos [θ]} 3 sin [θ] (- H 2 w 2 (2Y 2 ΔX 2 -3XYΔXΔY + X 2 ΔY 2) + v 2 Y (5f 3 Hw + 4f 2 H 2 Y + 10f 2 w 2 Y + 10fHwY ² + 2H ² Y ³ ) sin [θ] ² )-Y ² cos [θ] ⁴ (-H ² w ² (YΔX-XΔY) ² + v ² (f ⁴ H ² + 20f ³ HwY + 12f ^{2 H 2 Y 2 + 15f 2} w 2 Y 2 + 10fHwY 3 + H 2 Y 4) sin [θ] 2) + cos [θ] 2 (H 4 X 2 Y 2 ΔX 2 + 2f 2 H 4 XYΔXΔY-2H ^{4 X 3 YΔXΔY + 2fH 3 wXY} 2 ΔXΔY + H 4 X 4 ΔY 2 + 2f 3 H 3 wYΔY 2 - 2fH 3 wX 2 YΔY 2 + 2f 2 H 4 Y 2 ΔY 2 + f 2 H 2 w 2 Y 2 ΔY 2 ^{- 2fH 3 YΔY (HXYΔX-HX} 2 ΔY + fwYΔY) cot [θ] + f 2 H 4 Y 2 ΔY 2 cot [θ] 2 + 6f 2 H 2 w 2 Y 2 ΔX 2 sin [θ] 2 -6f 2 ^{H 2 w 2 XYΔXΔYsin [θ]} 2 + f 2 H 2 w 2 X 2 ΔY 2 sin [θ] 2 -2f 5 Hv 2 wYsin [θ] 4 - 2f 4 H 2 v 2 Y 2 sin [θ] 4 - ^{15f 4 v 2 w 2 Y 2} sin [θ] 4 - 20f 3 Hv 2 wY 3 sin [θ] 4 -6f 2 H 2 v 2 Y 4 sin [θ] 4) + f (fH 2 (HXΔX + fwΔY + HYΔY) ² sin [θ] ² + f ³ H ² w ² ΔX ² sin [θ] ⁴ -f ³ v ² (fw + HY) ² sin [θ] ⁶ + H ² (-H ² X ² YΔX ² + ^{H 2 X 3 ΔXΔY- 2fHwXYΔXΔY-H} 2 XY 2 ΔXΔY + fHwX 2 ΔY 2 - f 2 w 2 YΔY 2 + H 2 X 2 YΔY 2 -fHwY 2 ΔY 2) sin [2θ ]) (Appendix 1.6)

[0109]

According to the present invention, as described above, even when a monocular camera is used, three-dimensional information of an image can be generated by using vehicle information, and a natural bird's-eye view without a sense of discomfort can be formed. This has the effect that a compact in-vehicle surrounding monitoring device can be manufactured.

Furthermore, since only one camera is required, complicated operations such as installation and adjustment can be reduced.

[Brief description of the drawings]

FIG. 1 is a schematic diagram for explaining the principle of image conversion.

FIG. 2 is a diagram showing an example of a real image captured by the real camera shown in FIG.

FIG. 3 is a diagram showing an example in which the real image shown in FIG. 2 is converted into a virtual image without considering height information.

FIG. 4 is a schematic diagram for explaining the principle of acquiring three-dimensional information based on an actual image.

FIG. 5 is a diagram showing a real image captured by the real camera shown in FIG. 4 at a certain time;

6 is a diagram showing a real image captured by the real camera shown in FIG. 4 at a time different from a certain time.

FIG. 7 is a schematic diagram for explaining the relationship between the turning of the vehicle, the imaging area, and the optical axis of the real camera.

8 is a partially enlarged view for explaining the relationship between the vehicle shown in FIG. 7 and an imaging area.

FIG. 9 is an enlarged view of the imaging region shown in FIG.

FIG. 10 is a schematic diagram for explaining the principle of calculating height information when vehicle speed information is given.

FIG. 11 is a schematic diagram for explaining the principle of calculating height information when the rotation direction of the vehicle is clockwise.

FIG. 12 is a block circuit diagram of the on-vehicle surroundings monitoring device according to the first embodiment of the present invention.

13 is a detailed block diagram of the on-vehicle periphery monitoring device shown in FIG.

FIG. 14 is a detailed block diagram of an in-vehicle surroundings monitoring apparatus according to Embodiment 2 of the present invention.

FIG. 15 is a detailed block diagram of an in-vehicle surroundings monitoring apparatus according to Embodiment 3 of the present invention.

[Explanation of symbols]

 2 Real camera 3 Virtual camera 8 Vehicle 20 Digital video camera 21 Vehicle speed detection device 22 Image information processing device

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) G06T 7/20 G06T 7/20 B 5L096 15/00 100 15/00 100A G08G 1/16 G08G 1/16 D H04N 13/00 H04N 13/00 F-term (reference) 3D020 BA20 BC13 BD10 BE03 5B080 BA05 GA21 5C054 AA01 AA05 CA04 CC05 EA01 EA05 FA01 FA02 FC01 FD03 HA30 5C061 AA21 AB03 AB08 AB17 5H180 AA01 CC04 LL01 LL09 A0309

Claims (10)

[Claims] 1. A real camera, which is photographed by a real camera arranged so as to look at an object to be photographed from a certain direction, by a virtual camera virtually arranged so as to look at the object to be photographed from another direction. An image conversion method for converting a captured virtual image into a captured virtual image, wherein the object to be captured is moved relative to the real camera based on installation data of the real camera on a vehicle and optical characteristic data of the real camera. An image conversion method characterized in that the obtained moving image information is analyzed to obtain three-dimensional information of the image. 2. A real image taken by a real camera arranged to view an object to be photographed from one direction by a virtual camera virtually arranged to observe the object to be photographed from another direction. An in-vehicle periphery monitoring device that converts an image into a captured virtual image to create and display an overhead view, wherein the real camera is a digital video camera that is mounted on a vehicle and captures an image of the periphery of the vehicle. A vehicle speed detection device that detects a vehicle speed of the vehicle, an image information processing device that converts a real image taken by the real camera into a virtual image taken by the virtual camera to create a bird's-eye view, and displays the bird's-eye view A display device to perform, the image information processing device is based on the vehicle speed information obtained by the vehicle speed detection device based on the height information contained in the real image captured by the real camera Vehicle surroundings monitoring apparatus characterized by creating the bird's-eye view and calculated. 3. The vehicle according to claim 1, further comprising a steering angle detecting device for detecting a steering angle of a steering wheel, and at the time of turning the vehicle based on steering angle information obtained by the steering angle detecting device and vehicle speed information obtained by the vehicle speed detecting device. The in-vehicle periphery monitoring device according to claim 2, wherein the overhead view is created by calculating height information included in a real image captured by the real camera. 4. An image buffer for temporarily storing an image processed by the image information processing device when the vehicle stops and displaying the image on the display device.
The in-vehicle periphery monitoring device according to claim 2 or 3, further comprising a selector device. 5. The image buffer / selector device according to claim 1,
The vehicle-mounted vehicle according to claim 4, wherein an image change based on an object approaching into an imaging area of the real camera is detected while the vehicle is stationary, and an image captured by the real camera is displayed on the display device. Ambient monitoring device. 6. The image information processing apparatus according to claim 1, further comprising: distance data from the center of the vehicle to the center of the imaging area; installation data of the real camera with respect to the vehicle; pixel size data of the real camera; and focal length data. Is prepared in advance, the image information processing apparatus performs an optical flow detection based on a moving image captured by the digital video camera to generate an optical flow map of the entire screen, and the optical flow map And height information map generating means for calculating height information in a real space for each pixel based on the data and the vehicle speed to generate a height information map, and the height information map and the moving image 3. A creating means for creating the overhead view from digital data. In-vehicle surrounding monitoring device. 7. The image information processing apparatus prepares in advance installation data of the real camera with respect to the vehicle, pixel size data and focal length data of the real camera, and the image information processing apparatus Optical flow map generating means for performing optical flow detection based on a moving image captured by a camera to generate an optical flow map of the entire screen, and a turning radius and a turning direction from the optical map, the vehicle speed, and the data, and The turning radius calculation means to be determined, and a deviation angle between a tangent perpendicular to the turning radius of the vehicle and the optical axis of the real camera based on the turning radius information obtained by the turning radius calculation means and the data. Deviation angle calculating means for calculating, the optical flow map, the respective data, and the turning radius calculating means. Height information map generating means for calculating height information in a real space for each pixel based on a shift angle obtained by the step and the shift angle calculating means and the vehicle speed, and generating a height information map; The in-vehicle periphery monitoring device according to claim 2, further comprising: a creating unit configured to create the overhead view from a height information map and digital data of the moving image. 8. The vehicle according to claim 1, further comprising a steering angle detecting device for detecting a steering angle of a steering wheel, wherein the image information processing device includes: setting data of the real camera with respect to the vehicle; pixel size data of the real camera; Distance data and deviation angle relation data between a deviation angle and a steering angle between a tangent perpendicular to the turning radius of the vehicle and the optical axis of the real camera are prepared in advance, and the image information processing apparatus is Optical flow map generating means for performing optical flow detection based on a moving image captured by a moving image camera to generate an optical flow map of the entire screen, and the deviation angle data obtained based on the steering angle of the steering angle detection device The height information in the real space is calculated for each pixel based on the vehicle speed, the data, the data, and the optical flow map to generate a height information map. Means for generating height information map,
The in-vehicle periphery monitoring device according to claim 2, further comprising: a creating unit configured to create the overhead view from the height information map and the digital data of the moving image. 9. An image buffer for temporarily storing an image processed by the image information processing device when the vehicle is stopped and displaying the image on the display device.
A selector device, the image information processing device compares the optical flow map while the vehicle is stationary with the optical flow map while the vehicle is running based on the optical flow map, detects a significant image change, and detects the image buffer Image change detection means for transmitting an image change detection signal to the selector device is provided, the image buffer selector device, based on the image change detection signal, the vehicle stationary image and the virtual image obtained immediately before the still image 9. The in-vehicle surroundings monitoring device according to claim 6, wherein the information is switched and transmitted to the display device. 10. The installation data is data relating to a mounting height of the real camera to the vehicle, a depression angle with respect to the ground, and a distance from the ground to the real camera on the optical axis center line of the real camera. The in-vehicle surroundings monitoring device according to any one of claims 6 to 9, wherein: