JP2011500407A - Method for calibrating a device comprising at least one omnidirectional camera and an optical display unit - Google Patents

Abstract

The invention relates to a method for calibrating an object, in particular a device comprising at least one omnidirectional camera (2) and an optical display unit arranged in a vehicle (1), in which the image displayed by the display unit is The visual field of the virtual camera assumed above (1) is reproduced. At that time, the image of the virtual camera is projected into one object coordinate system, and the point (Xw) generated in the object coordinate system at that time , Xw · y) is projected into the omnidirectional camera, and the pixel (x _p ) of the virtual camera image is assumed around the object (1) when the virtual camera image is projected onto the object coordinate system. The virtual is projected so that the pixels are projected onto the assumed plane, projected on the virtual plane within the circle or ellipse, while the pixels (x _p ) of the image outside the circle or ellipse rise from the edge of the circle or ellipse. turtle Is transformed into the object coordinate system.
[Selection] Figure 3

Description

  The present invention relates to a method for calibrating an object, in particular a device comprising at least one omnidirectional camera and an optical display unit arranged in a vehicle, in which the image displayed by the display unit is assumed above the object. In this case, the image of the virtual camera is projected into one object coordinate system, and the image of the omnidirectional camera is projected onto the object coordinate system. The invention further relates to a method of image processing and image display using a device comprising at least one omnidirectional camera and an optical display unit arranged on an object, in particular a vehicle.

  Omnidirectional cameras are basically known. An omnidirectional camera can be used to capture an omnidirectional image (also called a bird's eye view or a bird's eye view), and the image information of the omnidirectional image includes, for example, data such as an image position related to the position of the captured object or object, It contains the distance data between the object and the omnidirectional camera.

  An omnidirectional camera is used, for example, to monitor the surroundings of the vehicle in front, side and / or rear of the vehicle. At that time, for example, an omnidirectional camera is used to identify objects or objects around the vehicle. In order to be able to determine, in particular, the position and / or distance of the identified object sufficiently accurately, the exact position and the correct orientation of the omnidirectional camera relative to the vehicle is known. In particular, the omnidirectional camera displays an image captured by the camera on an optical display unit and shows it to the driver, so that a vehicle with poor visibility, such as a truck, can be used for navigation when traveling backward, for example. In this case, a significantly distorted image of the omnidirectional camera is converted to provide a bird's eye view to the driver. The problem here is that the nearby area around the vehicle is very well captured, but objects farther away are not displayed.

  Patent Document 1 discloses an image processing technique. In this case, a large number of camera images around the vehicle are taken. These images are transformed so that they are interpreted as being projected on the inner surface of a certain hemisphere or kite, so that image points farther away are also displayed. At that time, the strain increases outward from the central region of the hemisphere. This method requires high calculation cost.

US Pat. No. 7,161,616B1

  The object of the present invention is to provide a more improved calibration method by means of an apparatus having at least one omnidirectional camera and an optical display unit, and at least one omnidirectional camera and optical arrangement arranged on an object, in particular a vehicle. A method for performing image processing and image display using an apparatus having a display unit is presented.

  With regard to a method for calibrating the device, the object of the present invention is solved by the features of claim 1. With regard to image processing and image display methods, the object of the present invention is solved by the features of claim 5.

  Advantageous developments of the invention are disclosed in the dependent claims.

  In accordance with the present invention, an apparatus having at least one omnidirectional camera and an optical display unit arranged on an object, in particular a vehicle, is constructed by the method described below. A virtual camera is defined above the object, and an image of this camera is displayed by the display unit. The image of the virtual camera is projected on the object coordinate system. Coordinate points generated in the object coordinate system at this time are projected into the omnidirectional camera or onto the coordinate system.

  When the virtual camera image is projected onto the object coordinate system, the pixels of the virtual camera image are projected onto the virtual plane within one circle or ellipse assumed around the object. Pixels in the image outside the circle or ellipse are transformed into the object coordinate system by the virtual camera so that the pixels are projected onto the virtual surface rising from the edge of the circle or ellipse, with the plane in the object coordinate system The height of the upper pixel is proportional to the pixel distance from the center point of the circle or ellipse. The system calibrated in this way displays the perimeter that is closer to the object in a circle or ellipse over a wide range, while also displaying the farther away perimeter that is not visible with known methods. The The enlarged bird's-eye view obtained in this way is intuitive and generates a wide field of view. For the driver of the vehicle, parking steering especially while traveling backwards becomes easier.

  Preferably, the surface has the shape of the side surface of the truncated cone. In doing so, the side surface of the truncated cone has a linear gradient, and therefore the projection can be calculated on this plane in a particularly simple manner. Furthermore, the side surface of the truncated cone provides an advantage that the distortion generated in the image is remarkably reduced in the case of an ellipse, for example, as a projection surface compared to a surface having a non-linear gradient. The use of the side surface of the truncated cone provides an image display that is particularly intuitive and easy to understand for the driver. In this case, the distortion only appears at the edge region of the side surface of the truncated cone, and on the contrary, the display of the center of the image takes place in a particularly advantageous manner without distortion.

  The omnidirectional camera is preferably calibrated with respect to internal parameters using a calibration body that contains the entire field of view of the camera. The calibration body has a barrel shape in particular and is provided with an annular mark inside. In order to calibrate an omnidirectional camera, the indicia are measured with sub-pixel accuracy, in particular by the method described in Non-Patent Document 1 (T. Luhmann. . For example, with respect to the object coordinate system, camera external parameters such as camera translation and rotation are specified by an omnidirectional camera attached to the object. In this case, a rectangular mark can be used. For example, in this case, Non-Patent Document 2 (LE Krueger, C. Woehler, A. Wurz-Wessel, F. Stein. In-factory calibration of multi-systems on SPIR. : Pages 126 to 137 (September 2004) 4, 5) are used.

  The internal parameters of the camera are, for example, a focal length, a pixel size (width and height), an image center point, and the like regarding the coordinate system and distortion.

  The device so calibrated is used in a method of image processing and image display, in which the object periphery is imaged by an omnidirectional camera and reproduced on a display unit in accordance with the calibration.

  In doing so, preferably two omnidirectional cameras are used, which can be arranged, for example, in the area of the roof edge on the back of the vehicle. However, more omnidirectional cameras or just one omnidirectional camera may be provided.

  In particular, a range captured by a plurality of omnidirectional cameras is divided into a plurality of cameras non-symmetrically with respect to the vehicle. This division is described, for example, in German Patent Application Publication No. 102006003538B3. This avoids the disappearance of another object around the vehicle from view as a result of assuming a flat perimeter that is brought from the rear projection to the virtual plane. At that time, each point in the object coordinate system is projected onto an omnidirectional camera arranged behind and to the right of the vehicle. Each point on the left of the vehicle is projected onto an omnidirectional camera arranged on the left.

  Preferably, in the case of a vehicle traveling backward and / or in a reverse gear, the traveling operation is predicted depending on the steering angle, and the image is superimposed on the display unit. In this way, the driver can determine whether the desired position can be reached when the vehicle is driven backward at the current steering angle.

  In order to simplify the prediction, the calculation is based on a single track model, in which one virtual wheel is modeled in the middle of the axle for every two wheels of an axle.

  In the following, embodiments of the present invention will be described in detail with reference to the drawings.

It is the back of the vehicle provided with two omnidirectional cameras. It is a schematic diagram of a standard bird's-eye view of a vehicle and its surroundings known in the prior art. 1 is a schematic bird's eye view of a vehicle and its surroundings enlarged for image processing and image display using a method according to the present invention. These are a bird's-eye view of an enlarged view of the vehicle and its surroundings, displayed on an optical display unit in the vehicle, and a predicted traveling operation at the time of reverse traveling superimposed thereon. It is a single trajectory model for running motion prediction.

In the drawings, members corresponding to each other are denoted by the same reference numerals.

  FIG. 1 is a rear view of an object 1 formed as a vehicle with two omnidirectional cameras 2. Reference numeral 1 is also used for a vehicle in the following.

  The omnidirectional camera 2 photographs the periphery of the vehicle 1 at a very wide angle from the side and the rear of the vehicle 1. The processed camera image is displayed on a display unit (not shown) in the vehicle 1. The camera 2 is calibrated with respect to the vehicle 1 for this purpose. Since the camera 2 is fixed, the calibration needs to be performed only once.

The purpose of calibration is to first display the vehicle 1 and its surroundings in a bird's eye view. For this purpose, one virtual camera is defined on the vehicle 1. The virtual camera pixel x _p = (x _p .x, x _p .y) ^T is projected onto the virtual plane Y = 0 (assuming a flat world), and the object coordinate system adjusted by the vehicle 1 at that time The point _Xw is generated at. :
X _w = λX _r + _{C p}
that time,
λ = C _p · y / X _r · y
_{X r = R p X p =} R p (x p .x, x p .y, f) T
In this case, f is the focal length of the virtual camera, R _p and C _p are the rotation and translation of the virtual camera with respect to the object coordinate system, and the coordinate origin of this coordinate system may be at the center of the bumper on the back of the vehicle 1, for example. The y-axis points up vertically.

The rear-projected point _Xw is projected into at least one omnidirectional camera 2, that is, into an image photographed thereby. This projection is described in detail in Non-Patent Document 3 (C. Topepfer, T. Ehlgen. A unified omnidirectional camera model and ist applications. In Omnivs (February 2007)).

As a result of assuming the periphery to be flat from the rear projection on the virtual plane, in order to avoid the disappearance of other objects around the vehicle 1 from view, A metric split is selected. At that time, in each point X _w of the object coordinate system, the point of the right and rear of the vehicle 1 is projected to the omnidirectional camera 2 disposed on the right side. Each point _Xw on the left side of the vehicle 1 is projected onto the omnidirectional camera 2 arranged on the left side. This detail is described, for example, in German Patent Application Publication No. 102006003538B3.

  FIG. 2 shows a schematic diagram of the vehicle 1 and its surroundings in a standard bird's-eye view achieved by such a calibrated device having two omnidirectional cameras 2 and an optical display unit. Calibration is used to achieve a corrected display of the indicated distortion, as suggested by the chessboard pattern. Non-symmetric division of the field of view in the two omnidirectional cameras 2 is indicated by the chessboard pattern being displayed differently. The driver of the vehicle 1 can check whether there is an obstacle around the vehicle 1. Within this range, prior art methods are known.

In accordance with the invention, the device comprising two omnidirectional cameras 2 and an optical display unit is calibrated so that rear projection onto the virtual plane Y = 0 is performed only in the inner region 3 around the vehicle 1. This inner region 3 is in a virtual circle with a radius r around the coordinate origin of the vehicle 1 or object coordinate system. Each pixel x _p in the virtual camera image is back projected onto the virtual plane, thereby overlapping, resulting in a point X _w in the object coordinate system. Interpolation of the projected points X _w results in intensity value of a pixel x _p.

ここでｒは円の半径であり、ｍは立ち上がる面である。こうしてもたらされた拡大された鳥瞰図が図３に示されている。このようにして、運転者の視野領域は、内側領域３内では従来同様ひずむことはない。内側領域３の外では、車両１のさらに別の周辺が表示される。 Point X _w outside the inner region 3, not on a virtual plane, is projected on a surface that rises from the edge of the circle. In other words, the height X _{w · Y} of the distance between the point X _w is dependent on the center point of the circle:

Here, r is the radius of the circle, and m is the rising surface. An enlarged bird's-eye view resulting from this is shown in FIG. In this way, the driver's field of view is not distorted in the inner region 3 as in the prior art. Outside the inner region 3, yet another periphery of the vehicle 1 is displayed.

FIG. 4 shows a bird's-eye view in which the scene of the vehicle 1 and its surroundings shown in the optical display unit in the vehicle 1 is enlarged, and a predictive traveling operation at the time of reverse traveling superimposed thereon. Driving operation, the movement of the rear bumper first passage 4, the movement of the front bumper is displayed on the second passage 5, the vehicle 1 in its passage, as long as the current steering angle [delta] _R is not changed, the steering angle is moved correspondingly [delta] _R. When the steering angle [delta] _R is changed, the prediction is to be adapted accordingly thereto. The passage line 4.1 represents the rear bumper. 4.2 represents a distance 1 m behind the bumper, which may correspond to, for example, a turning range of the rear gate of the vehicle 1. Line 4.3 represents the distance of the length of the vehicle 1 approximately. The radius r of the circle surrounding the inner region 3 should correspond at least to the distance from the line 4.3 to the rear bumper.

  In order to simplify the prediction, the calculation is based on a so-called single track model 6 in which one virtual wheel 7 is modeled in the center of the axle for every two wheels of one axle of the vehicle 1. . The front and rear virtual wheels 7 are connected by a fixed line. This model is sufficient under the restriction that the vehicle 1 does not move around its longitudinal axis and the load on any wheel is kept the same. This assumption is valid at low speeds, such as during parking steering while traveling backwards. In this context, based on this model assumption, one or more passages can be displayed in the image, which allows the driver to recognize the situation in particular intuitively. In particular, parking steering of the vehicle is simplified.

The following fixed parameters are considered in the single trajectory model 6:
l _w vehicle width l _FR distance between front and rear axles l distance between _RQ rear axle and rear bumper Further variable parameters are used:
δ _R rudder angle

Yet another parameter is dependent on the steering angle [delta] _R:
r _F virtual front wheel 7 of the movement radius r _R virtual after the wheels 7 of the movement radius r _Q movement of the moving radius r _P rear bumper left outside point of the rear bumper center point radius r _S rear bumper of the right external point movement radius Ψ _Q steering angle δ from the instantaneous center of rotation 8, which is determined by _R, the angle a line connecting the rear wheel 7 and rear bumper center point forms.

リヤバンパー左右外点の各運動半径は、ｒ_Ｑに相似に演算することができる。

The model is constructed using the following equation:
r _F = l _FR / sin δ _R
r _R = r _F · cos δ _R
tan Ψ _Q = l _RQ / r _R
r _Q = r _R / cos Ψ _Q
General relationship: 1 / cos ² α = 1 + tan ² α leads to:

Each movement radius of the rear bumper left and right outer point can be computed in similar to r _Q.

  The omnidirectional camera 2 is particularly formed as a mirror lens camera.

  The described method can likewise be carried out using only one omnidirectional camera 2 or using two or more omnidirectional cameras 2.

  The described method can in principle be carried out on any object 1.

  The camera 2 can be arranged in the area of the roof edge of the vehicle 1.

  The inner region 3 can be described using one ellipse instead of one circle.

  The virtual plane can correspond to the ground on which the vehicle 1 is placed.

DESCRIPTION OF SYMBOLS 1 Object, vehicle 2 Omnidirectional camera 3 Inner area | region 4.1 Line 4.2 Line 4.3 Line 5 Second path 6 Single track model 7 Virtual wheel 8 Instantaneous rotation center C _p Translation of virtual camera δ _R Steering angle f Focal length of virtual camera 1 _W vehicle width 1 _FR distance between front and rear axles 1 _RQ distance between rear axle and rear bumper Ψ _Q angle formed by virtual rear wheel and rear bumper center point r radius r _F motion of virtual front wheel movement radius of radius r _P rear bumper of the left outside the point movement radius r _Q rear bumper movement of the center point radius r _R virtual rear wheel 6
r _s rotation x _p points X _{w · y} height of points in the object coordinate system in the virtual camera pixel X _w object coordinate system of the rear bumper right outside point of movement radius R _p virtual camera

Claims (9)

In a method for calibrating a device comprising at least one omnidirectional camera (2) and an optical display unit arranged on an object (1), in particular a vehicle (1), the vehicle displays an image displayed by the display unit. The virtual camera field of view assumed above is reproduced, and an image from the virtual camera is projected onto one object coordinate system, and points (X _w , X _{w · y} ) generated in the object coordinate system are A method of projecting into an omnidirectional camera,
When the virtual camera image is projected onto the object coordinate system, the image cell (x _p ) of the virtual camera within the circle or ellipse assumed around the object (1) is projected onto the virtual plane. The pixel (x _p ) of the image outside the circle or ellipse is transformed into the object coordinate system by the virtual camera so that it is projected onto a virtual plane rising from the edge of the circle or ellipse, and the object coordinates The height of the point (X _w , X _{w · y} ) on the virtual plane in the system is proportional to the distance from the point (X _w ) to the center point of the circle or ellipse. Method.   The method of claim 1, wherein the virtual surface has a shape of a side surface of a truncated cone.   3. The omnidirectional camera (2) is calibrated with respect to internal parameters using a calibration body comprising the (2) entire field of view of the omnidirectional camera. The method described.   The method according to claim 3, wherein the calibration body has a barrel shape with an annular mark inside.   In a method of image processing and image display using an apparatus comprising at least one omnidirectional camera (2) and an optical display unit arranged on an object (1), in particular a vehicle (1), the apparatus comprises: Calibrated by the method described in (4) to (4), and the periphery of the object (1) is photographed by the omnidirectional camera (2) and reproduced on the display unit in accordance with the calibration. Method.   Method according to claim 5, characterized in that images of two omnidirectional cameras (2) arranged on the back of the vehicle (1) are recorded.   The method according to claim 6, characterized in that the range captured by the omnidirectional camera (2) is divided into the omnidirectional camera (2) non-symmetrically with respect to the vehicle (1). In the case of the vehicle (1) traveling backward or in the reverse gear, the traveling operation is predicted depending on the steering angle (δ _R ), and an image of the predicted traveling operation is displayed on the display unit. 8. A method according to any one of claims 5 to 7, characterized in that it is superimposed. The prediction is based on a single track model (6), where one virtual wheel (7) is modeled in the center of the axle for every two wheels of one axle of the vehicle (1). 9. A method according to claim 8, characterized in that

Images