DE102019202269B4 - Method for calibrating a mobile camera unit of a camera system for a motor vehicle - Google Patents

Abstract

A method for calibrating a mobile camera unit (20) of a camera system (22) for a motor vehicle (10), the camera unit (20) being designed to capture an optical flow (42), comprising the steps: - Attaching the mobile camera unit (20) at any position (p) on a surface (18) of the motor vehicle (10) such that at least two mutually perpendicular edges (38) of the surface (18) lie in a field of view (28) of the camera unit (20); Detection of static position parameters (pus, pos, pls, prs) of the at least two edges (38); - Dynamic detection of dynamic position parameters (pud, pod, pld, prd) of the at least two edges (38) with the aid of the detected optical flow ( 42); - Fusion of the static position parameters (pus, pos, pls, prs) with the dynamic position parameters (pud, pod, pld, prd) to form actual position parameters (put, pot, plt, prt); - Calculation of local coordinates (x , Y Z ) the camera unit (20) based on the determined actual position parameters (put, pot, plt, prt) and intrinsic parameters of the camera unit (20); - transmission of the local coordinates (x, y, z) to a central processing unit (54) of the Camera system (22).

Description

The invention relates to a method for calibrating a mobile camera unit of a camera system for a motor vehicle, in particular a commercial vehicle.

In the field of motor vehicles, single or multi-camera systems are finding an increasingly broad range of applications. In addition to pure comfort systems to improve the driver's view, these offer alternative or additional driver assistance functions (ADAS systems) or are used as camera monitor systems as mirror replacement systems. The camera units of such systems can be operated individually or in combination.

The majority of the camera units are fixedly installed on the respective motor vehicle. The tolerances of the mounting positions (extrinsic parameters) are usually adjusted during production. Particularly in camera systems in which viewing areas of different camera units are combined with one another (for example by stitching in surround view systems), it is known to align the camera units with high precision, for example with large calibration patterns on the floor on which the motor vehicle is located. This process is carried out in production, for example.

A change in the positions of the camera units, for example through mechanical, thermal or other influences, usually requires recalibration in the workshop.

The known camera systems therefore require fixed, explicitly known and calibrated positions of the camera units.

However, it would be desirable to be able to attach a mobile camera unit temporarily to a motor vehicle and use it in a camera system of the motor vehicle.

US 2016/0 284 087 A1 discloses a method with which an orientation of a camera on the vehicle relative to the vehicle can be estimated on the basis of a movement of a vehicle.

In DE 10 2010 048 143 A1 describes a method for calibrating a camera arranged on a vehicle, current camera parameters being continuously determined and a current pitch angle of the vehicle being taken into account in the calibration.

DE 10 2005 050 363 A1 discloses a camera system for a vehicle with a camera detachably connected to the vehicle, wherein the camera can be coupled to a plurality of coupling stations fixedly connected to the vehicle.

US 2014/0 085 469 A1 discloses a calibration method of cameras using images generated by imaging a peripheral area of a vehicle by the cameras, the peripheral area of a vehicle including a calibration index preset on a road surface on which the vehicle is positioned.

The object of the invention is therefore to propose a method for calibrating a mobile camera unit.

This object is achieved with a method with the combination of features of claim 1.

Advantageous refinements of the invention are the subject of the dependent claims.

• - Befestigen der mobilen Kameraeinheit an einer beliebigen Position an einer Fläche des Kraftfahrzeuges derart, dass wenigsten zwei senkrecht zueinander angeordnete Kanten der Fläche in einem Sichtbereich der Kameraeinheit liegen;
• - Statisches Detektieren von statischen Positionsparametern der wenigstens zwei Kanten;
• - Dynamisches Detektieren von dynamischen Positionsparameter der wenigstens zwei Kanten mithilfe des erfassten optischen Flusses;
• - Fusionieren der statischen Positionsparameter mit den dynamischen Positionsparametern zu tatsächlichen Positionsparametern;
• - Berechnung von lokalen Koordinaten der Kameraeinheit auf Basis der ermittelten tatsächlichen Positionsparameter und intrinsischer Parameter der Kameraeinheit;
• - Übermitteln der lokalen Koordinaten an eine zentrale Recheneinheit des Kamerasystems.

In a method for calibrating a mobile camera unit of a camera system for a motor vehicle, in which the camera unit is designed to detect an optical flow, the following steps are carried out:

• Fastening the mobile camera unit at any position on a surface of the motor vehicle in such a way that at least two mutually perpendicular edges of the surface are in a field of view of the camera unit;
• Static detection of static position parameters of the at least two edges;
• - Dynamic detection of dynamic position parameters of the at least two edges with the aid of the detected optical flow;
• Fusing the static position parameters with the dynamic position parameters to form actual position parameters;
• - Calculation of local coordinates of the camera unit on the basis of the determined actual position parameters and intrinsic parameters of the camera unit;
• - Transmission of the local coordinates to a central processing unit of the camera system.

With this method, it is possible to combine a mobile camera unit with a camera system of a motor vehicle and to attach the mobile camera unit temporarily to the motor vehicle (for example by magnet), the camera unit in the without explicit calibration Manufacturing or in a workshop. The camera unit determines its own position parameters on the motor vehicle and can transmit its local coordinates to a central processing unit of the motor vehicle, for example a mirror replacement system. The central processing unit can then transform the local coordinates of the camera unit into system coordinates and thus integrate the data from the camera unit with appropriate stitching.

The motor vehicle can be, for example, a utility vehicle such as a truck, a bus, a construction machine, an agricultural machine or a car.

The mobile camera unit can, for example, be a camera with a fisheye lens, the optical lens of which has an opening angle> 180 °. As a result, the camera unit can “look” to the side to such an extent that it can “see” at least two edges of the rear wall surface of the trailer, for example when mounted on a rear wall of a truck trailer. It is also possible for the camera unit to be arranged such that it can “see” all edges of the rear wall, in particular four edges.

Classic image processing methods can be used for static detection of the position parameters of the at least two edges. For example, a horizontal edge of a body / suitcase of the trailer or horizontal edges of a bumper of the motor vehicle can be used for this, which do not move relative to the camera unit and thus allow static detection of position parameters.

If the motor vehicle is in motion, the camera can capture the optical flow image through image capture and thus detect position parameters of the at least two edges that are moving dynamically.

The position parameters of the at least two edges thus detected statically on the one hand and dynamically on the other hand from the optical flow are then superimposed or merged. From the fusion of these two types of position parameters, the actual position parameters of the two relevant edges can be determined with a significantly higher degree of certainty and precision. Since intrinsic parameters of the camera unit are known, the distances between the camera unit and the two edges can be determined using the position parameters of the two edges and the local coordinates of the camera unit on the surface of the motor vehicle can thus be calculated.

The camera unit thus independently determines a local position on the surface of the motor vehicle to which it is attached. The position is a combination of several correlated parameters that the camera unit can determine independently. This set of parameters is then transmitted to an overall system, in particular a central processing unit of a camera system, into which the field of view or the camera functionality itself is to be integrated. The camera system can then perform a precise integration of the camera unit into the entire camera system from the combination of the transmitted parameters and its own known or determined parameters.

Before dynamic detection, a spatial position “below” relative to the camera unit is advantageously detected by detecting a surface on which the motor vehicle is arranged, with the aid of an acceleration sensor being used for this purpose. The acceleration sensor is part of the camera system and can be arranged, for example, on the mobile camera unit. The acceleration sensor can be, for example, a 3-axis or 6-axis acceleration sensor, via which the camera unit can determine where “down” is. This is done on the basis of the acceleration due to gravity, by means of which the acceleration sensor can determine the orientation of an image sensor of the camera unit around a Z-axis that is perpendicular to the ground.

The optical flow for the subsurface is preferably recorded first. After the camera unit knows where “down” is by detecting the spatial position, it can determine the optical flow for the subsurface, in particular the floor on which the motor vehicle is moving.

The detected optical flow is advantageously specified more precisely with the aid of acceleration data from the moving motor vehicle. The camera unit can optionally detect whether the motor vehicle is being accelerated or decelerated in order to be able to determine the optical flow more precisely. Optionally, however, the camera unit can also receive data from the motor vehicle if, for example, a receiver is coupled to a BUS system of the motor vehicle.

The optical flow is advantageously recorded from at least one further spatial position “right”, “left” and / or “above” relative to the camera unit.

An image analysis device is advantageously provided which is designed to recognize the at least two edges, in particular more than two edges, of the motor vehicle on the basis of a jump in the detected optical flow. If the optical flow is recorded in several spatial positions as described above, it can thus be recognized in which spatial position a corresponding edge can be found. Thus, for example, trailer edges of a body of a motor vehicle on a truck can be recognized by evaluating the optical flow, since a jump in the optical flow forms at the trailer edges. In the area behind the edges, which are covered by the trailer and can no longer be seen through the optics of the camera unit, the optical flow is in principle zero.

A trajectory of the motor vehicle in motion is advantageously determined from the optical flow of several spatial positions and the plausibility of the acquired data relating to the optical flows of the several spatial positions is checked using an actual trajectory of the motor vehicle in motion. The movement or trajectory of the vehicle can be determined from the correlation of the recorded data with respect to the optical flows. Since the actual trajectory of the motor vehicle is stored in the central processing unit, it is possible to carry out a plausibility check and filtering of all data obtained and thus to obtain a more robust information base, on the basis of which the relevant edges of the motor vehicle can be recognized.

In an exemplary method not belonging to the invention for operating a camera-monitor system, especially designed for a motor vehicle and having a mobile camera unit, or a multi-camera system or an ADAS system, a method for calibrating a mobile camera unit in accordance with the claims 1 to 7 specified procedure.

Furthermore, according to the invention, a camera monitor system with a mobile camera unit and a central processing unit is designed to carry out a method as described above.

• 1 eine Draufsicht von oben auf ein Kraftfahrzeug mit einem Systemkoordinatensystem, das eine mobile Kameraeinheit aufweist, welche lokale Koordinaten aufweist;
• 2 eine vergrößerte Draufsicht von oben auf das Kraftfahrzeug aus 1 mit mobiler Kameraeinheit, die eine Fischaugen-Linse aufweist und somit über einen Öffnungswinkel von > 180° verfügt;
• 3 eine Ansicht von hinten auf das Kraftfahrzeug aus 1 mit mobiler Kameraeinheit;
• 4 eine perspektivische Ansicht auf das Kraftfahrzeug aus 1 zum Veranschaulichen eines optischen Flusses;
• 5 ein schematisches Flussdiagramm eines Verfahrens zum Kalibrieren der mobilen Kameraeinheit aus 1 bis 3;
• 6 eine schematische Darstellung einer Recheneinheit, in der das Verfahren gemäß 5 durchgeführt werden kann.

An advantageous embodiment of the invention is explained in more detail below with reference to the accompanying drawings. It shows:

• 1 a plan view from above of a motor vehicle with a system coordinate system having a mobile camera unit which has local coordinates;
• 2 an enlarged top plan view of the motor vehicle 1 with a mobile camera unit that has a fisheye lens and thus an opening angle of> 180 °;
• 3 a rear view of the motor vehicle 1 with mobile camera unit;
• 4th a perspective view of the motor vehicle 1 to illustrate an optical flow;
• 5 a schematic flow diagram of a method for calibrating the mobile camera unit 1 to 3 ;
• 6th a schematic representation of a computing unit in which the method according to 5 can be carried out.

1 shows a plan view of a motor vehicle 10 this one as a truck 12 with tractor 14th and trailers 16 is shown. On the trailer 16 , especially on a rear surface 18th , is a mobile camera unit 20th attached, which is part of a camera system 22nd is also on the tractor 14th fixed mirror replacement cameras 24 belong.

In 1 are the field of vision 26th one of the mirror replacement cameras 24 as well as the field of vision 28 the mobile camera unit 20th shown in cross section.

The overall system 30th from truck 12 and associated camera system 22nd defines a system coordinate system 32 with the system coordinates xs, ys, zs. In this system coordinate system 32 is the mobile camera unit 20th attachable at different positions p, the positions p being defined by local coordinates x, y, z of the camera unit 20th .

As the camera unit 20th at different positions p on one of the surfaces 18th of the motor vehicle 10 can be attached, it is important that this position p, defined by the local coordinates x, y, z, is known and to the camera system 22nd can be transmitted, and so the mobile camera unit 20th into the camera system 22nd integrate by taking the local coordinates x, y, z of the mobile camera unit 20th can be transformed into system coordinates xs, ys, zs and thus an integration can be realized, for example with appropriate stitching.

This is made possible by the mobile camera unit 20th is enabled to independently determine their local position p on the motor vehicle 10 to determine.

2 shows a plan view of the motor vehicle 10 out 1 in an enlarged view in the area of the mobile camera unit 20th that are on the back face 18th is attached. As with the aid of a schematically illustrated beam path 36 can be seen, the camera is formed with a fisheye lens. The fisheye lens 34 has an opening angle of> 180 °, so that if them on the rear surface 18th of the trailer 16 of the motor vehicle 10 is mounted, all edges 38 of the trailer 16 "able to see. There 2 shows a plan view from above and thus essentially a cross-sectional view, only the edge is schematic here 38 above the fisheye lens 34 to see the fisheye lens 34 can of course also capture the edges on the left, right and below.

3 Figure 12 shows a rear view of the trailer 16 of the truck 12 standing on a subsurface 40 is located. In this rear view are all four edges 38 - relative to the mobile camera unit 20th , "Below", "above", "right", "left" - to be seen. The distances au, ao, ar and al of these edges 38 to the mobile camera unit 20th are shown with dashed arrows.

These distances ouch , ao , ar , al can the mobile camera unit 20th detect statically. It receives static position parameters pus, pos, prs, pls of the edges 38 which it records using classic image processing methods.

At the same time is the mobile camera unit 20th able as in 4th is clearly shown in the perspective view, an optical flow 42 to capture when the truck is 12 is in motion. The mobile camera unit 20th can do the optical flow 42 for all spatial positions 44 “Down”, “up”, “right” and “left” relative to the camera unit 20th capture. The optical flow data 42 who have favourited the mobile camera unit 20th recorded are sent to an in 3 Image analysis device shown 46 sent and evaluated, based on a jump in the respectively detected optical flow 42 for the four spatial positions 44 the edges 38 can be recognized. This makes it possible to set dynamic position parameters pud , pod , prd , pld the edges 38 to detect.

As in 3 to see features the camera unit 20th via an accelerometer 48 , which is a 3 or 6-axis accelerometer 48 can be. About this accelerometer 48 can the mobile camera unit 20th determine with the help of the acceleration of gravity where the spatial position 44 "Down" is located. The camera unit 20th therefore knows when capturing the optical flow 42 that this is the optical flow 42 the spatial position "below" and thus that of the subsurface 40 acts. Here it is possible to get acceleration data of the moving vehicle 10 to take help and thus to record whether the motor vehicle 10 is accelerated or decelerated. Thus, the detected optical flow 42 for the underground 40 can be determined even more precisely. This also allows the optical flow 42 for the remaining spaces 44 “Right”, “left”, “top” can be precisely determined.

Accordingly, there are now the static position parameters pus, pos, prs, pls and the dynamic position parameters pud , pod , prd , pld available, which can then be superimposed or merged. The actual position parameters can be derived from the fusion of these position parameters, some of which may still be incomplete or occasionally incorrect or ambiguous put , pot , prt , plt determine with a significantly higher degree of certainty and precision.

In the in 3 The embodiment shown, the static detection takes place for the lower edge 38 of the trailer 16 with the help of a horizontal edge 38 of a structure 50 of the trailer 16 , however, it is also possible to use the 3 shown horizontal edges 38 a bumper 52 that lie underneath.

In 3 are in addition to the statically determined distances ouch , ao , ar , al , which are indicated by dashed arrows, also the distances ou , oo , or , ol drawn dynamically via the optical flow 42 have been determined. Here you can see that ou is different from au. By means of the optical flow 42 and intrinsic parameters of the camera unit 20th becomes the distance ou to the ground 40 immediately behind the trailer 16 determined. This distance ou also gives the height of the attached camera unit 20th contrary. In the in 3 The illustrated embodiment, the distances ao, ar, al are each identical to the distances oo , or , oil . However, it is also possible that these distances also differ from one another and can therefore also be checked for plausibility or corrected by merging the data.

With the help of intrinsic parameters of the camera unit 20th which are known can thus use the local coordinates x, y, z of the camera unit 20th to be determined. These are subsequently sent to an in 6th Central processing unit shown 54 of the entire camera system 22nd sent.

When the motor vehicle 10 is in motion, it follows a certain trajectory 56 , in the 4th is shown. The data received from the camera unit 20th are recorded, correlate to one another, so that the movement or trajectory is also derived from this 56 of the motor vehicle 10 can be determined. From the comparison of the trajectory determined from the camera data 56 with the actual trajectory 56 of the motor vehicle 10 all data obtained can be checked for plausibility and filtered in order to obtain a more robust information base.

In 5 the process steps taking place in the process described above are shown schematically. First is the mobile camera unit 20th on the surface 18th of the motor vehicle 10 attached. Then first the static position parameters pus, pos, prs, pls of the edges 38 detected, after which, as soon as the motor vehicle 10 is in motion, including the dynamic position parameters pud , pod , prd , pld the edges 38 are recorded. The two kinds of position parameters become actual position parameters put , pot , prt , plt fused and from this the local coordinates x, y, z of the camera unit 20th calculated. In the last step, these local coordinates x, y, z are sent to the central processing unit for further processing 54 transmitted.

6th shows a camera system 22nd , which can be, for example, an ADAS system or a simple multi-camera system, but in the present embodiment it is a camera-monitor system 58 (Mirror replacement system) is what the mobile camera unit 20th comprises, and in which the method for calibrating the mobile camera unit 20th is carried out to total the camera system 22nd to be able to operate. The schematic representation in 6th shows next to the mobile camera unit 20th that are flexible on the motor vehicle 10 can be attached, also a permanently installed camera unit 60 , with both camera units 20th , 60 Camera data to the camera system 22nd deliver. The one from the mobile camera unit 20th The data supplied are stored in a first processing unit 62 evaluated with regard to the static position parameters pus, pos, prs, pls. The camera system also includes 22nd a second processing unit 64 by the image analyzer 46 is formed, and over which from the respective optical flows 42 the dynamic positional parameters pud , pod , prd , pld can be calculated. The ones in both arithmetic units 62 , 64 determined data become a third processing unit 66 transmitted which from the central processing unit 54 further data on the trajectory 56 of the motor vehicle 10 and additional acceleration data 68 receives. The third computing unit can use this 66 the local coordinates x, y, z of the mobile camera unit 20th calculate and to the central processing unit 54 to transfer. The central processing unit 54 processes these determined local coordinates x, y, z and can then use them from the camera unit 20th captured images on a monitor 70 , together with images that the camera unit 60 returns, represent in system coordinates xs, ys, zs.

The arithmetic units 62 , 64 , 66 can in an alternative embodiment also in the mobile camera unit 20th be included. The individual steps can then be performed on the mobile camera unit depending on the system design 20th and / or on the central processing unit 54 , for example an automobile ECU, take place. In case of doubt, the mobile camera unit can 20th determine the local coordinates x, y, z completely independently.

With the help of known optical properties of the camera unit 20th and their optics, the areas of interest can be optimized, for example by restricting a search area.

The optical flow 42 for the different spatial positions 44 and the geometric calculations are carried out in a normalized manner and after the step of lens correction. However, it can also be the case that these calculations are made with non-rectified data.

Known methods such as “tracking” common features via different camera units could optionally also be used 20th , 60 can also be used across the board to create more redundancy and, ideally, more robustness.

The method described is particularly interesting for a mobile camera system 22nd with changing followers 16 or attached machines. There will be an optimal integration of the camera unit 20th in multi-camera / multi-field of view systems. This avoids the need for calibration in the workshop.

Most known methods are based on common features with different camera units 20th , 60 to pursue. However, the camera units are often 20th , 60 , for example a camera arm and a mobile reversing camera, are very far apart. In addition, there is the very large opening angle of the fisheye optics on the mobile camera unit 20th . Both properties together mean that the classic "feature tracking" methods using camera units 20th , 60 are difficult to implement. The approach described above consists in combining the respective local information using known knowledge and therefore being able to convert the coordinate systems into one another.

In the present example, it can be known, for example, that the trailer rear wall on which the camera unit 20th is pinned to one side of the trailer 16 borders. Thus there is an intersection of the two surfaces 18th a side face 18th of the trailer 16 and the rear of the trailer 16 that is on an edge 38 bump. This edge 38 can, for example, be tracked using mirror replacement cameras and thus stored dynamically in a system coordinate system.

The method naturally also enables the calibration of statically mounted cameras, for example rear-view cameras, and can also be used with optics with an opening angle of> 180 ° in just one plane.

Claims (8)

A method for calibrating a mobile camera unit (20) of a camera system (22) for a motor vehicle (10), the camera unit (20) being designed to detect an optical flow (42), comprising the steps: - Fastening the mobile camera unit (20) at any position (p) on a surface (18) of the motor vehicle (10) in such a way that at least two mutually perpendicular edges (38) of the surface (18) are in a field of view (28) of the Camera unit (20) lying; - Static detection of static position parameters (pus, pos, pls, prs) of the at least two edges (38); - Dynamic detection of dynamic position parameters (pud, pod, pld, prd) of the at least two edges (38) with the aid of the detected optical flow (42); - Merging the static position parameters (pus, pos, pls, prs) with the dynamic position parameters (pud, pod, pld, prd) to form actual position parameters (put, pot, plt, prt); - Calculating local coordinates (x, y, z) of the camera unit (20) on the basis of the determined actual position parameters (put, pot, plt, prt) and intrinsic parameters of the camera unit (20); - Transmission of the local coordinates (x, y, z) to a central processing unit (54) of the camera system (22). Procedure according to Claim 1 , characterized by detecting a spatial position (44) “below” relative to the camera unit (20) by detecting a surface (40) on which the motor vehicle (10) is arranged with the aid of an acceleration sensor (48). Procedure according to Claim 2 , characterized by detecting the optical flow (42) for the underground (40). Procedure according to Claim 3 , characterized by refining the detected optical flow (42) with the aid of acceleration data (68) of the moving motor vehicle (10). Method according to one of the Claims 1 to 4th , characterized by detecting the optical flow (42) from at least one further spatial position (44) “right”, “left” and / or “above” relative to the camera unit (20). Method according to one of the Claims 1 to 5 , characterized by providing an image analysis device (46) which is designed to recognize the at least two edges (38), in particular more than two edges (38), of the motor vehicle (10) on the basis of a jump in the detected optical flow (42). Procedure according to Claim 6 , characterized by determining a trajectory (56) of the motor vehicle (10) in motion from the optical flow (42) of several spatial positions (44) and also checking the plausibility of the recorded data with regard to the optical flows (42) of the several spatial positions (44) With the aid of an actual trajectory (56) of the motor vehicle (10) in motion. Camera monitor system (58) with a mobile camera unit (20) and a central processing unit (54), which are designed to implement a method according to one of the Claims 1 to 7th perform.

Images