WO2009043789A1

WO2009043789A1 - Sensor unit and method for calibration thereof

Info

Publication number: WO2009043789A1
Application number: PCT/EP2008/062832
Authority: WO
Inventors: Wendelin Feiten; Gisbert Lawitzky
Original assignee: Siemens Aktiengesellschaft
Priority date: 2007-09-27
Filing date: 2008-09-25
Publication date: 2009-04-09
Also published as: EP2193334A1; DE102007046287B4; DE102007046287A1

Abstract

Different sensors such as laser scanners (32) and video cameras (35) are first mounted securely onto a common substrate (30) and calibrated to each other. This considerably simplifies subsequent calibration of the resultant multi-sensor system (11) at the employment site. The incorporation of a video camera which is precalibrated in this manner, makes possible the automatic, computerized extraction of the location of the tracks from the video picture of the video camera when using a loading crane system (10). Thus, the need for manually positioned calibration elements is eliminated. This reduces faults in ongoing operation during activation of the multi-sensor system.

Description

description

Method for calibrating a sensor arrangement and sensor arrangement

A sensor assembly, for example mounted on a crane, is used to measure (or estimate) the position and attitude of moving objects, such as the crane itself or a cargo, e.g. a container. Other uses include the measurement of the position and position of a vehicle or a movable component of the crane itself into consideration.

The crane may be, for example, a loading crane. Loading cranes are used at freight transhipment points, warehouses, in assembly halls and shipyards as well as in track construction. In a loading crane for motor vehicles, the floor is inclined relative to the loading crane, so that water can drain. Furthermore, tracks for trucks are marked on the ground under the loading crane.

One embodiment of a loading crane is a gantry crane. This spans a loading and working area like a portal. As a rule, its sidewalls with wheels run on two parallel rails. On the crane bridge, the horizontal part of the gantry crane, a trolley moves with a hoist.

Alternatively, a rail slewing crane can be mounted on the crane bridge.

Furthermore, a gantry crane, a gantry crane, a gantry crane and a gantry crane are also possible.

Moving parts of a crane are z. For example, the trolley or the spreader - a hoist with which containers can be grabbed.

In the context of a loading crane, sensor array measurements serve as a basis to signal truck drivers where to stop them. Furthermore, due to such measurements, the crane itself can be controlled.

The sensor arrangement may, for example, be composed of one or more of the following elements: a 3D laser scanner, a pivotable 2D laser scanner or a video camera. The elements of the sensor arrangement are usually mounted in such a way in the structure of the crane that - in the case of a gantry crane - several tracks for trucks or railroad railroad tracks are covered.

In order for the sensor assembly to be used in the manner described, it is necessary to calibrate it. This means that the following coordinate systems must be related to each other:

The sensor coordinate system of one or more laser scanners installed in the sensor arrangement,

the camera coordinate system of one or more cameras, provided that they are installed in the frame of the sensor arrangement,

the crane coordinate system of the crane with respect to e.g. Trolley and spreader are located,

- The ground coordinate system, with respect to which, if necessary, tracks for trucks or railroad railroad tracks are given, which are loaded and unloaded.

The calibration is achieved in the prior art, for example, in that a specially prepared and prepared for this purpose calibration body is placed on the ground in the area of the crane and measured manually with respect to the crane or the crane coordinate system by a surveyor. In addition, lanes can be measured manually with respect to the crane or to the calibration body. The sensor arrangement subsequently detects the calibration body, from which coordinate transformations between the sensor coordinate system and the other coordinate systems can be obtained. The disadvantage here is that the ongoing operation of the crane for calibration longer time must be interrupted.

Frequently, structuring features can be found in the surroundings of the loading crane system, which can only be detected optically. An example of this is ground markings painted on the ground indicating the location of the lanes. When incorporating an optical sensor (such as a camera) into the sensor array, a coordinate transformation between the camera coordinate system and the other coordinate systems must be determined.

This is made more difficult in the applications described in large crane systems by their large dimensions and non-standardized visual characteristics. For example, the lane markings may be different. For this reason, it is necessary in the prior art to introduce further calibration bodies, which are precisely defined in terms of their optical and geometrical features, into the system in order to carry out the calibration on the basis of these. In the context of automation of

Loading crane system is a specific calibration of the sensor arrangement for the respective workflows desirable. Even a manual measurement of the lane markers means additional effort here.

It is therefore the task of specifying a method for calibrating a sensor arrangement and a sensor arrangement, which simplify the calibration of the sensor arrangement at the place of use.

This object is achieved by the method for calibrating a sensor arrangement and the sensor arrangement according to the independent patent claims. Preferred developments emerge from the dependent claims.

In the method for calibrating a sensor arrangement, the following steps are carried out: First, a first sensor and a second sensor are mounted on a carrier, whereby the sensor assembly is formed. Subsequently, a coordinate transformation between a first coordinate system of the first sensor and a second coordinate system of the second sensor is determined. In a third step, the sensor assembly is mounted on a site. Finally, the sensor arrangement is calibrated in relation to an environment at the place of use, wherein the determined coordinate transformation between the first coordinate system and the second coordinate system is used.

The sensor arrangement has a carrier on which a first sensor with a first coordinate system and a second sensor with a second coordinate system are mounted.

Furthermore, the sensor arrangement has a computing unit on which a coordinate transformation between the first coordinate system and the second coordinate system is stored.

By mounting the first and second sensors in advance on a common support, the sensor assembly thus formed can already be calibrated during manufacture. As a result, a pre-calibrated multi-sensor system is formed. This considerably simplifies later calibration at the place of use.

The calibration of the sensor arrangement thus takes place in two steps:

1. First, the coordinate systems of the sensors contained in the sensor assembly are calibrated to each other in the factory. This can be done, for example, with reference to each other calibrated calibration and calibration.

2. After mounting the sensor assembly at the job site, the calibration is done in relation to the new environment. This makes the procedure for calibration particularly robust and accurate. The ongoing operation at the site is less disturbed by the switching on of the sensor arrangement. Furthermore, it is possible by the upstream calibration to include a video camera in the sensor array, as this does not have to be recalibrated after installation at the site. The use of a video camera supports the modeling of other aspects of the respective application, such as persons, previously unmodeled vehicles or freights.

In a development of the method, the sensor arrangement is mounted on an object at the place of use and acquires measurement data of its surroundings. The measured data identifies soil measurement data for parts of a soil underneath the object as well as object measurement data for parts of the object. The object measurement data are used to determine a soil coordinate system from the soil measurement data. A coordinate transformation between the first coordinate system and the ground coordinate system is calculated. The sensor array is calibrated relative to the field environment using the coordinate transformation between the first coordinate system and the second coordinate system and the coordinate transformation between the first coordinate system and the ground coordinate system.

In a development of the sensor arrangement, this is set up for mounting on an object. It has an arithmetic unit, which is set up for the identification of soil measurement data for parts of a soil under the object as well as object measurement data for parts of the object in the measurement data. The arithmetic unit is further configured to determine a ground coordinate system from the ground measurement data using the object measurement data, to calculate a coordinate transformation between the first coordinate system and the ground coordinate system, and to calibrate the sensor array based on the coordinate transformation between the first coordinate system and the second coordinate system and the coordinate transformation between the first coordinate system and the ground coordinate system.

The refinements have the advantage that it is possible to dispense with a separate calibration body for calibrating the sensor arrangement. This saves costs. Furthermore, eliminates the burden of creating, interim storage, placement and clearing the calibration. The connection of the sensor arrangement thus requires less effort and causes a lower disturbance of the current operation.

According to a development, the object is a crane, in particular a loading crane, gantry crane, bridge crane, semi-portal crane, gantry crane or portal crane, or any movable or static object on which the sensor arrangement can be mounted. A gantry crane offers the advantage that its pronounced symmetry properties can be used for the calibration.

According to a development, the sensor arrangement comprises one or more pivotable 2D laser scanners. The use of a tiltable 2D laser scanner offers the advantage of its large field of view that in addition to vehicles in the working area of the crane, large parts of the supporting structure of the crane itself can be detected.

According to a development, the sensor arrangement comprises a camera, for example a video camera. This has the advantage that on the ground mounted lane markers can be detected by the camera and included in the calibration.

According to a development, the parts of the object are side walls of a gantry crane. This offers the advantage that the orientation of these side walls can be used to determine the object coordinate system.

In the following an embodiment of the invention will be explained in more detail with reference to figures. Show it: 1 crane with a sensor arrangement and a cargo under the crane,

FIG. 2 shows distance measurement data of the sensor arrangement,

3 shows a crane coordinate system, a sensor coordinate system and a ground coordinate system, FIG.

FIG. 4 is a flowchart of the method;

Figure 5 shows a sensor arrangement in which a plurality of sensors are pre-mounted on a support.

Figure 1 shows a crane 10. On the crane 10, a sensor assembly 11 is mounted, which consists of two elements in the case shown in Figure 1. Also shown is a cargo 12, such as a container on a truck, which is detected by the sensor assembly 11. Also seen in Figure 1 are wheels 14 with which the crane 10 can be moved on rails. A floor 15 under the crane 10 is inclined, so that water can flow away. On the floor 15 lane markers 13 are attached, which mark tracks for vehicles.

FIG. 2 shows measurement data of the sensor arrangement 11, in this case distance measurement data of a laser scanner. In the measurement data, ground measurement data 21 of parts of the floor 15 under the crane 10 and crane measurement data 22 of parts of the crane 10 can be identified. This allows a geometric measurement of the crane and its working space.

The measurement data can be obtained, for example, by pivoting a 2D laser scanner over the parts of the crane 10 and the parts of the floor 15.

The crane measurement data 22 are here as rectangles in 3D out-segmented side walls of the crane 10. Measured data 21 are accordingly outsourced ground points. Of these, only a subset may need to be used to achieve sufficient accuracy; This saves computing time and storage requirements.

FIG. 3 again shows the crane 10, its wheels 14, the sensor arrangement 11 as well as the floor 15 and the lane markings 13. Additionally shown are a ground coordinate system 16, a crane coordinate system 17 and a sensor coordinate system 18 of the sensor arrangement 11.

FIG. 4 shows a flowchart for calibrating the sensor arrangement 11. In a first step 1, the sensor arrangement 11 acquires measurement data of its surroundings as shown in FIG. These are distance measurement data. In a second step 2, the ground measurement data 21 shown in FIG. 2 and the crane measurement data 22 are identified in the measurement data. This can be done computer-aided using plan drawings, which are accurate enough with respect to the mounting position of the sensor assembly 11 and its elements and other elements of the crane 10. The measurement data are given here as 3D measurement points, which are all initially present in the sensor coordinate system 18 of the sensor arrangement 11.

In a third step 3, the crane measurement data 22 are used to determine a ground coordinate system 16 from the ground measurement data 21. In a fourth step 4, a coordinate transformation between a sensor coordinate system 18 of the sensor arrangement and the ground coordinate system 16 is calculated, by means of which the sensor arrangement 11 is calibrated.

The method for calibrating the sensor arrangement 11 uses symmetries, such as surface symmetries or translation symmetries in three-dimensional space. Such symmetries, the crane 10 on - especially in the design as a gantry crane. In this case, the symmetries are extracted from the crane measurement data 22 and possibly the ground measurement data 21. This is recognizing that a surface of 3D measurement points represented by the crane measurement data 22 has a normal vector that can be used as the y direction vector of the crane coordinate system 17. Furthermore, it is possible to make use of the fact that a surface of 3D measurement points represented by the ground measurement data 21 has a normal vector which can be selected as the z direction vector of the ground coordinate system 16. Surprisingly, the x-direction vector of the ground coordinate system 16 can be selected to be identical to the x-direction vector of the crane coordinate system 17. This x-direction vector is both parallel to the ground 15 and parallel to a surface represented by the crane measurement data 22. This becomes clear with the example of the gantry crane. As it travels on rails, they run parallel to the ground as well as parallel to the inner walls of the gantry crane. The x-direction vector of both the ground coordinate system 16 and the crane coordinate system 17 can thus be selected parallel to the rails. Thus, the x-direction vector is perpendicular both to the normal vector of the area of 3-D measurement points represented by the ground measurement data 21 and to the normal vector of the area of 3-D measurement points represented by the crane measurement data 22. On the basis of this, these coordinate systems can be successively developed.

In the following, an algorithm suitable for this purpose is given by way of example. Any other algorithms are possible. In the alternative, the algorithm calculates the crane coordinate system 17, which is used for the derivation of the ground coordinate system 16.

In the scenario of a gantry crane in a harbor, 22 3D measurement points are identified as crane measurement data that belong to the sidewall (the so-called "sill bar"), on the one hand 3D measurement points belonging to the seaward side wall and, on the other hand, 3D measurement points that belong to the landside side wall. For gantry cranes on the mainland, an arbitrary ge other characterization of the two sides of the gantry crane to be selected.

A major axis transformation (described in: Jonathon Shlens: "A Tutorial on Principal Component Analysis", available on the Internet on July 2, 2007 at http://www.snl.salk.edu/~shlens/pub/notes/pca.pdf) The 3D measuring points of the seaward side wall of the crane 10 as well as on the 3D measuring points of the landside side wall of the crane 10 allows in each case the formation of a normal vector in the center of gravity of the respective 3D measuring points. From these two normal vectors, the y-direction vector of the crane coordinate system 17 is now formed by averaging. The convention is that the y-direction vector points to the sea side.

Also by main axis transformation, a normal vector is formed on the 3D measurement points in the ground measurement data 21 and selected as z direction vector of the ground coordinate system 16.

Subsequently, the x direction vector of both the ground coordinate system 16 and the crane coordinate system 17 is obtained from the cross product of the y direction vector of the crane coordinate system 17 and the z direction vector of the ground coordinate system 16.

In the next step, the z direction vector of the crane coordinate system is calculated from the cross product of the x direction vector and the y direction vector of the crane coordinate system 17. Accordingly, the y-direction vector of the ground coordinate system 16 results from the cross product of the z-direction vector and the x-direction vector of the ground coordinate system 16.

As the origin of the crane coordinate system 17, the center between the base points of the normal vectors formed from the crane measurement data 22 is first selected. Subsequently, the 3D measurement points in the crane measurement data 22, which belong to the seaward side wall and the side wall on the landside, are transformed into the crane coordinate system 17. In the course of this, their expansion into x-

Direction determined as minimum and maximum. Thereafter, the crane coordinate system 17 is shifted in its x-direction so that its origin lies in the middle between the determined minimum and maximum. Thus, the crane coordinate system 17 is completely determined.

As the origin of the ground coordinate system 16, the center of gravity of the 3D measurement points in the ground measurement data 21 is first determined. Subsequently, the z-axis of the crane coordinate system 17 is cut with the plane formed by the 3D measurement points in the ground measurement data 21. The intersection chosen as the origin of the ground coordinate system 16.

Now, lanes under the crane 10 can be manually between

Tracks on which the wheels 14 of the crane 10 run are measured. The position of the center between the tracks relative to the sensor arrangement 11 results from the previously determined coordinate systems.

On the basis of the determined coordinate systems, a coordinate transformation to the sensor coordinate system 18 of the sensor arrangement 11 is calculated in each case. With this, the sensor assembly 11 is calibrated.

FIG. 5 shows a crane 10 which can be moved on wheels 14 via a floor 15. On the floor lane markers 13 are located. Furthermore, two sensor arrangements 11 are shown in FIG. The left sensor assembly 11 is shown enlarged within the dashed circle. The sensor arrangements 11 have a sensor coordinate system 18. The sensor arrangement 11 shown enlarged consists of a carrier 30, on which a first Sensor 32 and a second sensor 35 are mounted. The first sensor 32 is mounted via a rotary drive 31 on the carrier 30, whereby the first sensor 32 is pivotally mounted. Furthermore, the first sensor 32 has a first coordinate system 181 and the second sensor has a second coordinate system 182. In the following, it is assumed that the sensor coordinate system 18 coincides with the first coordinate system 181.

In the context of a crane installation, the first sensor 32 is a 2D laser scanner, for example, which can be pivoted via the rotary drive 31 and used to record a two-dimensional environment image. The second sensor 35 is approximately a camera, e.g. a video camera. The video camera is particularly suitable for detecting the lane markers 13 on the floor 15 under the crane 10. The carrier 30 may for example be designed as a mounting frame.

First, the first sensor 32 and the second sensor 35 are mounted on the carrier 30, whereby the sensor assembly 11 is formed. Subsequently, a coordinate transformation between the first coordinate system 181 of the first sensor 32 and the second coordinate system 182 of the second sensor 35 is determined. This can be done, for example, with reference to each other calibrated calibration and calibration.

In a third step, the sensor assembly 11 or the carrier 30 is mounted on the site, such as the crane bridge. Finally, the sensor arrangement 11 is calibrated in relation to its surroundings at the place of use according to the method described above, wherein the determined coordinate transformation between the first coordinate system 181 and the second coordinate system 182 is used.

In addition to the method described above, the visible in the video image of the video camera lane markers 13 with be known methods of image processing are extracted, which are initially present in the second coordinate system 182. By applying the coordinate transformation between the first coordinate system 181 and the second coordinate system 182, the position of the lane markers 13 in the first coordinate system 181 can be determined. Although no distance information is included in the video image. However, these can be determined by the boundary condition that the lane markings 13 must lie in the plane of the floor 15 spanned by the x and y direction vector of the ground coordinate system 16. Since the coordinate transformation between the sensor coordinate system 18 (identical to the first coordinate system 181 above) and the ground coordinate system 16 has previously been determined, the position of the lane markers 13 in the first coordinate system 181 can be calculated herewith.

Thus, in this application, the lanes can be detected automatically under the crane system and included in an environmental model of the crane system. Such an environment model allows modeling and control of the plant. In addition to the position of the coordinate systems, the position of the lane markings 13 or of the lanes is thus also determined computer-aided.

Only the factory calibration of the video camera to the first coordinate system 181 or sensor coordinate system 18 present 3D measurement points of the laser scanner allows, as described above at the site automatically from the simultaneous image of the video camera, the position of the lane markers 13 extract.

The same applies to any other areas of application of the sensor arrangement 11. The exemplary embodiments described can also be implemented analogously in other scenarios.

Claims

claims

A method of calibrating a sensor assembly (11), comprising the steps of: - a first sensor (32) and a second sensor (35) being mounted on a support (30), thereby forming the sensor assembly (11), a coordinate transformation between one first coordinate system (181) of the first sensor (32) and a second coordinate system (182) of the second sensor (35) is determined

the sensor arrangement (11) is mounted at a place of use,

- The sensor assembly (11) is calibrated in relation to an environment at the site, the coordinate transformation between the first coordinate system (181) and the second coordinate system (182) is used.

2. Method according to claim 1, - in which further sensors are mounted on the carrier (30),

- In each case a coordinate transformation between the first coordinate system (181) of the first sensor

(32) and coordinate systems of the other sensors is determined,

- In which the additionally determined coordinate transformations are used after assembly at the site in the calibration of the sensor array.

3. The method of claim 1, wherein as the first sensor (32) a laser scanner and as a second sensor (35) a camera, in particular a video camera is used.

4. The method of claim 1, wherein - the sensor assembly (11) is mounted at the site on an object and recorded measurement data of their environment, in the measurement data, ground measurement data (21) for parts of a floor (15) under the object as well as object measurement data for parts of the object are identified,

the object measurement data are used to determine a ground coordinate system (16) from the ground measurement data (21),

- a coordinate transformation between the first coordinate system (181) and the ground coordinate system (16) is calculated, - the sensor array (11) is calibrated in relation to the environment at the site, wherein the coordinate transformation between the first coordinate system (181) and the second Coordinate system (182) and the coordinate transformation between the first coordinate system (181) and the ground coordinate system (16) is used.

5. The method according to claim 3 and 4,

in which track markings (13) are extracted from an image of the camera, which are initially present in the second coordinate system (182),

in which the sensor arrangement (11) is calibrated in relation to the track markings (13) by the coordinate transformation between the first coordinate system (181) and the second coordinate system (182) and the coordinate transformation between the first coordinate system (181) and the Ground coordinate system (16) is applied to determine the location of the lane markers (13) in the first coordinate system (181).

6. The method of claim 4, wherein the object is a crane (10), in particular a loading crane, gantry crane, bridge crane, semi-portal crane, gantry crane or portal crane.

7. The method of claim 6, wherein the parts of the object are side walls of a gantry crane.

8. The method of claim 4, wherein the plan drawings of the object are used to identify the ground measurement data (21) and the object measurement data computer-aided.

9. The method according to claim 4,

in which a y-direction vector of an object coordinate system is determined on the basis of the object measurement data,

in which a z direction vector of the ground coordinate system (16) is determined on the basis of the ground measurement data (21), in which an x direction vector of both the ground coordinate system (16) and the object coordinate system is determined by means of a Cross product of the y-direction vector of the object coordinate system and the z-direction vector of the ground coordinate system (16) is calculated, - in which a z-direction vector of the object coordinate system of a cross product of the x-direction vector and the y-direction vector of the object Coordinate system is calculated,

in which a y-direction vector of the ground coordinate system (16) is calculated from a cross product of the z-direction vector and the x-direction vector of the ground coordinate system (16).

10. The method of claim 9, wherein the y-direction vector of the object coordinate system is formed by main axis transformation or principal component analysis by a normal vector at the center of gravity of 3D measurement points in the object measurement data.

11. The method of claim 9, wherein the z-direction vector of the ground coordinate system (16) by means of main axis transformation or principal component analysis by a normal vector in the center of gravity of 3D measurement points in the soil measurement data (21) is formed.

12. sensor arrangement (11),

- with a carrier (30) to which a first sensor (32) with a first coordinate system (181) and a second sensor sor (35) are mounted with a second coordinate system (182), and

- With a computing unit on which a coordinate transformation between the first coordinate system (181) and the second coordinate system (182) is stored.

13. Sensor arrangement (11) according to claim 12, wherein the first sensor (32) is a laser scanner and the second sensor (35) is a camera, in particular a video camera.

14. Sensor arrangement (11) according to claim 13, wherein the laser scanner on the support (30) via a rotary drive (31) is pivotally mounted.

15. Sensor arrangement (11) according to claim 12,

in which additional sensors with additional coordinate systems are additionally mounted on the carrier (30),

- In which on the arithmetic unit additional coordinate transformations between the first coordinate system (181) and the coordinate systems of the other sensors are stored.