DE102021124430B3 - Visualize lidar measurement data - Google Patents

Abstract

A method for visualizing lidar measurement data is specified, in particular lidar measurement data from a laser scanner, with remission values associated with the lidar measurement data being recorded and color values being reconstructed from the remission values using a machine learning method.

Description

The invention relates to a method for visualizing lidar measurement data according to the preamble of claim 1 and a system with a lidar sensor and a control and evaluation unit for carrying out the method according to claim 11.

A lidar sensor (light detection and ranging, in analogy to radar) measures spatially resolved distances to objects touched, so that the lidar measurement data enables a three-dimensional detection and measurement of an environment. A typical and widely used lidar sensor is a laser scanner. There are also other lidar sensors such as time-of-flight cameras and solid-state scanners that mimic the macroscopic scanning mechanism of the laser scanner using micromechanical means or even purely electronic means.

In a laser scanner, a scanning beam scans an area and measures the light travel time of the remitted or reflected light, which can be converted into a distance using the speed of light. Two types of time-of-flight methods are widely used. With phase-based methods, the light transmitter modulates the scanning beam and the phase between a reference and the received scanning beam is determined. Pulse-based methods imprint a significant pattern on the scanning beam, for example a narrow pulse lasting only a few nanoseconds, and determine the time at which this pattern is received. In a generalization known as the pulse averaging method, a number of pulses or a pulse sequence are sent out and the received pulses are statistically evaluated.

Conventional laser scanners have a rotating mirror or a polygon mirror wheel in order to periodically scan a surveillance plane or a segment of a surveillance plane. Alternatively, an active measuring head rotates with a light transmitter and light receiver. 3D laser scanners capture a three-dimensional space and not just an area. This can be realized by movement around another axis, be it in a pivoting or a full rotation movement. In the DE 10 2008 032 216 A1 For example, the entire scanning unit including transmitter, receiver and rotating mirror is arranged on a deflection plate. the EP 1 965 225 A2 reveals a wide variety of options for combining the rotating movement of a laser scanner with an additional tilting movement.

Also known are multilayer laser scanners which have a plurality of scanning beams arranged one above the other in elevation. That's it U.S. 8,767,190 B2 It is known to provide a separate light transmitter and light receiver for several scanning planes, so that the individual scanning planes can be adjusted in the desired height or elevation grid. the US 2017/0289524 A1 discloses an optical system for detecting distance information, in which a series arrangement of light sources is provided on the transmission side and an image sensor is provided on the reception side behind a large receiving lens with an aperture array.

Laser scanners and, more generally, lidar sensors are excellent for capturing geometries. In mobile applications, such as mounted on a vehicle, robot or hand truck, consistent and accurate metric maps of the environment can be recorded. This can then be used as a basis for navigation, for example (in particular SLAM, Simultaneous Localization and Mapping), or geometric variables such as the floor area of a hall can be determined from the map.

It is also often desirable to give people who are not physically present a visual impression of the environment using a map, for example for planning, monitoring or other diagnostics that are also spatially or temporally offset or as part of a virtual reality application. Dealing with lidar technology and the maps generated from it is typically not part of the expertise of the relevant specialists. Therefore, a natural interpretability with a low-threshold, intuitive access is desired, which neither a lidar point cloud nor a corresponding map can convey.

A conventional approach consists of combining a color camera with the lidar sensor and fusing its image data with the lidar measurement data. The result is easier to grasp visualizations in the form of a colored point cloud or a colored surface reconstruction (meshes). However, the additional camera including a suitable lens and the effort for the camera calibration for the lidar sensor as a prerequisite for data fusion increase the price of the overall structure considerably. In addition, the additional color detection can in some cases significantly lengthen the measurement cycles. Thus, the acceptance of the technology, which is actually increased by a lowered entry threshold, is ultimately bought too expensively.

An alternative approach, which does not require an additional color camera, includes the remission values (RSSI, Received Signal Strength Indicator) of the lidar sensor, which are derived from the backscattered reception energy. On the one hand, however, these are scalar quantities, which only allow monochromatic visualization. With the colors, psychologically very important information is missing in order to find your way around intuitively. Secondly, a lidar system usually does not work in the visible spectrum, but typically in the (near) infrared range. The remission values can therefore not be interpreted directly by the human eye, but must be translated into false colors or brightness. While false colors are, so to speak, by definition unintuitive and once again require experience on the part of the viewer, the brightness obtained from the infrared has the problem that in some cases it differs significantly from what is expected. Plants are highly reflective because of their chlorophyll, and black car tires in the visible spectrum glow in the infrared, so to speak.

the U.S. 2020/0357190 A1 deals with such a 3D representation of lidar point clouds in which the colors are linearly assigned to the RSSI along the rainbow. Physiologically, however, there is no linear relationship between brightness and color, so only false colors arise, and consequently the difficulties just described arise here.

In recent years, methods of machine learning and in particular deep neural networks (deep learning) have solved many classic problems such as segmentation, object recognition, superresolution or noise reduction with previously unimagined efficiency, especially in image processing. Approaches were also developed to recover the color information from monochromatic images (colorization). Related is the work of Zhang, Richard, Phillip Isola, and Alexei A. Efros. "Colorful image colorization." European conference on computer vision, Springer, Cham, 2016. The web link https://deoldify.ai/ shows further examples for coloring, which also include video sequences.

However, these machine learning methods cannot handle lidar measurement data. This has to do with the very different properties in terms of the field of view, spatial arrangement of the measurement data, time behavior and the spectral ranges of a lidar sensor and a camera. In the prior art discussed above, not even this problem is recognized, let alone solved.

The two papers Kim, Hyun-Koo, et al. "Asymmetric Encoder-Decoder Structured FCN Based LiDAR to Color Image Generation", Sensors 19.21 (2019): 4818 and Kim, Hyun-Koo, et al., "Color Image Generation from Range and Reflection Data of LiDAR" Sensors 20.18 (2020) : 5414 each deal with the coloring of LIDAR data using a neural network.

It is therefore the object of the invention to improve the visualization of lidar measurement data.

This object is achieved by a method for visualizing lidar measurement data and a system with a lidar sensor and a control and evaluation unit for carrying out the method according to claims 1 and 11, respectively. Lidar measurement data is the measurement data generated by a lidar sensor, typically in the form of a 3D point cloud, although the specific representation is irrelevant. The lateral spatial resolution (x,y) is preferably created by macroscopic-mechanical movement such as in a laser scanner with a rotating mirror or rotating measuring head, alternatively by micro-mirrors or electrical control of a solid-state sensor. The distance dimension Z is recorded using the time-of-flight method of the lidar principle, in which a light signal is sent out, received again and the time-of-flight is measured.

In addition, the energy of the received light signal is measured in order to obtain a remission value for the respective measurement points of the lidar measurement data. The various physical terms for the strength of the received signal, such as intensity, level, energy or others, are interchangeable, and in the following the RSSI value is usually used as a substitute for the remission value. Color values are reconstructed from the respective remission values. As explained in the introduction, this reconstruction is clearly underdetermined and therefore highly ambiguous, since the color space has three dimensions, for example in the form of luminance and a, b in the CIE Lab color space or red, green and blue in RGB color coding, while the remission value only is a scalar.

The invention is based on the basic idea of using a machine learning method for the reconstruction of the color values from the remission values. This process can also be referred to as coloring (colorization) of lidar measurement data. Input data for the machine learning method are therefore the remission values and, at least implicitly, the geometric relationships of the measurement points based on their arrangement, and output data are plausible color values that are assigned to a respective measurement point. In contrast to the conventional approach as in U.S. 2020/0357190 A1 , which assigns (false) colors to a single remission value, the machine learning process relates the neighborhood and thus the struc environment. At least implicitly, according to the invention, the knowledge of known objects and their typical coloring is included.

The invention has the advantage that a color reproduction that is at least plausible, if not close to or true to reality, is made possible. This significantly increases the attractiveness and intuitive accessibility of lidar applications, since the lidar measurement data are at least much more similar to the usual film recordings or photographs. This succeeds, although the lidar measurement data only provide remission data with all the problems with color assignment described in the introduction. Compared to the use of an additional color camera, a significant cost reduction of the entire system is achieved, and this in relation to the system structure, installation and cabling, the software and the recording of the lidar measurement data. In particular, the measuring cycles are not lengthened, which is very noticeably the case with conventional high-resolution laser scanners with additional color recording.

A two-dimensional remission image is preferably generated from the lidar measurement data and the associated remission values. In the simplest case, the measured distances of the lidar measurement data are simply left out and the remission values are assigned to the (x,y) values, analogous to a photograph. A more complex rendering takes into account the 3D relationships known from the LiDAR measurement data and the perspective of the remission image. The dimensions of the reflectance image are thus already adjusted to a conventional color image and, after coloring, can be displayed directly on a conventional display.

Preferably, the remission image is transformed into a format that corresponds to the format of color images used to train the machine learning method. In other words, the remission image is processed geometrically in order to adopt a comparable format to the color images that were used for training and that are ultimately also intended to be generated for visualization. Most lidar sensors have a very different field of view than a camera, e.g. 360° lateral and 30° in elevation of a multi-layer or height-adjustable laser scanner versus 70° in both directions of a camera, as well as different optics and perspectives. However, the structure of the lidar sensor used to record the lidar measurement data is known, so that it can be transformed mathematically.

The remission image is preferably generated by means of a projection of the lidar measurement data into a virtual plane, in particular a spherical, cylindrical or stereographic projection. This is an example of a specific representation (rendering) of the lidar measurement data. The special projections mentioned take into account the recording principle of the lidar sensor and are therefore particularly suitable for generating remission images that the machine learning process can deal with, which ultimately compares with ordinary color images, albeit in a complex manner.

The reflectance values are preferably corrected using a relationship between distance and reflectance loss. A lidar sensor and a camera also differ in this point, in particular the dynamic range of a lidar sensor is much higher for physical reasons. The RSSI value thus depends not only on the reflectivity of the object being touched, but also on its distance. This is an entirely unfamiliar effect for the brightness of a camera recording of this magnitude and for color perception or recording. Since the distance was measured using the time of flight of light and is therefore known, this distance dependency can be calculated out. For this purpose, for example, a table (LUT, lookup table) with RSSI values of targets with different reflectivity is recorded at different distances and corrected with it or with a corresponding function or interpolation.

The lidar measurement data is recorded with infrared light, with the remission values being first transformed from the infrared spectrum into the visible spectrum. The use of infrared light is common for time-of-flight methods. However, no brightness levels corresponding to a black-and-white image are recorded with this, so that existing machine learning approaches for coloring cannot be used. Therefore, a preparatory transformation into the visible spectrum makes sense. This transformation can follow a fixed calculation rule. Alternatively, a machine learning method, in particular a neural network, is also used for this purpose. This further method of machine learning relates to the preparatory step of converting to the visible spectrum and must not be confused with that for the actual coloring. However, it is alternatively conceivable that a common machine learning process carries out both steps. However, this would at least require a different training, if not a different architecture than for just one of the two combined steps alone.

The remission values are first transformed into a red value. Because of the spectral Proximity between infrared and red, such a transformation to a red value can be superior to that to a gray or lightness value, both in terms of ease of transformation and the ultimate color result.

The machine learning method preferably has a linear model, a decision tree, a neural network or a combination of several neural networks, a Gaussian process regression, a k-nearest neighbor method or a support vector machine. The appropriate machine learning method can be selected based on the available evaluation resources, training data and the requirements of the possible applications. The machine learning method here means that with which color values are reconstructed from remission values. Likewise, the selection of possible machine learning methods for preparatory steps such as the transformation from the infrared spectrum to the visible spectrum is available.

The machine learning method preferably has a deep convolutional neural network. Although there are other possibilities of machine learning, some of which are mentioned in the previous paragraph, the deep convolutional neural networks have recently shown their superiority over all other conventional methods and also methods of machine learning. Approaches for coloring monochrome camera images can be applied preferably after some of the preparatory steps discussed, in particular for the geometric adjustment between lidar measurement data and a camera image as well as the transformation from the infrared spectrum to the visible spectrum. An example of the implementation of the invention that is also made accessible to lidar measurement data is the approach of the work by Zhang et al.

The reflectance values are first transformed into red values, and then green and blue values are reconstructed from the red values using the machine learning method. The red values can be transformed particularly well from the remission values because of the spectral proximity to the infrared range. The additional color dimensions, i.e. green values and blue values, are then reconstructed using the machine learning method. This deliberately deviates from the CIE Lab color space from Zhang et al. to better reflect the nature of lidar measurement data. Alternatively, however, it is also conceivable to transform the remission values into luminance values and to reconstruct the color values a, b of the CIE Lab color space using the machine learning method.

The machine learning method is preferably pre-trained with a general image database. There are countless freely available images, for example from an image database such as https://www.image-net.org/ or the LOCO data set at https://github.com/tum-fml/loco. A huge treasure trove of training images is thus available. A special feature of the problem of coloring is that the transformation in the opposite direction from color to black and white - or to red - is completely trivial and the otherwise tedious step of annotating or labeling can be automated very easily.

The machine learning method is preferably trained with data from an application environment, in particular an industrial environment. This enables specific training for special features of the actual application situation. Such individual training is preferably based on general training, so that far fewer training examples are required. Typical objects to be trained individually in an industrial environment are hall walls and floors, machines or indoor vehicles.

Training data for the machine learning method are preferably generated by jointly recording lidar measurement data and color images. A combination of a lidar sensor and a color camera is only used for training. Costs such as recording times play a subordinate role here, since the combination is only needed temporarily. The training is significantly more specific, since lidar measurement data is now also available as the original input data.

The system according to the invention, consisting of a lidar sensor for recording lidar measurement data and a computing unit, is designed for visualizing or coloring the lidar measurement data using a method according to the invention. A display is preferably also provided on which the result can be presented, in particular in the form of a color image with the reconstructed colors. The computing unit can be virtually any computing unit or plurality of homogeneous or heterogeneous computing units, such as an internal digital building block such as a CPU, an FPGA, a GPU, an NPU of the lidar sensor or several of them, or any form of computers , which are connected directly or via a network or a cloud and in turn have any number of such digital building blocks. The most computationally intensive step, coloring or color inference, does not need to be done on-site or in real-time for most practical use cases, so the on acquisition of the lidar measurement data is not affected.

The method according to the invention can be developed in a similar way and shows similar advantages. Such advantageous features are described by way of example but not exhaustively in the dependent claims which follow the independent claims.

• 1 eine schematische Schnittdarstellung eines mehrlagigen Laserscanners als Beispiel eines Lidar-Sensors; und
• 2 ein beispielhaftes Ablaufdiagramm für das Einfärben von Lidar-Messdaten.

The invention is explained in more detail below, also with regard to further features and advantages, by way of example on the basis of embodiments and with reference to the attached drawing. The illustrations of the drawing show in:

• 1 a schematic sectional view of a multilayer laser scanner as an example of a lidar sensor; and
• 2 an example flowchart for coloring lidar measurement data.

1 FIG. 1 shows a schematic sectional illustration through an exemplary laser scanner 10, which is designed in multiple layers for a 3D acquisition of lidar measurement data. The laser scanner 10 roughly comprises a movable scanning unit 12 and a base unit 14. The scanning unit 12 is the optical measuring head, while the base unit 14 accommodates further elements such as a supply, evaluation electronics, connections and the like. During operation, a drive 16 of the base unit 14 is used to set the scanning unit 12 in a rotational movement about a rotational axis 18 in order to periodically scan a monitoring area 20 in this way.
In the scanning unit 12, a light emitter 22 with a plurality of light sources 22a, for example LEDs or lasers in the form of edge emitters or VCSELs, generates a plurality of transmitted light beams 26 with a mutual angular offset with the aid of common transmitting optics 24, which are emitted into the monitoring area 20. Alternatively, the plurality of transmitted light beams 26 can arise via fewer or only one light source and at least one beam splitter or pattern generation element. The number of four transmission light beams 26 is only an example, there can be more or fewer. A single transmitted light beam is conceivable, then a 2D laser scanner is created, which can be expanded to a 3D laser scanner by additional movement in elevation.

If the transmitted light beams 26 hit an object in the monitored area 20 , corresponding reflected light beams 28 return to the laser scanner 10 . The reflected light beams 28 are guided by receiving optics 30 to a light receiver 32 with a plurality of light receiving elements 32a, each of which generates an electrical reception signal. The light receiving elements 32a may be separate components or pixels of an integrated array such as photodiodes, APDs (Avalanche Diode) or SPADs (Single-Photon Avalanche Diode).

Light transmitter 22 and light receiver 32 are in 1 shown embodiment arranged together on a printed circuit board 34, which lies on the axis of rotation 18 and is connected to the shaft 36 of the drive 16. This is only to be understood as an example; practically any number and arrangement of printed circuit boards is conceivable. The basic optical structure with a biaxially adjacent light transmitter 22 and light receiver 32 is also not mandatory and can be replaced by any design known per se from single-beam optoelectronic sensors or laser scanners. An example of this is a coaxial arrangement with or without a beam splitter.

A non-contact supply and data interface 38 connects the movable scanning unit 12 to the stationary base unit 14. A control and evaluation unit 40 is located there, which can also be accommodated at least partially on the printed circuit board 34 or at another location in the scanning unit 12. The control and evaluation unit 40 controls the light transmitter 22 and receives the received signals from the light receiver 32 for further evaluation. It also controls the drive 16 and receives the signal from an angle measuring unit, not shown, which is generally known from laser scanners and which determines the respective angular position of the scanning unit 12 .
For the evaluation, the distance to a touched object is preferably measured using a known time-of-flight method. Together with the information about the angular position from the angle measuring unit, two-dimensional polar coordinates of all object points in a scanning plane are available after each scan period with angle and distance. The respective scanning plane is also known via the identity of the respective reflected light beam 28 and its detection in one of the light receiving elements 32a, so that a three-dimensional spatial area is scanned overall. The object positions or object contours are thus known and can be output via a sensor interface 42, for example as a 3D point cloud. Conversely, the sensor interface 42 or another connection (not shown) serves as a parameterization interface.

In addition to the distance measurement, the control and evaluation unit 40 is able to determine the intensity of the transmitted light beams 28 that are reflected. A respective remission value is thus available for the measuring points, in particular an RSSI (Received Signal Strength Value).

The laser scanner 10 shown has a rotating measuring head, namely the scanning unit 12. Not only one transceiver module can rotate as shown here, other such modules with a height offset or an angular offset with respect to the axis of rotation 18 are conceivable. Alternatively, a periodic deflection by means of a rotating mirror or a faceted mirror wheel is also conceivable. This requires additional considerations because how the multiple transmitted light beams 26 fall into the monitored area 20 depends on the respective rotational position. They rotate around each other with the movement of the rotating mirror, but in a predictable way that can therefore be compensated for in the 3D measuring points.

Instead of a laser scanner 10, another lidar sensor can be used for recording lidar measurement data, for example a 3D point cloud of the surroundings, together with associated remission values. These use, for example, matrix arrangements of single-light transmitters and single-light receivers, micro-actuators for scanning, such as a piezo actuator, a MEMS mirror or an adaptive lens or liquid lens, or purely electronic scanning methods such as optical phased arrays or acousto-optical modulators.

2 shows an example flowchart for coloring lidar measurement data using a machine learning method. Steps S1 and S2 relate to its training phase and steps S3 to S8 to the application phase, which is subdivided into step S3 with acquisition of the lidar measurement data, steps S4 to S7 with processing of the lidar measurement data and step S8 with the actual color inference.

In a step S1, the machine learning method is generally trained or pre-trained. The details of the training depend on the specific machine learning method used, which is discussed further below at step S8. For example, in supervised learning of a convolutional network, monochrome training images are presented, with the original color image dictating the desired outcome. Public databases can be used for this, such as those accessible at https://www.image-net.org/ or https://github.com/tum-fml/loco. Pairs of monochrome and color images can be created very easily, since color reduction is very easy to do, in contrast to coloring.

Instead or as an optional supplement, application-specific training takes place in a step S2. Training is not done with random images, but with those from industrial environments or even the specific application environment. It is also possible to train with lidar measurement data or remission values instead of images. In order to obtain associated color images, a lidar sensor can be linked to a color camera and calibrated. In this way, validation and test data sets are also obtained in order to find optimal networks or to investigate different network architectures experimentally. Instead of a combination of a lidar sensor and a color camera, the use of a time-of-flight camera with an integrated color camera is conceivable, with its characteristics and thus training results not being transferrable to other lidar sensors or laser scanners without restrictions.

In the application phase, lidar measurement data and associated remission values are recorded in a step S3. For this purpose, for example, the laser scanner 10 according to FIG 1 or another lidar sensor is used. The lidar measurement data and remission values can be composed of several measurements from the same lidar sensor at the same or different location or from a combination of several lidar sensors.

The lidar measurement data and remission values are now processed in steps S4 to S7. Steps S4 to S6 create an adjustment in format and geometry, in particular generate a remission image R(x,y) that is compatible with the color images I(x,y) used in training, i.e. covers comparable geometric viewing areas and thus the same objects depicts. In step S7, the transition from the typical IR spectrum of a time-of-flight measurement to the visible spectrum is completed, for which purpose the remission image R(x,y) is transformed to an estimated, monochrome image T(x,y) in the visible spectral range. These steps S4 to S7 are optional; in principle, the lidar measurement data and remission values as such can form the input data for the machine learning method. However, significantly better results are achieved with individual, several or all of these steps.

In a step S4, a remission image R(x,y) is generated from the remission values. The lidar measurement data is typically available as a 3D point cloud and cannot be directly compared with a color image. A format adjustment is created by generating the remission image.

A geometric adjustment follows in a step S5. A lidar sensor, in particular a laser scanner, typically has a very large horizontal opening angle of up to 360° with only a limited vertical extent in comparison to an opening angle of, for example, 70° horizontally and vertically, which extends more evenly in both directions, of a camera. Without compensatory measures, only a very narrow overlapping area would result.

This is most evident with a 2D laser scanner. In order to obtain any height resolution at all, the scan plane is varied by an additional actuator, and the respective height lines of the successive scans are arranged to form a remission image. In a multi-layer scanner like that of 1 the contour lines are generated simultaneously.

There is the possibility of reprojecting the lidar measurement data or remission values into a virtual plane, for example using spherical, cylindrical or stereographic projection. Color images for training can be projected accordingly. The projections have their respective advantages and disadvantages, and it also depends on the desired visualization result which distortions and distortions should be avoided or tolerated.
If the requirements cannot be met by grid corrections or geometric projections, more complex techniques can be used. An example of this is the structure of a 3D mapping graph, for example according to Grisetti, Giorgio, et al. "A tutorial on graph-based SLAM", IEEE Intelligent Transportation Systems Magazine 2.4 (2010): 31-43 with a reprojection of the resulting point cloud into synthetic remission images in the appropriate perspective and orientation. In addition, a surface estimation per truncated signed distance function can be performed on the 3D mapping graph, see Klingensmith, Matthew, et al. "Chisel: Real Time Large Scale 3D Reconstruction Onboard a Mobile Device using Spatially Hashed Signed Distance Fields." Robotics: science and systems. Vol. 4. No. 1. 2015. This can mitigate the problem of different exposure of the surroundings of a passive camera and an active lidar sensor, for example by means of additional shading (shader). Synthetic light sources that are positioned to mimic real light incidence, such as on windows or skylights, can help further reduce the difference between a synthetic remission image and a camera-like brightness image.

In a step S6, the remission values are corrected as a function of the distance. The reflectivity of the touched object point is of interest for the visualization or the coloring, without a distance-related energy loss. Since the respective distance is measured, a correction only requires knowledge of the corresponding distance-dependent relationship between measured remission and reflectivity. For this purpose, targets with a defined reflectivity, for example with 2%, 10%, 50% and 100%, can be measured in advance with the lidar sensor at different distances. A look-up table for the correction results, which can be appropriately interpolated or otherwise represented and used if required.

In a step S7, remission values recorded with the lidar sensor in the infrared are transferred to the visible spectrum. Expressed somewhat more formally, a function f is used, which converts reflectance values of the reflectance image R(x,y) into values of a single-channel image in the visible spectrum T(x,y). In this case, T(x,y) preferably corresponds to a color channel of the later colored image, ie it contains luminance values for the CIE Lab color space, for example. First of all, the direct interpretation of the remission values measured in the infrared as brightness in the visible spectrum is possible. Due to the spectral closeness of red and infrared, it can be better not to fill T(x,y) with brightness values, but rather as the red channel of an RGB image.

A method of machine learning, in particular a neural network, can be used not only for the actual coloring in step S8, which will be explained immediately, but also for the function f that maps R(x,y) to Î(x,y). However, semantic information, as identified almost incidentally with methods of machine learning (see, for example, Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556, 2014), can also can be obtained classically, for example by Hough transformation, in order to specifically detect round objects in the image. In each case, the reflectance image R(xy) generated in the infrared is converted into a feature vector, analogously to the color image I(x,y) used for training or for estimating the function f. Of the color image, however, only that channel is used into which f transforms, for example the luminance channel according to the CIE Lab color model or the red channel of an RGB image, which is physically closer to the infrared spectrum. A common procedure in training f or the neural network for implementing f then minimizes a loss function, i.e. the error between the current prediction f(R(x,y)=Î(x,y) of the neural network and the actual one Color channel I(x,y) For associated object areas F(n), this can be done with a Euclidean loss function as a minimization of L = Σn ||Î(F(n)) -I(F(n)) || ² formulate.

In a step S8, after these preparations, which, as mentioned, can at least partially be omitted depending on the embodiment, the actual coloring or color inference takes place using the machine learning method trained in steps S1 and S2. The color channel transformed from the remission values is specified, for example the luminance channel or the red channel, and the two other color channels a, b or green and blue are plausibly supplemented. Thanks to the preparations, a method for coloring that is known per se, as described in the work by Zhang et al. applicable, to which reference is made for further details. If the starting point for this is the red channel instead of the luminance channel, the perceptual differences should be properly weighted and the loss function adjusted for this: the direct Euclidean loss in the CIE Lab domain becomes a transformation of the G, B estimates together with that in this one Step given R-channel into the CIE lab space to calculate the adjusted loss.

Using a deep convolution mesh as in Zhang et al. is particularly beneficial. In principle, other methods of machine learning are conceivable, for example a linear model, a decision tree, a neural network, a Gaussian process regression, a k-nearest neighbor method or a support vector machine. Alternatively, color information can also be assigned in the classic way via object detectors, in particular by means of template matching on the processed remission images. Detected objects are assigned a plausible color, for example by accessing a previously created database with objects and associated colors.

Coloring and color inference can be based on single measurements or single scans, or rendered remission images are generated from specifically chosen locations in the map to produce dense, more photo-like images. In connection with a better comprehensible representation, methods of style transfer are also conceivable, in which case, unlike the normal case of a style transfer, no spatial changes are made in the image. When capturing lidar measurement data with multiple lidar sensors, a mutual RSSI comparison analogous to step S6 is advantageous.

The color inference can usually be done offline, i.e. after the actual recording of the lidar measurement data, so that no additional costs and computing times are incurred in the sensor system itself. The coloring can even be a by-product or combined with other image processing steps such as object detection or object segmentation. For example, the work Dharmasiri, Thanuja, Andrew Spek, and Tom Drummond deals with such a combination of tasks. "Joint prediction of depths, normals and surface curvature from rgb images using cnns." 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2017, to which additional reference is made.

Claims (11)

Method for visualizing lidar measurement data recorded with infrared light, in particular lidar measurement data from a laser scanner (10), with the lidar measurement data being used to record associated remission values and color values to be reconstructed from the remission values using a machine learning method, characterized in that first the reflectance values are transformed into red values and then green and blue values are reconstructed from the red values using the machine learning method. procedure after claim 1 , whereby a two-dimensional remission image is generated from the lidar measurement data and the associated remission values. procedure after claim 2 , where the remission image is transformed into a format corresponding to the format of color images used to train the machine learning method. procedure after claim 1 or 2 , wherein the remission image is generated by means of a projection of the lidar measurement data in a virtual plane, in particular a spherical, cylindrical or stereographic projection. Method according to one of the preceding claims, in which the reflectance values are corrected using a relationship between distance and reflectance loss. Method according to one of the preceding claims, wherein the machine learning method comprises a linear model, a decision tree, a neural network, a Gaussian process regression, a k-nearest neighbor method or a support vector machine. A method according to any one of the preceding claims, wherein the machine learning method comprises a deep convolutional neural network. Method according to one of the preceding claims, wherein the method of the machine learning with a general image database. Method according to one of the preceding claims, wherein the machine learning method is trained with data from an application environment, in particular an industrial environment. Method according to one of the preceding claims, wherein training data for the machine learning method are generated by joint recording of lidar measurement data and color images. System with a lidar sensor (10) for recording lidar measurement data and with a computing unit that executes a visualization of the lidar measurement data using a method according to one of the preceding claims.

Images