DE102020101706A1 - Method for generating depth image pairs for a database - Google Patents

Abstract

Method for generating valid basic distance data for a time-of-flight camera, in which in a detection step, at least one time-of-flight camera and a laser scanner, at least piecewise distance values of a scene are recorded, wherein in a first processing step, the distance values of the time-of-flight camera with the distance values of the laser scanner is a probable point in space of the scenery are assigned, and in a second processing step the data are optimized by means of image processing and combined into training distance data pairs and stored in a database.

Description

The invention relates to a method for generating depth image pairs for a database for use in a time-of-flight camera according to the preamble of the independent claim.

Such time-of-flight cameras or time-of-flight camera systems of this type relate in particular to all time-of-flight or TOF camera systems that obtain transit time information from the phase shift of an emitted and received radiation. PMD cameras with photonic mixer detectors (PMD) are particularly suitable as time of flight or TOF cameras, as they are, for example, in FIG DE 197 04 496 C2 are described. The PMD camera allows in particular a flexible arrangement of the light source and the detector, which can be arranged both in a housing and separately.

The object of the invention is to improve the accuracy of a time-of-flight camera.

The object is achieved by the method according to the invention.

The invention is explained in more detail below on the basis of exemplary embodiments with reference to the drawings.

• 1 einen Aufbau zur Erfassung und Einlernen valider Entfernungsdaten,
• 2 eine erfindungsgemäße Datenerfassung und -aufbereitung.

They show schematically:

• 1 a structure for recording and teaching in valid distance data,
• 2 a data acquisition and processing according to the invention.

1 schematically shows a structure according to the invention with a 3D laser scanner on a tripod and with at least one TOF camera mounted on the scanner. As in 2 As shown, the scan and ToF image are registered and processed with image processing algorithms to create a database of accurate training data pairs. Error-free depth values and optionally processed ToF images are then used to train a machine learning model. The model receives ToF images as input and outputs exact depth maps for any ToF cameras.

The time-of-flight camera used measures phase-shifted periodic functions, the phase being proportional to the depth d. It is common practice to design these measurements as real (Re) and imaginary (Im) parts of a complex sinusoidal signal (often referred to as a raw image). The phase is defined as: $$\mathrm{\varphi }=\text{atan}2\left(-\text{re},\text{in the}\right)$$

The depth is then calculated as: $$\text{d}=\text{ur /}2\mathrm{\pi \varphi }$$

The unique range (ur) of a ToF camera is limited by the use of its modulation frequency (f). $$\text{ur}=\text{c /}2\text{f}$$

With c = speed of light.

1. a) Das Rauschen wird in der Regel mit räumlichen, kernelbasierten (adaptiven-) Filterverfahren reduziert. Auch eine zeitliche Filterung wird verwendet.
2. b) Mehrwege-Interferenz entsteht durch Mehrfachreflexionen in der Szene, bevor das Licht zur Kamera zurückkehrt. Dies führt zu überbestimmten Tiefenwerten und ist stark szenenabhängig. Derzeit ist kein klassischer Algorithmus in der Lage, diesen Fehler in Echtzeit zu korrigieren.
3. c) Die Fehler durch die Phasenzuordnung (phase unwrapping) werden typischerweise mit dem chinesischen Restwert Theorem und mehreren Frequenzen gelöst. Fehler in der Zuordnung zu einem der möglichen Eindeutigkeitsbereichen können insbesondere durch Rauschen entstehen. Der maximale Bereich der Messung ist zudem noch durch das kleinste gemeinsame Vielfache der eindeutigen Bereiche der verwendeten Modulationsfrequenzen begrenzt.

The depth measurement suffers from several errors. In particular, noise, multipath interference and so-called "phase unwrapping" (phase assignment) errors must be taken into account.

1. a) The noise is usually reduced with spatial, kernel-based (adaptive) filter methods. Temporal filtering is also used.
2. b) Multipath interference is caused by multiple reflections in the scene before the light returns to the camera. This leads to overdetermined depth values and is heavily dependent on the scene. No conventional algorithm is currently able to correct this error in real time.
3. c) The errors due to the phase assignment (phase unwrapping) are typically solved with the Chinese residual value theorem and several frequencies. Errors in the assignment to one of the possible uniqueness areas can arise in particular due to noise. The maximum range of the measurement is also limited by the smallest common multiple of the unique ranges of the modulation frequencies used.

Machine learning models are able to deal with these problems, but require large amounts of training data with known error-free depth data. Ray tracing methods are currently used to generate significant amounts of synthetic training data with appropriate error-free depth. Unfortunately, synthetic data has significant disadvantages due to various simplifications and inadequacies made in the ray tracer.

As described at the beginning and in 1 shown, the data acquisition consists of a laser scanner, which is arranged, for example, on a motorized tripod. In addition, at least one time-of-flight camera is mounted on the scanner (optionally with rotating stepper motor). The data can, for example, via a computer, which is connected to the laser scanner and USB, for example via Wifi is connected to the time-of-flight camera (s) and possibly stepper motors. The PC is primarily used to coordinate automatic laser scanning and ToF image acquisition.

2 shows a possible procedure according to the invention for generating valid distance data for a neural network. The basic idea is to offer the system as large a number of different scenarios as possible with different problems in the distance measurement. For example, some scenarios can have corners and edges that favor multipath propagation.

A 3D laser scanner, for example, can be used as the reference measuring system, distance references or a reference depth image can be obtained from the scan point cloud. Due to the point-like scanning of the scanner, this measuring method is typically not affected by multipath propagation and thus provides an essentially error-free depth image. The Tof camera, on the other hand, is susceptible to multi-path propagation due to the flat lighting of the scene. The aim of the procedure according to the invention is, among other things, to train such effects and to make the learned data available in a neural network.

After the basic data has been recorded, the raw data is processed in such a way that the depth images from the time-of-flight camera can be registered together with the point cloud from the laser scanner. In particular, an extrinsic calibration can also be advantageous here in order to correct undesired stereoscopic effects, for example. These effects occur due to the finite distance between the scanner and the ToF sensor.

After the raw data pairing, image processing can be provided which, for example, generates a depth image from the scan point cloud or scan raw data. The scan depth image can then be paired with the raw ToF images, for example, or the raw ToF images are converted into another representation, such as a phase or distance image. In principle, the desired representation can also be calculated from the raw ToF images at a later point in time.

Finally, the image processing outputs error-free depth images with associated ToF images. Additional processing of the ToF images is optional as described above (phase, depth, noise filter, calibration ...).

The training data pairs obtained on the basis of preferably several thousand recorded different scenarios are stored in a database or a neural network. This database is universal for the respective ToF camera type and can be used to improve the distance measurements

1. a) Montage der Lichtlaufzeitkamera auf einem Motor pro Rotationsachse. Eine Software ändert dann automatisch die Blickwinkel der Kamera durch Drehen der Motoren und nimmt so mehrere einzigartige Bilder pro Scan auf.
2. b) Verwendung von mehreren Lichtlaufzeitkameras in verschiedenen Positionen und Winkeln, die auf dem Laserscanner montiert sind. Dies hat den zusätzlichen Vorteil, dass verschiedene Kameras mit unterschiedlichen Kalibrierfehlern verwendet werden können. Dadurch wird die statistische Wirksamkeit der Informationserweiterung (augmentation) erhöht.
3. c) Aufnahme der ToF-Bilder bei verschiedenen Belichtungszeiten, um verschiedene Signal-Rausch-Verhältnisse darzustellen (z.B. 5 verschiedene Stufen).

An essential aspect of the invention is data replication. This means that the amount of training data pairs per scan should be much larger than 1. Preferably tens of thousands of data pairs should be created. Each scan should preferably provide> = 10 data pairs in order to build up a comprehensive database for training. Several methods are suitable to achieve this:

1. a) Installation of the time-of-flight camera on one motor per axis of rotation. Software then automatically changes the camera's angle of view by rotating the motors and thus takes several unique images per scan.
2. b) Use of several time-of-flight cameras in different positions and angles, which are mounted on the laser scanner. This has the additional advantage that different cameras with different calibration errors can be used. This increases the statistical effectiveness of information augmentation.
3. c) Recording the ToF images at different exposure times in order to show different signal-to-noise ratios (eg 5 different levels).

All methods can be combined to achieve the best possible data amplification. For a laser scanner with a field of view (fov) of 300 ° × 360 ° and a ToF camera with 45 ° x60 ° fov, we can call up ~ 36 unique points of view. With 5 exposure times we get 5x36 = 180 training samples per scan, so that a database of 100x180 = 18k training pairs is created.

As in 1 As shown, the machine learning model receives training pairs from the database. The model is entered either in the form of raw ToF images or, if necessary, further processed images (phase, depth, ...). The model outputs depth images (or images on a lower level, such as phase, raw data, ...). Training on experimental data is another crucial part of the invention. This method leads to a better generalization of the machine learning model, since no approximations or assumptions (e.g. only Lambertian scattering, pure Gaussian noise) have to be made in order to calculate synthetic ToF images. Therefore, training on experimental data is more efficient and leads to better trained models (regardless of the specific model architecture).

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• DE 19704496 C2 [0002]

Claims (2)

Method for generating depth image pairs for a database in the form of a neural network for a time-of-flight camera that works according to the phase measurement principle, with a reference distance measuring system for the acquisition of reference depth images and with at least one time-of-flight camera, wherein the reference distance measuring system has a larger detection area and a higher spatial resolution than the at least one time-of-flight camera (ToF camera), wherein the detection area of the at least one time of flight camera lies within the detection area of the reference distance measuring system, - wherein in one detection step the time of flight camera and the reference distance measuring system detect a three-dimensional scene, and a raw ToF image from the time-of-flight camera is recorded together with the raw data from the reference measurement system. where, based on the raw data of the reference measurement system, a reference depth image is combined with a raw ToF image or a processed ToF depth image to form a training data pair, the training data pairs being stored in a database in the form of a neural network. Time-of-flight camera that accesses the database with the training distance data obtained according to the aforementioned method to determine distance values.

Images