DE102021128537A1 - Time-of-flight camera and method for operating one - Google Patents

Abstract

Method for operating a time-of-flight camera that includes a time-of-flight sensor with time-of-flight pixels, with the steps: - Acquisition of raw data using the time-of-flight sensor, - Correction of the raw data by the raw data of the time-of-flight sensor being fed to a machine learning model trained with regard to multipath propagation and an inference of the model machine learning is used to correct the raw data, the machine learning model having been trained with experimental multipath error-prone training data pairs,- providing the corrected raw data for a subsequent determination of distance values.

Description

The invention relates to a time-of-flight camera and a method for operating such a camera, a use and a measurement setup according to the species of the independent claims.

Time-of-flight cameras or time-of-flight camera systems relate in particular to all time-of-flight or TOF camera systems that obtain transit-time information from the phase shift of an emitted and received radiation. PMD cameras with photomixing detectors (PMD) are particularly suitable as time-of-flight or TOF cameras, as they are used, for example, in DE 197 04 496 C2 are described. In particular, the PMD camera allows a flexible arrangement of the light source and the detector, which can be arranged either in one housing or separately.

A device for determining distances is known from US 2017/0 262 768 A1, in which distances are determined using a machine learning model. This model was trained with at least one pair of simulated raw data. The model is trained to determine distance values in a single step from the stored raw time-of-flight sensor data, with the machine learning model being trained using simulated raw data containing simulated multipath propagations.

the EP 3 832 351 A1 describes an apparatus and method for training a machine learning model for a time-of-flight camera. The model is trained with simulated data based on a synthetic scene and a simulated response of an emitted TOF signal.

The object of the invention is to improve the accuracy of a time-of-flight camera.

The object is achieved by the method according to the invention.

• - Erfassung von Rohdaten mit Hilfe des Lichtlaufzeitsensors,
• - Korrektur der Rohdaten,

indem die Rohdaten des Lichtlaufzeitsensors einem hinsichtlich Mehrwegausbreitung trainierten Modell maschinellen Lernens zugeführt werden und eine Inferenz des Modells maschinellen Lernens zur Korrektur der Rohdaten verwendet wird,
wobei das Modell maschinellen Lernens mit experimentellen
mehrwegfehlerbehafteten Trainingsdatenpaare trainiert wurde,

• - Bereitstellung der korrigierten Rohdaten für eine nachfolgende Ermittlung von Entfernungswerten.

A method for operating a time-of-flight camera that includes a time-of-flight sensor with time-of-flight pixels is advantageously provided, with the steps:

• - Acquisition of raw data using the time-of-flight sensor,
• - correction of the raw data,

by feeding the raw data of the time-of-flight sensor to a machine learning model trained with regard to multipath propagation and using an inference of the machine learning model to correct the raw data,
where the machine learning model with experimental
was trained with multipath error-prone training data pairs,

• - Provision of the corrected raw data for a subsequent determination of distance values.

This procedure has the advantage that, before the raw data is forwarded, the raw data can be improved with regard to artefacts that occur as a result of multipath propagation. Corrected raw data is then available for the subsequent process steps without the subsequent process steps having to be additionally adjusted.

A subsequent determination of distance values preferably takes place without using a machine learning model, starting from the corrected raw data.

This procedure has the advantage that the machine learning model is only used to determine and compensate for multipath propagations, while the distance is calculated, for example, using typical and known algorithms.

Furthermore, the machine learning model designed as a convolutional neural network is useful.

Furthermore, it is advantageous to construct a time-of-flight camera for carrying out one of the aforementioned methods.

It is particularly useful to use the machine learning model that was trained using any of the above methods as a cost function for training other machine learning models.

A measurement setup with which training data for use in the time-of-flight camera can be experimentally determined is also helpful.

The invention is explained in more detail below using exemplary embodiments with reference to the drawings.

• 1 einen Aufbau zur Erfassung und Einlernen valider Rohdaten,
• 2 eine erfindungsgemäße Datenerfassung und -aufbereitung
• 3 ein Training eines Modells maschinellen Lernens,
• 4 eine Verwendung des Modells maschinellen Lernens bei der Mehrwegfehlerabschätzung.

They show schematically:

• 1 a structure for recording and teaching valid raw data,
• 2 an inventive data acquisition and processing
• 3 a training of a machine learning model,
• 4 a use of the machine learning model in multipath error estimation.

1 shows schematically a structure according to the invention with a 3D laser scanner on a tripod and with at least one time-of-flight camera mounted on the scanner.

The time-of-flight camera used measures phase-shifted periodic functions, where the phase is proportional to the depth d. It is common to construct these measurements as the real (Re) and imaginary (Im) parts of a complex sine wave (referred to herein as raw data). The phase is defined as: $$\mathrm{\varphi =}\text{atan}2\left(-\text{re}\text{,In the}\right)$$

The depth is then calculated as: $$\text{i.e}=\text{ur}/2\mathrm{\pi }\mathrm{\varphi }$$

Mit c = Lichtgeschwindigkeit.The unique range (ur) of a ToF camera is limited by the use of its modulation frequency (f). $$\text{ur}=\text{c}/\text{2f}$$

With c = speed of light.

1. a) Das Rauschen wird in der Regel mit räumlichen, kernelbasierten (adaptiven-) Filterverfahren reduziert. Auch eine zeitliche Filterung wird verwendet.
2. b) Mehrwege-Interferenz entsteht durch Mehrfachreflexionen in der Szene, bevor das Licht zur Kamera zurückkehrt. Dies führt zu überbestimmten Tiefenwerten und ist stark szenenabhängig. Derzeit ist kein klassischer Algorithmus in der Lage, diesen Fehler in Echtzeit zu korrigieren.
3. c) Die Fehler durch die Phasenzuordnung (phase unwrapping) werden typischerweise mit dem chinesischen Restwert Theorem und mehreren Frequenzen gelöst. Fehler in der Zuordnung zu einem der möglichen Eindeutigkeitsbereichen können insbesondere durch Rauschen entstehen. Der maximale Bereich der Messung ist zudem noch durch das kleinste gemeinsame Vielfache der eindeutigen Bereiche der verwendeten Modulationsfrequenzen begrenzt.

The depth measurement suffers from several errors. In particular, noise, multipath interference and so-called "phase unwrapping" (phase assignment) errors must be taken into account.

1. a) The noise is usually reduced with spatial, kernel-based (adaptive) filtering methods. Temporal filtering is also used.
2. b) Multipath interference is caused by multiple reflections in the scene before the light returns to the camera. This leads to overdetermined depth values and is highly scene-dependent. No classic algorithm is currently able to correct this error in real time.
3. c) The phase unwrapping errors are typically solved using the Chinese remainder theorem and multiple frequencies. Errors in the assignment to one of the possible unambiguous areas can arise in particular from noise. The maximum range of the measurement is also limited by the smallest common multiple of the unambiguous ranges of the modulation frequencies used.

Machine learning models are able to deal with these problems but require large amounts of training data with known sound depth data. In contrast to synthetic training data that is often used, experimental training data does not have a so-called domain gap between the data that a machine learning model sees in training and the data that a real time-of-flight camera measures for inference. Therefore, the accuracy of the machine learning model when trained on experimental data is significantly higher than when using synthetic data.

As described above and in 1 shown, the data acquisition or the measurement setup consists of a laser scanner, which is arranged, for example, on a motorized tripod. In addition, at least one time-of-flight camera is mounted on the scanner (optionally with a rotating stepper motor). The data can, for example, be sent via a computer that is connected to the laser scanner and USB to the time-of-flight camera(s) and stepper motors, for example, via Wifi. The PC is preferably used to coordinate the automatic laser scanning and the time-of-flight image acquisition.

This laser scanner is a reference measurement system here and is used to obtain a raw reference data item from the scan point cloud. Due to the punctiform sampling of the scanner, this measurement method is typically not affected by multipath propagations and thus provides an essentially error-free depth image. Likewise, there are no phase assignment errors and the noise is very low. The time-of-flight camera, on the other hand, is particularly susceptible to multipath propagation (besides other measurement errors) due to the planar illumination of the scene. The aim of the procedure is, among other things, to train such effects and to make the learned data available in a neural network or machine learning model.

1. 1. Eine extrinsische Kalibration der Lichtlaufzeitkamera mit dem Laserscanner liefert eine fehlerfreie Punktewolke aus der Perspektive der Lichtlaufzeitkamera (gemessen vom Laserscanner) und ein Rohdatum aus der Messung der Lichtlaufzeitkamera.
2. 2. Die resultierende fehlerfreie Punktwolke wird durch Rasterung in ein Tiefenbild gewandelt und mit folgender Formel in Lichtlaufzeitrohdaten berechnet: $$r{e}_d=A_d\text{sin}\left(\frac{2\pi d_d}{ur}\right)$$
   
   $$i{m}_d=A_d\text{cos}\left(\frac{2\pi d_d}{ur}\right)$$
   

As in 2 shown, the results of the scan and a raw time-of-flight datum are registered and processed with image processing algorithms. A database with training data pairs consisting of error-free raw data from the laser scanner and error-prone raw data from the time-of-flight camera is created. Since the laser scanner does not measure raw data, but directly determines a point cloud, a raw time-of-flight datum is determined from the point cloud in 2 steps:

1. 1. An extrinsic calibration of the time-of-flight camera with the laser scanner provides an error-free point cloud from the perspective of the time-of-flight camera (measured by the laser scan ner) and a raw data from the measurement of the time-of-flight camera.
2. 2. The resulting error-free point cloud is converted into a depth image by rasterization and calculated in raw light time data with the following formula: $$\mathrm{right}{e}_{\mathrm{i.e}}=A_{\mathrm{i.e}}\text{sin}\left(\frac{2\pi {\mathrm{i.e}}_{\mathrm{i.e}}}{\mathrm{and}\mathrm{right}}\right)$$
   
   $$i{m}_{\mathrm{i.e}}=A_{\mathrm{i.e}}\text{cos}\left(\frac{2\pi {\mathrm{i.e}}_{\mathrm{i.e}}}{\mathrm{and}\mathrm{right}}\right)$$
   

The error-free raw data is defined here as a real and imaginary part (red and imd). The matching (erroneous) raw data from the time-of-flight camera are referred to as re and im and thus form the training data pair.

The training data pairs are then used to train a machine learning model. This model takes raw data as input and outputs corrected raw data.

The basic idea of the data generation is to offer the system the largest possible number of different scenarios with different problems in the scene measurement. For example, some scenarios may have corners and edges that favor multipath propagation.

The pairs of training data obtained from preferably a few thousand different scenarios are stored in a database.

This database can be used universally for the respective time-of-flight camera type and can be used to train a model. learning to improve raw data.

1. a) Montage der Lichtlaufzeitkamera auf einem Motor pro Rotationsachse. Eine Software ändert dann automatisch die Blickwinkel der Kamera durch Drehen der Motoren und nimmt so mehrere einzigartige Bilder pro Scan auf.
2. b) Verwendung von mehreren Lichtlaufzeitkameras in verschiedenen Positionen und Winkeln, die auf dem Laserscanner montiert sind. Dies hat den zusätzlichen Vorteil, dass verschiedene Kameras mit unterschiedlichen Kalibrierfehlern verwendet werden können. Dadurch wird die statistische Wirksamkeit der Informationserweiterung (augmentation) erhöht.
3. c) Aufnahme der Lichtlaufzeitbilder bei verschiedenen Belichtungszeiten, um verschiedene Signal-Rausch-Verhältnisse darzustellen (z.B. 5 verschiedene Stufen).

A major disadvantage of experimental training data is the high time required to generate a reasonable set of training data pairs. From this point of view, data proliferation is particularly important: This means that the amount of training data pairs per scan should be as large as possible. Preferably, several tens of thousands of training data pairs should be created. Therefore, each scan should deliver >=10 data pairs in order to build up a comprehensive database for training with a reasonable amount of time. Several methods are suitable to achieve this:

1. a) Mounting the time-of-flight camera on one motor per axis of rotation. Software then automatically changes the camera's viewing angles by rotating the motors, capturing multiple unique images per scan.
2. b) Use of multiple time-of-flight cameras in different positions and angles mounted on the laser scanner. This has the additional advantage that different cameras with different calibration errors can be used. This increases the statistical effectiveness of the information augmentation.
3. c) Recording of the time-of-flight images at different exposure times to show different signal-to-noise ratios (eg 5 different levels).

All methods can be combined to achieve the best possible data amplification. For a laser scanner with a field of view (fov) of 300°x360° and a time-of-flight camera with a 45°x60° fov, we can get -36 unique viewpoints. With 5 exposure times we get 5x36=180 training patterns per scan, so that with 100 measurement processes a database of 100x180 = 18k training pairs is created.

As in 2 shown, the machine learning model obtains training pairs from the database. Training on experimental data is a crucial part of the invention. This method leads to a better generalization of the machine learning model since no approximations or assumptions (e.g. only Lambertian scattering, pure Gaussian noise) have to be made to calculate synthetic ToF images. Therefore, training on experimental data leads to significantly better trained models than on synthetic data (regardless of the specific model architecture).

A time-of-flight camera measures the distance from the camera to the object in a scene on each pixel. However, the measurement is falsified by several reflections in the scene. This error is called multipath interference error (MPI error). Therefore, in the invention, a machine learning model is trained to estimate the MPI error in the time-of-flight camera measurement. Potential applications are: flagging the defective pixels, correcting the defective pixels, or using this model as a cost function in other training routines.

Multipath interference errors are caused by multiple reflections in the scene. As a result, the light path is longer than the direct path and thus increases the distance on the pixel. Furthermore, this error is highly scene dependent since it depends on the geometry in the scene. The proportions of constructive and destructive inter ference of the indirect paths depend essentially on the modulation frequency, which leads to a strong frequency dependence of this error.

A time-of-flight measurement M can be divided into parts from direct and indirect paths: $$M=M_{\mathrm{i.e}}+M_i$$

The direct path M _d denotes the measurement without MPI error.

If one knows M _d and M from an MPI-free measurement, the true MPI error results in $$M_i=M-M_{\mathrm{i.e}}$$

The measurement M _{d is} determined using a device without MPI errors (laser scanner) and M is determined using particularly low-noise measurements of a time-of-flight camera.

As in 3 shown, time-of-flight data M are passed to the machine learning model in the training. Using an appropriate cost function, one compares the prediction M _i,v with the true MPI error M _i,w . The L1 cost function would be suitable as an example: $${\mathrm{<figure-callout\; id="1"\; label="K\; L"\; filenames="DE102021128537A1\_0010.png"\; state="\{\{state\}\}">K</figure-callout>}}_L=\left|M_{i,v}-M_{i,w}\right|$$

Of course, other cost or optimization functions can also be used.

Neural convolution networks are particularly suitable for the machine learning model.

Based on the cost function, the weights and biases of the machine learning model are then adjusted using a suitable optimization method (e.g. Adam) in the error feedback method.

At the end of the training process, you have a trained model that estimates the MPI error with a high level of accuracy.

As in 4 shown, the MPI error can be determined for each pixel. Time of flight data can be raw data, corrected raw data (based on a calibration). Multi-frequency processes or coded modulation are also possible.

Furthermore, the model trained in this way can advantageously in turn be used as a cost function for training further models of machine learning. In particular, it can also be provided that models with different degrees of training are used for the training of further models.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 19704496 C2 [0002]
• EP 3832351 A1 [0004]

Claims (7)

Method for operating a time-of-flight camera that includes a time-of-flight sensor with time-of-flight pixels, comprising the steps: - Acquisition of raw data using the time-of-flight sensor, - correction of the raw data, by feeding the raw data of the time-of-flight sensor to a machine learning model trained with regard to multipath propagation and using an inference of the machine learning model to correct the raw data, where the machine learning model was trained with experimental multipath error-prone training data pairs, - Provision of the corrected raw data for a subsequent determination of distance values. procedure after claim 1 , in which distance values are determined from the corrected raw data without using a machine learning model. Method according to one of the preceding claims, in which the machine learning model is in the form of a neural convolution network. Time-of-flight camera designed to carry out one of the aforementioned methods; with at least one processor comprising a machine learning model. Use of the machine learning model according to one of the methods according to claim 1 until 3 was trained, as a cost function for training other machine learning models. Measurement setup for generating experimental training data pairs with a reference system and a time-of-flight camera, whereby the training data pairs are generated by combining high-precision and point-by-point sampled distance measurements of the reference system with measurements from a time-of-flight camera. measurement setup claim 6 , in which data is multiplied by automatically changing the parameters of the measurement setup.

Images