DE102021109386A1 - Method for correcting depth images of a time-of-flight camera - Google Patents

Abstract

1. A method for correcting depth images of a time-of-flight camera that determines a phase shift between an emitted and received modulated light,
whereby the light is emitted with different modulation frequencies in at least two phase measurements,
in which an unsupervised overall consistency cost function is optimized in an unsupervised training phase of a machine learning model. Whereby, when the time-of-flight camera is in operation, a depth image is determined using the trained machine learning model.

Description

The time-of-flight camera or time-of-flight camera system is intended to include, in particular, systems that gain distances from the phase shift of an emitted and received radiation. PMD cameras with photonic mixer detectors (PMD) are particularly suitable as time-of-flight or ToF cameras, as they are, inter alia, in FIG DE 197 04 496 A1 are described.

From the US 2017/0262768 A1 a TOF system is known which determines distances from a phase and amplitude of light emitted and reflected and received by a scene. A trained 'machine learning' system is used to determine the distance, which determines distance values from the raw data of the sensor. The system is trained on the basis of simulated raw TOF data and simulated 'multi path interference' data.

the DE 11 2011 104 644 T5 deals with a mobile robot, in particular a driverless transport system, in which the environment is captured with a camera and streamed on a cloud host. To recognize dangerous situations, a monitored learning algorithm is provided, which is trained on the basis of real images of the surroundings, so that after the training is completed, a danger-image-class model is available that can be transferred to a complete fleet of mobile robots if necessary. The classifier provided for this purpose is a machine learning algorithm which typically uses an iterative training algorithm, for example to optimize a cost function.

The object of the invention is to improve the distance measurement of a time-of-flight camera system.

The object is achieved in an advantageous manner by the method according to the invention for correcting depth images of a time-of-flight camera.

• - Optimierung einer unüberwachten Gesamt-Konsistenz-Kostenfunktion durch,
  1. a) Ermittlung einer unüberwachten trigonometrischen Konsistenz-Kostenfunktion durch Herstellung einer Konsistenz der Rohdaten hinsichtlich trigonometrischen Eigenschaften (g₁ K_trig),
  2. b) Ermittlung einer unüberwachten Distanz-Konsistenz-Kostenfunktion durch Herstellung einer Konsistenz der Rohdaten hinsichtlich Distanzen (g₂ K_d) zu wenigstens zwei unterschiedlichen Modulationsfrequenzen,
  3. c) Einmalige Bildung einer Maske ausgehend von den in Schritt a) und b) ermittelten unüberwachten trigonometrischen und Distanz-Konsistenz-Kostenfunktion, um bereits konsistente Bildbereiche zu identifizieren,
  4. d) Ermittlung einer unüberwachten Abweichungs-Konsistenz-Kostenfunktion (g₃ K_abs) ausgehend von der Maske gemäß Schritt c)
  5. e) Optimierung von Gewichten und Vorspannungen eines Modells maschinellen Lernens anhand der aus den Schritten a) bis d) gebildeten unüberwachten Gesamt-Konsistenz-Kostenfunktionen,

wobei im Betrieb der Lichtlaufzeitkamera ein Tiefenbild anhand des trainierten Modells maschinellen Lernens ermittelt wird (Inferenz).According to the invention, a method is provided for correcting depth images of a time-of-flight camera that determines a phase shift between an emitted and received modulated light,
whereby the light is emitted with different modulation frequencies in at least two phase measurements,
in which the following steps are carried out in an unsupervised training phase:

• - Optimization of an unsupervised overall consistency cost function through,
  1. a) Determination of an unsupervised trigonometric consistency-cost function by establishing a consistency of the raw data with regard to trigonometric properties (g ₁ K _trig ),
  2. b) Determination of an unsupervised distance-consistency-cost function by establishing a consistency of the raw data with regard to distances (g ₂ K _d ) for at least two different modulation frequencies,
  3. c) One-time creation of a mask based on the unsupervised trigonometric and distance-consistency-cost functions determined in steps a) and b) in order to identify already consistent image areas,
  4. d) Determination of an unsupervised deviation-consistency-cost function (g ₃ K _abs ) based on the mask according to step c)
  5. e) Optimization of weights and biases of a machine learning model on the basis of the unsupervised overall consistency cost functions formed from steps a) to d),

wherein during operation of the time-of-flight camera, a depth image is determined on the basis of the trained machine learning model (inference).

the 1 shows schematically the procedure according to the invention. In the current state of the art, the raw images from a ToF camera are calculated using classic algorithms to create depth maps. These depth maps contain artifacts such as noise or multipath interference / Multi Path Interference MPI. Instead, the invention uses a machine learning model to correct artifacts. The model is trained unsupervised on the basis of training data without a true depth map and thus generates error-free depth maps. The generation of a large amount of data from true depth maps (“ground truth”) means enormous effort. For example, through computationally intensive simulations of the data or the addition of a high-precision measuring device with a long recording time. Both methods do not give perfect trues Depth maps, but again contain their own artifacts. This includes, for example, the simplification of calculations in the simulation (domain gap between simulated and experimental data) or different perspectives from the high-precision measuring device and the time-of-flight camera (areas are covered or missing).

This enormous technical problem is circumvented by the invention in that it is no longer necessary to generate true depth maps in sufficient quantities for training machine learning models.

In contrast to monitored machine learning, unsupervised machine learning does not require any true annotations (in this case: no true depth map). Therefore, the data acquisition is significantly simplified.

The raw images of a ToF camera are measurements of phase-shifted periodic functions. The phase is proportional to the depth d and can be determined for each image point (pixel). The raw data are usually sinusoidal signals, which are referred to as real (Re) and imaginary (Im) parts. This results in the phase: $$\varphi =\text{atan2}\left(-\mathrm{R.}e,\mathrm{I.}m\right)$$

The depth is thus: $$d=\frac{ur}{2\pi }\varphi$$

The uniqueness range (ur) of a ToF camera is determined by the modulation frequency (f). $$ur=\frac{c}{2f}$$

The speed of light is denoted by c.

1. a) Zur Rauschkorrektur werden häufig räumliche Filter (Bilateral, Anisotrope Diffusion, ...) und/oder zeitliche Filter (Kalman, ...) angewendet.
2. b) Mehrwegeinterferenzfehler werden durch mehrfache Reflektionen in der Szene hervorgerufen. Dadurch ist der Lichtpfad länger als der direkte Weg und erhöht somit die Distanz auf dem Pixel. Weiterhin ist dieser Fehler in höchstem Maß szenenabhängig. Zum derzeitigen Stand ist kein klassischer Algorithmus bekannt welcher diesen Fehler in Echtzeit korrigiert.
3. c) Bewegungsartefakte entstehen dadurch, dass Real- und Imaginärteil nacheinander aufgenommen werden und daher zeitversetzt zueinander sind. Bei schnellen Bewegungen von Objekten in der Szene oder der Kamera treten daher Distanzfehler auf.
4. d) Phasenzuordnungsfehler treten bei der Zuordnung von Phasen mehrerer Frequenzen auf. Wird ein Pixel einer falschen Phase zugeordnet, so treten große Abweichungen in etwa von einem Eindeutigkeitsbereich ur auf. Üblicherweise werden Phasen mittels des chinesischen Restwerttheorems zugeordnet, bei hohem Rauschen gibt es allerdings zunehmend Zuordnungsfehler. Des Weiteren ist die maximale Reichweite der Kamera durch das kleinste gemeinsame Vielfache der Eindeutigkeitsbereiche der einzelnen Frequenzen begrenzt.

The depth data is falsified by several measurement errors. Particularly noteworthy is the correction of noise errors, multipath interference errors, movement artifacts and phase assignment errors.

1. a) Spatial filters (bilateral, anisotropic diffusion, ...) and / or temporal filters (Kalman, ...) are often used for noise correction.
2. b) Multipath interference errors are caused by multiple reflections in the scene. As a result, the light path is longer than the direct path and thus increases the distance on the pixel. Furthermore, this error is highly dependent on the scene. At the present time, there is no known conventional algorithm that corrects this error in real time.
3. c) Movement artifacts arise from the fact that the real and imaginary parts are recorded one after the other and are therefore time-shifted to one another. With fast movements of objects in the scene or the camera, distance errors therefore occur.
4. d) Phase assignment errors occur when phases are assigned to several frequencies. If a pixel is assigned to an incorrect phase, large deviations occur approximately from a uniqueness range ur. Usually phases are assigned using the Chinese residual value theorem, but with high noise there are increasing assignment errors. Furthermore, the maximum range of the camera is limited by the smallest common multiple of the uniqueness ranges of the individual frequencies.

Models based on machine learning can correct these errors in real time. The greatest challenge, however, is to obtain precise training data for training the models. This is made much easier by unsupervised learning.

• • Im Trainingsvorgang werden spezielle unüberwachte Kostenfunktionen optimiert, welche Konsistenzen in den Tiefendaten und Rohdaten sowie korrekte Tiefenwerte bei mindestens zwei Modulationsfrequenzen sicherstellen.
• • Eine Konsistenz ist die Erhaltung trigonometrischer Eigenschaften (Additionstheoreme) der Rohdaten.
• • Eine weitere Konsistenz besteht darin, dass Distanzen bei Verwendung unterschiedlicher Modulationsfrequenzen gleichbleiben müssen.
• • Tiefenwerte der klassischen Tiefenberechnung, welche bereits konsistent sind (Weil dort z.B. keine Artefakte auftreten) werden ebenfalls genutzt, um zu große Abweichungen des Modells von der gemessenen Tiefe zu verhindern. Dabei wird eine Maske von bereits konsistenten Pixeln anhand eines Schwellwertes der Konsistenz-Kostenfunktionen einmalig für jeden Datensatz bestimmt. Diese Maske beinhaltet also Bildpunkte, die ein gutes Signal zu Rausch Verhältnis haben und auch keine MPI Artefakte enthalten, da sonst die trigonometrische Konsistenz und die Distanz Konsistenz verletzt werden würden. Da allein die Konsistenzen noch nicht zu absolut korrekten Distanzwerten führen, werden im Training die absoluten Abweichungen der Distanzen auf den maskierten Bildpunkten minimiert.
• • Die Kombination aus allen drei unüberwacht definierten Kostenfunktionen wird Gesamt-Konsistenz-Kostenfunktion genannt.

The invention is the unsupervised training process of a machine learning model with the following properties:

• • In the training process, special unsupervised cost functions are optimized, which ensure consistencies in the depth data and raw data as well as correct depth values for at least two modulation frequencies.
• • Consistency is the preservation of trigonometric properties (addition theorems) of the raw data.
• • Another consistency is that distances must remain the same when using different modulation frequencies.
• • Depth values of the classic depth calculation, which are already consistent (because, for example, no artifacts occur there) are also used to prevent the model from deviating too much from the measured depth. In this case, a mask of already consistent pixels is determined once for each data record on the basis of a threshold value of the consistency cost functions. This mask therefore contains pixels that have a good signal-to-noise ratio and also do not contain any MPI artifacts, since otherwise the trigonometric consistency and the distance consistency would be violated. Since the consistencies alone do not lead to absolutely correct distance values, the absolute deviations of the distances on the masked image points are minimized during training.
• • The combination of all three unsupervised defined cost functions is called the overall consistency cost function.

During training, the quality of the error-corrected depth map t is calculated using the differentiable and unsupervised total consistency cost functions and the weights and biases of the machine learning model are adjusted in the error feedback process.

Here, an unsupervised overall consistency cost function is determined from a summary of an unsupervised trigonometric, depth and deviation cost function.

The unsupervised trigonometric consistency cost function applies to at least one frequency pair f, f̃ and is calculated for all pixels j = 1, ..., N: $$\begin{array}{l}{K}_{triG}=\frac{1}{N}{\displaystyle \sum _{f\ne \tilde{f}}{\displaystyle \sum _j\left|\mathrm{A.}dd\mathrm{R.}e{\left(\mathrm{R.}{e}_{f,j},\mathrm{I.}{m}_{f,j}\right)}^n-\mathrm{A.}dd\mathrm{R.}e{\left(\mathrm{R.}{e}_{\tilde{f},j},\mathrm{I.}{m}_{\tilde{f},j}\right)}^m\right|}}\\ \text{+}\left|\mathrm{A.}dd\mathrm{I.}m{\left(\mathrm{R.}{e}_{f,j},\mathrm{I.}{m}_{f,j}\right)}^n-\mathrm{A.}dd\mathrm{I.}m{\left(\mathrm{R.}{e}_{\tilde{f},j},\mathrm{I.}{m}_{\tilde{f},j}\right)}^m\right|\end{array}$$

$$AddIm\left(Re,Im\right)=I{m}^2-R{e}^2$$

AddRe and Addlm denote the following addition theorems: $$\mathrm{A.}dd\mathrm{R.}e\left(\mathrm{R.}e,\mathrm{I.}m\right)=2\mathrm{I.}m\mathrm{R.}e$$

$$\mathrm{A.}dd\mathrm{I.}m\left(\mathrm{R.}e,\mathrm{I.}m\right)=\mathrm{I.}{m}^2-\mathrm{R.}{e}^2$$

The additions are carried out n times (or m times) so that the following relationship must apply to the frequency pair f, f̃: $$nf-m\tilde{f}=0$$

The unsupervised depth consistency cost function for at least two frequencies is: $$K_d=\frac{\mathrm{I.}}{N}{\displaystyle \sum _{f\ne \tilde{f}}{\displaystyle \sum _j\left|t_{f,j}-t_{\tilde{f},j}\right|}}$$

On the basis of the unmonitored trigonometric and depth-consistency cost function, a mask M and with it a deviation-consistency cost function can be determined. The trigonometric consistency and the depth consistency are therefore calculated once for each pixel j for each measurement by the time-of-flight camera. A threshold value S can be used to determine those pixels which are already free of errors in the unprocessed image. $${\mathrm{M.}}_j=\left(K_{triG,j}+K_{d,j}\right)\le \mathrm{S.}$$

The collection of the error-free pixels M _j forms the consistent mask M.

The pixel-by-pixel absolute deviation on a consistent mask M is referred to as the deviation cost function and ensures correct depth values: $$K_{abs}=\frac{\mathrm{I.}}{N_{\mathrm{M.}}}{\displaystyle \sum _i\left|T_i-t_i\right|}$$

T _i denotes the original depth values (which can be assumed to be error-free) on the mask M for each masked pixel i = 1,..., N _m . This means that this cost function is also defined in an unsupervised manner.

Overall, an unsupervised overall consistency cost function with weights g ₁ and g _{2 is} minimized: $$K_{Ges}=K_{abs}+G_1{K}_{triG}+G_2{K}_d$$

The training routine is thus completely unsupervised: a true depth map is not required for any of the cost functions.

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• DE 19704496 A1 [0001]
• US 2017/0262768 A1 [0002]
• DE 112011104644 T5 [0003]

Claims (8)

Method for correcting depth images of a time-of-flight camera that determines a phase shift between an emitted and received modulated light, with the light being emitted with different modulation frequencies in at least two phase measurements, in which the following steps are carried out in an unsupervised training phase without a true depth map: - Optimization of a unsupervised overall consistency cost function by, a) determining an unsupervised trigonometric consistency cost function by establishing a consistency of the raw data with regard to trigonometric properties (g ₁ K _trig ), b) determining an unsupervised distance consistency cost function by establishing a consistency of the raw data with regard to Distances (g ₂ K _d ) to at least two different modulation frequencies, c) One-time formation of a mask based on the unsupervised trigonometric and distance-consistency cos determined in step a) and b) ten function to identify already consistent image areas, d) determination of an unsupervised deviation-consistency-cost function (g ₃ K _abs ) based on the mask according to step c) e) optimization of weights and biases of a machine learning model based on the steps a ) to d) formed, unsupervised overall consistency cost function, with a depth image being determined on the basis of the trained machine learning model during operation of the time-of-flight camera. Procedure according to Claim 1 , in which the quality of the error-corrected depth map is calculated using a differentiable cost function and the weights and biases of the machine learning model are adjusted in the error feedback method. Method according to one of the preceding claims, in which the unsupervised trigonometric consistency cost function for at least one frequency pair f, f̃ and is calculated for all pixels j = 1, ..., N from: $$\begin{array}{l}{K}_{triG}=\frac{1}{N}{\displaystyle \sum _{f\ne \tilde{f}}{\displaystyle \sum _j\left|\mathrm{A.}dd\mathrm{R.}e{\left(\mathrm{R.}{e}_{f,j},\mathrm{I.}{m}_{f,j}\right)}^n-\mathrm{A.}dd\mathrm{R.}e{\left(\mathrm{R.}{e}_{\tilde{f},j},\mathrm{I.}{m}_{\tilde{f},j}\right)}^m\right|}}\\ \text{+}\left|\mathrm{A.}dd\mathrm{I.}m{\left(\mathrm{R.}{e}_{f,j},\mathrm{I.}{m}_{f,j}\right)}^n-\mathrm{A.}dd\mathrm{I.}m{\left(\mathrm{R.}{e}_{\tilde{f},j},\mathrm{I.}{m}_{\tilde{f},j}\right)}^m\right|\end{array}$$

Method according to one of the preceding claims, in which the depth consistency cost function for the at least two frequencies of the function $$K_d=\frac{1}{N}{\displaystyle \sum _{f\ne \tilde{f}}{\displaystyle \sum _j\left|t_{f,j}-t_{\tilde{f},j}\right|}}$$

enough. Method according to one of the preceding claims, in which, proceeding from the unsupervised trigonometric and depth consistency cost function, a mask M and, with the mask M, a deviation consistency cost function under the specification $$K_{abs}=\frac{1}{N_{\mathrm{M.}}}{\displaystyle \sum _i\left|T_i-t_i\right|}$$

is determined. Method according to one of the preceding claims, in which a mask of consistent image points M using a threshold value S based on the cost functions in Claims 3 and 4th is calculated. $${\mathrm{M.}}_j=\left(K_{triG,j}+K_{d,j}\right)\le \mathrm{S.}$$

Method according to one of the preceding claims, in which a mask of consistent image points M using a threshold value S based on the cost functions in Claim 3 is calculated. $${\mathrm{M.}}_j=K_{triG,j}\le \mathrm{S.}$$

Method according to one of the preceding claims, in which a mask of consistent image points M using a threshold value S based on the cost functions in Claim 4 is calculated. $${\mathrm{M.}}_j=K_{d,j}\le \mathrm{S.}$$

Images