DE102019124141A1 - Time-of-flight camera system - Google Patents

Abstract

Time-of-flight camera system with lighting from several modulatable individual emitters, whereby the emitters can be switched on individually or in groups

Description

The invention relates to a time-of-flight camera system and a method for operating such a system according to the preamble of the independent claims.

The time-of-flight camera system should in particular include systems that determine distances directly from the time of flight or obtain the time of flight from the phase shift of an emitted and received radiation. PMD cameras with photonic mixer detectors (PMD) are particularly suitable as the time of flight or 3D cameras, as they are, inter alia, in FIG DE 197 04 496 A1 and can be obtained, for example, from the company 'ifm electronic GmbH' or 'pmdtechnologies ag' as an O3D camera or as a CamBoard. The PMD camera allows, in particular, a flexible arrangement of the light source and the detector, which can be arranged both in a housing and separately.

From the DE 197 04 496 A1 the determination of a distance or a corresponding phase shift of the light reflected from an object is also known. In particular, it is disclosed to shift the transmitter modulation in a targeted manner by 90 °, 180 ° or 270 ° in order to determine a phase shift and thus a distance from these four phase measurements via an arctan function.

The object of the invention is to improve the distance measurement of a time-of-flight camera system.

The object is achieved in an advantageous manner by the time-of-flight camera system and method according to the preamble of the independent claims.

The invention is explained in more detail below on the basis of exemplary embodiments with reference to the drawings.

• 1 schematisch ein Lichtlaufzeitkamerasystem,
• 2 eine modulierte Integration erzeugter Ladungsträger,
• 3 einen Querschnitt durch einen PMD-Lichtlaufzeitsensor mit Potentialverteilung,
• 4 einen zeitlichen Verlauf der Integrationsspannungen an einem Lichtlaufzeitpixel,
• 5 Verläufe der Ladungsintegration abhängig von der Phasenverschiebung und -lage,
• 6 eine Relation der Phasenverschiebung in einem IQ-Diagramm,
• 7 einen Modulationsverlauf über vier Phasenlagen,
• 8 ein erfindungsgemäßes Punktemuster.

Show it:

• 1 schematically a time-of-flight camera system,
• 2 a modulated integration of generated charge carriers,
• 3 a cross section through a PMD time-of-flight sensor with potential distribution,
• 4th a temporal course of the integration voltages at a light transit time pixel,
• 5 Course of the charge integration depending on the phase shift and position,
• 6th a relation of the phase shift in an IQ diagram,
• 7th a modulation curve over four phase positions,
• 8th a pattern of dots according to the invention.

In the following description of the preferred embodiments, the same reference symbols designate the same or comparable components.

1 shows a measurement situation for an optical distance measurement with a time-of-flight camera, as it is, for example, from FIG DE 197 04 496 A1 is known.

The time-of-flight camera system 1 comprises a transmission unit or a lighting module 10 with a lighting 12th and associated beam-shaping optics 15 and a receiving unit or time-of-flight camera 20th with a receiving optics 25 and a time of flight sensor 22nd .

The time of flight sensor 22nd has at least one transit time pixel, preferably also a pixel array, and is designed in particular as a PMD sensor. The receiving optics 25 typically consist of several optical elements in order to improve the imaging properties. The beam-shaping optics 15 of the transmission unit 10 can for example be designed as a reflector or lens optics. In a very simple embodiment, it is also possible, if necessary, to dispense with optical elements on both the receiving and transmitting sides.

The measuring principle of this arrangement is essentially based on the fact that, based on the phase shift of the emitted and received light, the transit time and thus the distance covered by the received light can be determined. For this purpose, the light source 12th and the time of flight sensor 22nd via a modulator 30th together with a specific modulation signal M _o with a basic phase position φ ₀ applied. In the example shown, there is also between the modulator 30th and the light source 12th a phase shifter 35 provided with the basic phase φ ₀ of the modulation signal Mon the light source 12th around defined phase positions φ _var can be moved. For typical phase measurements, phase positions of φ _var = 0 °, 90 °, 180 °, 270 ° are preferably used.

The light source sends according to the set modulation signal 12th an intensity modulated signal S _p1 with the first phase position p1 or p1 = φ ₀ + φ _var . This signal S _p1 or the electromagnetic radiation is in the illustrated case from an object 40 reflects and hits accordingly out of phase due to the distance covered Δφ (t _L ) with a second phase position p2 = φ ₀ + (φ _var + Δ ( _φ (t _L )) as the received signal S _p2 on the Time of flight sensor 22nd . In the time of flight sensor 22nd the modulation signal M _o becomes with the received signal S _p2 mixed, the phase shift or the object distance from the resulting signal d is determined.

The system also has a modulation control device 27 that, depending on the measurement task at hand, the phase position ( φ _var the modulation signal Mon changed and / or via a frequency oscillator 38 sets the modulation frequency.

As an illumination source or light source 12th Infrared light-emitting diodes are preferably suitable. Of course, other radiation sources in other frequency ranges are also conceivable, in particular light sources in the visible frequency range are also possible.

The basic principle of phase measurement is schematically shown in 2 shown. The upper curve shows the time course of the modulation signal M ₀ with which the lighting 12th and the time of flight sensor 22nd can be controlled. That from the object 40 reflected light hits as a received signal S _p2 phase-shifted according to its time of flight t _L Δφ (t _L ) on the time of flight sensor 22nd . The time of flight sensor 22nd collects the photonically generated charges q over several modulation periods in the phase position of the modulation signal Mon in a first accumulation gate Ga and in a phase position M ₀ + 180 ° shifted by 180 ° in a second accumulation gate Gb . From the ratio of those in the first and second gate Ga , Gb collected charges qa, qb can be the phase shift Δφ (t _L ) and thus a distance d of the object.

3 shows a cross section through a pixel of a photonic mixer such as that from FIG DE 197 04 496 C2 is known. The modulation photogates Gam, G0, Gbm form the light-sensitive area of a PMD pixel. In accordance with the voltage applied to the modulation gates Gam, G0, Gbm, the photonically generated charges q become either one or the other of the accumulation gate or integration node Ga , Gb steered. The integration nodes can be designed as a gate or also as a diode.

3b shows a potential curve in which the charges q in the direction of the first integration account Ga tile while the potential according to 3c the charge q in the direction of the second integration node Gb lets flow. The potentials are specified according to the applied modulation signals. Depending on the application, the modulation frequencies are preferably in a range from 1 to 100 MHz. A modulation frequency of 1 MHz, for example, results in a period of one microsecond, so that the modulation potential changes accordingly every 500 nanoseconds.

In 3a is also a readout unit 400 shown, which can possibly already be part of a PMD time-of-flight sensor designed as CMOS. The integration nodes designed as capacitors or diodes Ga , Gb integrate the photonically generated charges over a large number of modulation periods. In a known manner, they can then be sent to the gates Ga , Gb applied voltage, for example via the readout unit 400 can be tapped at high resistance. The integration times should preferably be selected so that the time-of-flight sensor or the integration nodes and / or the light-sensitive areas do not become saturated for the amount of light to be expected.

In 4th is a typical time course of the at the integration nodes Ga , Gb voltage present during a phase measurement U _a , U _b . _{Starting from a positive reset voltage U DRS} applied to the integration node after a reset, the voltage drops due to the accumulated photoelectrons at both integration nodes Ga , Gb from. According to the phase shift Δφ (t _L ) of the received signal, the voltages at the integration node drop Ga , Gb to different degrees. At the end of the integration time t _int , the at the integration node Ga , Gb applied voltage U _a , U _b read out. The voltage difference ΔU of the two tensions U _a , U _b corresponds to the difference in a known manner Δq the one at the integration node Ga , Gb accumulated charge q. The integration time t _int is preferably dimensioned so that there is no integration node Ga , Gb reached its saturation potential Us in a conventional exposure. A so-called SBI circuit for signal compensation can also be provided for greater signal strengths. Such circuits are for example from the DE 10 2004 016 626 A1 or DE 10 2005 056 774 A1 known.

5a and 5b show curves of the normalized charge difference Δq = q _a - q _b / (q _a + q _b ) as a function of the phase shift Δφ (t _L ) of the received light signal S _p2 with different phase positions. The 5a shows a curve for an unshifted modulation phase Mon with a phase position (φ _var = 0 °.

When the signal hits S _p2 without phase shift so Δφ (t _L ) = 0 °, for example when the transmission signal S _p1 is steered directly onto the sensor, the phases of the modulation are M ₀ and from the received signal S _p2 identical, so that all charge carriers generated are phase-synchronized at the first integration node Ga are detected and thus a maximum difference signal with Δq = 1 is present.

As the phase shift increases, the one at the first integration node increases Ga accumulated charge from and at the second integration node Gb to. With a phase shift of Δφ (t _L ) = 90 °, the charge carriers qa, qb are at both integration nodes Ga , Gb evenly distributed and the charge difference thus zero and after 180 ° phase shift "-1". As the phase shift increases, the charge on the first gate increases Ga closes again, so that as a result the charge difference increases again in order to then reach a maximum again at 360 ° or 0 °.

Mathematically, this is a correlation function of the received signal S _p2 with the modulating signal M ₀ . $$q\left(\tau \right)={\displaystyle {\int }_0{\mathrm{S.}}_{p2}\left(t-\tau \right){\mathrm{M.}}_0\left(t\right)dt}$$

In the case of modulation with a square-wave signal, a triangular function results as the correlation function. In the case of modulation with a sine signal, for example, the result would be a cosine function.

How 5a shows, a measurement of the phase with a phase position is only _{unambiguous up to a phase shift Δφ (t L} ) ≤ 180 °.

The IQ (in-phase quadrature) method is known for maximum detection of the phase shift, in which two measurements are carried out with phase positions shifted by 90 °, for example with the phase position φ _var = 0 ° and (φ _var = 90 ° a measurement with the phase position (φ _var = 90 ° is in 5b shown.

The relationship between these two curves can be shown in a known manner, for example, for sinusoidal curves in an IQ diagram according to FIG. 6th represent. As a first approximation, this representation can also easily be used for the triangular functions shown.

The phase angle can then be determined in a known way using an arctan function or arctan2 function: $$\phi =arctan\frac{\mathrm{\Delta }q\left({90}^{\circ }\right)}{\mathrm{\Delta }q\left(0^{\circ }\right)}$$

Due to the linear relationship between charge and voltage, the phase angle can also be determined using the voltage differences: $$\phi =arctan\frac{\mathrm{\Delta }U\left({90}^{\circ }\right)}{\mathrm{\Delta }U\left(0^{\circ }\right)}$$

In order to compensate for asymmetries of the sensor, for example, additional phase measurements shifted by 180 ° can be carried out so that the phase angle can be determined as follows. $$\phi =\mathrm{\Delta }\phi \left(t_{\mathrm{L.}}\right)=arctan\frac{\mathrm{\Delta }q\left({90}^{\circ }\right)-\mathrm{\Delta }q\left({270}^{\circ }\right)}{\mathrm{\Delta }q\left(0^{\circ }\right)-\mathrm{\Delta }q\left({180}^{\circ }\right)}$$

Wobei die Indizes die jeweilige Phasenlage der Differenzen a_i andeuten, mit $$a_1=\mathrm{\Delta }q\left(0^{\circ }\right)\text{ usw}.$$

Or in short: $$\phi =\mathrm{\Delta }\phi \left(t_{\mathrm{L.}}\right)=arctan\frac{a_{2-}{a}_{\mathrm{4th}}}{a_1-a_3}$$

Whereby the indices indicate the respective phase position of the differences a _i , with $$a_1=\mathrm{\Delta }q\left(0^{\circ }\right)\text{etc}.$$

From the phase shift φ or. Δφ (t _L ) can be used for object distances d , which are smaller than half the wavelength λ of the modulation frequency d ≤ λ / 2, determine a distance in a known manner. $$d=\mathrm{\Delta }\phi \left(t_{\mathrm{L.}}\right)\frac{\mathrm{<figure-callout\; id="2"\; label="\lambda "\; filenames="DE102019124141A1\_0012.png"\; state="\{\{state\}\}">\lambda </figure-callout>}}{2\pi }\cdot \frac{1}{2}$$

In 7th a complete set of a distance measurement with four phase positions of 0 °, 90 °, 180 ° and 270 ° is shown. In the case shown, charge carriers are integrated via modulation periods and a value a ₁ , a ₂ , a ₃ , a ₄ corresponding to the charge difference is read out in each phase position, from which, as already shown, a phase shift and a corresponding distance value can be determined.

In previous ToF systems, the 4-phase algorithm is used to determine the phase position within a range of unambiguity. With this algorithm, four support points are determined, which are equidistant in the uniqueness range from 0 to 2π with the same modulation frequency, and then the phase position of the fundamental wave, which runs through these support points, is calculated. The uniqueness area is converted from radians to a distance by a transformation factor.

Objects lead at distances that are in multiples of this uniqueness range (0 to 2π) to ambiguities of the phase position. These ambiguities can be converted into a clear phase position by a further phase measurement or by applying the 4 phase algorithm again with another, typically a lower, modulation frequency. This procedure therefore implies at least 8 measurements until a clear distance value can be calculated.

The course of the correlation function when using rectangular modulation signals results, as shown, in a triangular course of the correlation function. In 8th two correlations are shown for different frequencies. The function is drawn over the distance in order to show clearly the areas of uniqueness and their repetition. Due to the ratio of the selected modulation frequencies (30MHz / 15 MHz), the uniqueness of the maximum frequency is half as large as that of the lower frequency.

Currently, with many ToF cameras that work according to the phase ToF method (e.g. PMD cameras), scenes are completely illuminated, i.e. FoV (field of view) is almost identical to the Fol (field of illumination).

To compensate for lens vignetting, measuring distances, etc., lighting with increased intensities at the edge of the image is often used

• - Übersprechen von verschiedenen Bildteilen auf jedes Pixel bedingt durch Streulicht in der Empfangsoptik (Schmutz, Kratzer), Reflexen am Sensor oder an Linsen (Beschichtungen nicht Perfekt bzw. keine 100%ige Absorption im Sensor) etc.; d.h. die Abstandsinformationen eines nahen und eines fernen Objekts überlagert sich und kann sogar dominant werden, z.B. nahes weißes Objekt (oder sogar Retroreflektor) verursacht Streulicht auf Bildbereichen, welche in der Szene weit weg und/oder nur schwach reflektierend sind. Hier kann das Streusignal sogar größer sein, als das eigentliche Messsignal.
• - Übersprechen/Überlagerung von verschiedenen Beleuchtungsanteilen schon in der Szene, d.h. ein Bildpunkt wird in der Szene sowohl direkt von der Beleuchtung der ToF Kamera beleuchtet, als auch indirekt via eines Reflexes (z.B. direkt auf Wand vs. Erst auf Fensterscheibe und dann auf Wand)
• - Übersprechen/Überlagerung von Beleuchtungsanteilen aus der Messszene (rücklaufendes Licht) und direktem Licht aus dem Sender (hinlaufendes Licht), z.B. wenn Senderstrahlung in Empfangsweg hineingekoppelt wird, etwa bei gemeinsam genutztem Fenster für Sende und Empfangsweg. Hier kann Senderlicht in das Fenster eingekoppelt werden (Schmutz, Kratzer etc.) und aus demselben Grund vor dem Objektiv wieder ausgekoppelt werden
• - Begrenzte Messreichweite aufgrund von:
  • o limitierter Versorgungs- oder Kühlleistung der Beleuchtung
  • o Problemen mit Augen- oder Hautsicherheit, siehe Normen/Gesetze:
    • IEC60825-1:2007, IEC60825-1:2014, OStrV-Gesetz, RICHTLINIE 2006/25/EG DES EUROPÄISCHEN PARLAMENTS UND DES RATES vom 5. April 2006, IEC62471

The following problems arise with complete lighting, which can be reduced or solved with the new idea:

• - Crosstalk from different parts of the image on each pixel caused by stray light in the receiving optics (dirt, scratches), reflections on the sensor or on lenses (coatings not perfect or not 100% absorption in the sensor) etc .; ie the distance information of a near and a distant object is superimposed and can even become dominant, eg a near white object (or even retroreflector) causes scattered light on image areas which are far away in the scene and / or only weakly reflective. Here the scatter signal can even be larger than the actual measurement signal.
• - Crosstalk / superimposition of different lighting components already in the scene, i.e. a pixel in the scene is illuminated both directly by the lighting of the ToF camera and indirectly via a reflex (e.g. directly on the wall vs. first on the window pane and then on the wall)
• - Crosstalk / superimposition of lighting components from the measurement scene (returning light) and direct light from the transmitter (incoming light), e.g. when transmitter radiation is coupled into the receiving path, e.g. when a window is shared for the sending and receiving path. Here, transmitter light can be coupled into the window (dirt, scratches, etc.) and, for the same reason, coupled out again in front of the lens
• - Limited measuring range due to:
  • o Limited supply or cooling capacity of the lighting
  • o Problems with eye or skin safety, see standards / laws:
    • IEC60825-1: 2007, IEC60825-1: 2014, OStrV law, DIRECTIVE 2006/25 / EC OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of April 5, 2006, IEC62471

Illumination with point light patterns instead of a homogeneous light profile (light modulated in time so that ToF measurement is possible).

According to the invention, it is proposed to increase the resolution of a time-of-flight camera system by means of a switchable emitter or a scanning point light pattern.

The core idea of the invention is: the "sharpness" or the number of pixels is not specified by the sensor or the receiving optics themselves, but by the lighting in the form of light point arrays. In addition, the exact alignment of each light point can be determined by evaluating the signal intensity on neighboring pixels (subpixel scanning).

You can either use transmitters with more points of light as the number of receiver pixels, or transmitters with fewer points of light as the number of receiver pixels - but with a scanner unit.

For transmitters with more light points than the number of receiver pixels, the increase in resolution is achieved by switching the light points in groups, e.g. only every nth emitter per row and every nth column lights up. The final image can be put together by sequential recording by switching on the individual groups one after the other.

In the case of transmitters with fewer light points than the number of receiver pixels, but with a scanner / optics so that the point pattern can be aligned, the final image can again be put together by means of sequential recordings, in which one record is made for each scanner / lens position.

Number of effective measuring points = number of transmitter points x number of scanner positions.

Furthermore, strip or pattern lighting (e.g. dotted lighting line), which is moved in only 1 direction; then one can increase the resolution in one direction via subpixel resolution evaluation; The pattern is preferably moved perpendicular to the alignment of the light strip / light point pattern.

The light point pattern can also be generated from one or more sources and additional optics, such as holographic optical elements (HOE), diffractive optical elements (DOE), "engineered" diffusers, mirror arrays, micromirror arrays (DMD), etc.

• Ein ToF-Kamerasystem mit einer Beleuchtung aus vielen modulierten Einzelemittern, wobei die Emitter einzeln oder in Gruppen eingeschaltet werden können, , wobei die einzelnen Emitter monolithisch (z.B. VCSEL-Array) oder aus diskreten Einzelemittern oder aus einer Anordnung von Emitterarrays zusammengesetzt sind.

The following configurations are conceivable:

• A ToF camera system with lighting consisting of many modulated individual emitters, whereby the emitters can be switched on individually or in groups, whereby the individual emitters are monolithic (e.g. VCSEL array) or composed of discrete individual emitters or an arrangement of emitter arrays.

Furthermore, the exact alignment of the emission direction of the individual emitters is determined on the basis of the signal intensities of neighboring pixels.

It is also provided that the total number of individual emitters is greater than the number of receiver pixels.

It is preferably provided that the total number of individual emitters is less than the number of receiver pixels (down to one emitter) and a unit for aligning the light points (scanner) is used.

Furthermore, only some of the emitters are switched on

Furthermore, the final image can be built up from several individual images - in each of which other emitters are switched on and / or the light point alignment is realigned by means of a unit (scanner) - is put together.

Furthermore, the alignment of the individual light points can be determined on the basis of an intensity comparison of neighboring receiver pixels.

Furthermore, it is provided that the light point pattern is generated by means of one or more emitters, as well as additional optics, which increase the number of light points.

8th shows at the top left: exemplary emitter array, which can be switched into 9 groups. Only the emitters with the same number are activated at the same time

Right: lighting pattern in the scene if only the group of emitters is switched on with "1"

Bottom left: Position of the light points shown on a detector with 7x4 pixels; If the size of the depicted points of light (including the blurring of the lens) is larger than the pixel pitch, the position of the points of light in the scene can be determined by means of an intensity comparison using subpixel resolution. Although the receiver chip only has 7x4 pixels, a resolution of 9x6 can still be achieved through sequential recording.

List of reference symbols

1

Time-of-flight camera system

10

Lighting module

12th

lighting

20th

Receiver, time-of-flight camera

22nd

Time of flight sensor

27

Evaluation unit

30th

modulator

35

Phase shifter, lighting phase shifter

38

Modulation controller

40

object

400

Evaluation unit

φ, Δφ (t _L )

phase shift due to runtime

φ _var

Phasing

φ ₀

Base phase

Mon

Modulation signal

p1

first phase

p2

second phase

Sp1

Transmission signal with first phase

Sp2

Received signal with second phase

Ga, Gb

Integration node

Ua, ex

Tensions at the integration nodes

ΔU

Voltage difference

Δq

Charge difference

d

Object distance

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• DE 19704496 A1 [0002, 0003, 0009]
• DE 19704496 C2 [0017]
• DE 102004016626 A1 [0020]
• DE 102005056774 A1 [0020]

Claims (8)

Time-of-flight camera system with lighting from several modulatable individual emitters, whereby the emitters can be switched on individually or in groups. Time-of-flight camera system according to Claim 1 , in which the exact alignment of the emission direction of the individual emitters is determined based on the signal intensities of neighboring pixels. Time-of-flight camera system according to one of the preceding claims, in which the total number of individual emitters is greater than the number of receiver pixels. Time-of-flight camera system according to one of the preceding claims, in which the total number of individual emitters is less than the number of receiver pixels (down to one emitter) and a unit for aligning the light points (scanner) is used. Time-of-flight camera system according to one of the preceding claims, in which the emitters can be partially switched on Time-of-flight camera system according to one of the preceding claims, in which the final image is made up of several individual images, with different emitters being able to be switched on for each individual image. Time-of-flight camera system according to one of the preceding claims, in which an alignment of the individual light points can be determined on the basis of an intensity comparison of neighboring receiving pixels. Time-of-flight camera system according to one of the preceding claims, in which the light point pattern is generated by means of one or more emitters and additional optics which increase the number of light points.

Images