CN113945951B - Multipath interference suppression method in TOF (time of flight) depth calculation, TOF depth calculation method and device - Google Patents
Multipath interference suppression method in TOF (time of flight) depth calculation, TOF depth calculation method and device Download PDFInfo
- Publication number
- CN113945951B CN113945951B CN202111227054.8A CN202111227054A CN113945951B CN 113945951 B CN113945951 B CN 113945951B CN 202111227054 A CN202111227054 A CN 202111227054A CN 113945951 B CN113945951 B CN 113945951B
- Authority
- CN
- China
- Prior art keywords
- reflected light
- light signal
- indirect
- scene
- phase
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
- G01S17/894—3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/4802—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
Abstract
The invention discloses a multipath interference suppression method in TOF depth calculation, a TOF depth calculation method and a device, wherein the method comprises the following steps: receiving a first reflected light signal reflected by a scene collected by a TOF camera, wherein the first reflected light signal is obtained by carrying out dot-matrix illumination reflection on the scene by the TOF camera at a certain modulation frequency; performing phase calculation on the first reflected light signal to obtain the phase of the corresponding position of the dot-matrix illumination light spot; receiving a second reflected light signal reflected by a scene and acquired by a TOF camera, wherein the second reflected light signal is obtained by carrying out floodlighting reflection on the scene by the TOF camera at two different modulation frequencies; performing data reconstruction on the second reflected light signal, and representing the second reflected light signal as the sum of the direct component and the indirect component; resolving an indirect component from the second reflected light signal at a position corresponding to the dot-matrix illumination spot by using the phase; carrying out two-dimensional interpolation on the indirect components to obtain the indirect components of the whole scene; the indirect component of the entire scene is removed from the second reflected light signal.
Description
Technical Field
The invention relates to the technical field of 3D imaging of a time-of-flight camera, in particular to a multipath interference suppression method in TOF depth calculation, a TOF depth calculation method and a device.
Background
In recent years, three-dimensional imaging technology has become a popular field of research, and TOF (Time-of-Flight) depth imaging technology has also been rapidly developed. The TOF camera sends out infrared modulated light waves to a detected scene, and calculates the flight time of the light by receiving reflected light rays through a sensor, so that the depth information of the detected scene is obtained. At present, a continuous wave modulation method is mostly adopted, the flight time of light is recorded by detecting the phase shift of reflected light and emergent light, the working principle is simple, and the method is widely applied to various fields such as face recognition, human-computer interaction and the like.
Under ideal conditions, a floodlight beam emitted by a light source of the TOF camera returns to a photosensitive chip of the camera after being reflected once on the surface of a detected scene, and a depth image can be obtained to reflect the depth information of the scene through one-time floodlight illumination. However, in practical use, the light received by the light sensing chip not only includes the light directly reflected from the target surface of the scene, but also includes the reflected light from other surfaces, and the measured phase shift is calculated by the superposition of the directly reflected light and the reflected light from other paths, which is called multipath interference (MPI). The presence of multipath interference can cause large errors in depth measurements, which need to be suppressed.
At present, a plurality of schemes for restraining multipath interference of a TOF camera exist, including establishing a radiation model to estimate MPI and iteratively correcting a depth image, but the method is over theoretical, large in calculation amount and low in practicability; the method for separating multipath interference through transient imaging and sparse decomposition also leads to research of numerous scholars, but the method needs a large amount of modulation frequency and has high requirements on hardware; the rapid development of deep learning also provides a solution for the suppression of MPI, but the method requires a large amount of data for training, and the acquisition of training data and real data is difficult, so that the application of the method is limited. Therefore, the method has important significance for researching the TOF imaging multipath interference suppression method with simple hardware and simple and convenient calculation.
Disclosure of Invention
The embodiment of the application aims to provide a multipath interference suppression method in TOF depth calculation, a TOF depth calculation method and a device, which are used for separating a direct component of scene primary reflection from an indirect component caused by multipath interference so as to solve the technical problem that in the related technology, the depth measurement result has serious errors due to superposition of direct reflected light and other multiple path reflected light signals.
According to a first aspect of embodiments of the present application, there is provided a multipath interference suppression method in TOF depth resolution, including:
receiving a first reflected light signal reflected by a scene collected by a TOF camera, wherein the first reflected light signal is obtained by performing dot matrix illumination reflection on the scene by the TOF camera at a certain modulation frequency;
performing phase calculation on the first reflected light signal to obtain the phase of the corresponding position of the dot-matrix illumination light spot;
receiving a second reflected light signal reflected by a scene collected by a TOF camera, wherein the second reflected light signal is obtained by performing floodlight illumination reflection on the scene by the TOF camera at two different modulation frequencies;
performing data reconstruction on the second reflected light signal, representing the second reflected light signal as a sum of a direct component and an indirect component;
resolving an indirect component from the second reflected light signal at a position corresponding to the dot-matrix illumination spot using the phase;
carrying out two-dimensional interpolation on the indirect components to obtain indirect components of the whole scene;
removing an indirect component of the entire scene from the second reflected light signal.
According to a second aspect of embodiments of the present application, there is provided a TOF depth resolving method including:
receiving a first reflected light signal reflected by a scene collected by a TOF camera, wherein the first reflected light signal is obtained by performing dot matrix illumination reflection on the scene by the TOF camera at a certain modulation frequency;
performing phase calculation on the first reflected light signal to obtain the phase of the corresponding position of the dot-matrix illumination light spot;
receiving a second reflected light signal reflected by a scene collected by a TOF camera, wherein the second reflected light signal is obtained by performing floodlight illumination reflection on the scene by the TOF camera at two different modulation frequencies;
performing data reconstruction on the second reflected light signal, representing the second reflected light signal as a sum of a direct component and an indirect component;
resolving an indirect component from the second reflected light signal at a position corresponding to the dot-matrix illumination spot using the phase;
carrying out two-dimensional interpolation on the indirect components to obtain indirect components of the whole scene;
removing the indirect component of the whole scene from the second reflected light signal to obtain a direct component;
and solving the depth value for suppressing the multipath interference from the direct component.
Further, resolving an indirect component from the second reflected light signal using the phase at the corresponding position of the dot-matrix illumination spot, comprising:
fixing phase values of direct components of the low-frequency reflected light signals and the high-frequency reflected light signals in the second reflected light signals at the positions corresponding to the lattice-type lighting light spots respectively into phase values of low frequencies and high frequencies used by floodlight corresponding to the phases;
after the phase value is fixed, solving the amplitude and the phase of the indirect component from the second reflected light signal at the position corresponding to the dot-matrix illumination light spot according to the principle of minimum decomposition error;
and calculating the value of the indirect component at the position corresponding to the lattice type illumination light spot according to the calculated amplitude and phase of the indirect component.
Further, performing two-dimensional interpolation on the indirect component to obtain an indirect component of the whole scene, including:
when a region R with changed reflectivity exists in the scene surface, detecting to obtain the position of the region R, and estimating the relative reflectivity k of the region R;
after solving the indirect component MPI at the corresponding position of the dot-matrix illumination light spot, the indirect component MPI at the corresponding position of the dot-matrix illumination light spot belonging to the region R is subjected toRTo MPIR′＝MPIRK, the indirect component of the whole scene at the corresponding position of the lattice type lighting light spots becomes MPI';
to the lattice typeTwo-dimensional interpolation is carried out on the indirect component MPI ' at the position corresponding to the lighting light spot to obtain the indirect component MPI ' of the whole scene surface 'full；
The influence of the relative reflectivity k is added back to the indirect component corresponding to the region R,
obtaining indirect component MPI of whole scenefull。
According to a third aspect of embodiments of the present application, there is provided a multipath interference suppression apparatus in TOF depth resolution, including:
the system comprises a first receiving module, a second receiving module and a control module, wherein the first receiving module is used for receiving a first reflected light signal reflected by a scene collected by a TOF camera, and the first reflected light signal is obtained by performing dot-matrix illumination reflection on the scene by the TOF camera at a certain modulation frequency;
the first resolving module is used for performing phase resolving on the first reflected light signal to obtain the phase of the corresponding position of the dot-matrix illumination light spot;
the second receiving module is used for receiving a second reflected light signal reflected by a scene and collected by the TOF camera, and the second reflected light signal is obtained by performing floodlight illumination reflection on the scene by the TOF camera at two different modulation frequencies;
a first reconstruction module configured to perform data reconstruction on the second reflected light signal, and represent the second reflected light signal as a sum of a direct component and an indirect component;
the first decomposition module is used for decomposing indirect components from the second reflected light signals by utilizing the phases at the positions corresponding to the dot-matrix illumination light spots;
the first interpolation module is used for carrying out two-dimensional interpolation on the indirect components to obtain the indirect components of the whole scene;
a first removal module to remove an indirect component of the entire scene from the second reflected light signal.
According to a fourth aspect of embodiments of the present application, there is provided a TOF depth resolving device including:
the third receiving module is used for receiving a first reflected light signal reflected by a scene collected by a TOF camera, wherein the first reflected light signal is obtained by performing dot-matrix illumination reflection on the scene by the TOF camera at a certain modulation frequency;
the second resolving module is used for performing phase resolving on the first reflected light signal to obtain the phase of the corresponding position of the dot-matrix illumination light spot;
the fourth receiving module is used for receiving a second reflected light signal reflected by a scene and collected by the TOF camera, and the second reflected light signal is obtained by performing floodlight illumination reflection on the scene by the TOF camera at two different modulation frequencies;
a second reconstruction module for performing data reconstruction on the second reflected light signal and representing the second reflected light signal as a sum of a direct component and an indirect component;
the second decomposition module is used for decomposing indirect components from the second reflected light signals by utilizing the phases at the positions corresponding to the dot-matrix illumination light spots;
the second interpolation module is used for carrying out two-dimensional interpolation on the indirect components to obtain the indirect components of the whole scene;
a second removing module, configured to remove an indirect component of the entire scene from the second reflected light signal to obtain a direct component;
and the third calculation module is used for calculating the depth value for restraining the multipath interference from the direct component.
According to a fifth aspect of embodiments of the present application, there is provided an electronic apparatus, including:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a method as described in the first aspect or the second aspect.
According to a sixth aspect of embodiments herein, there is provided a computer readable storage medium having stored thereon computer instructions, characterized in that the instructions, when executed by a processor, implement the steps of the method according to the first or second aspect.
The technical scheme provided by the embodiment of the application can have the following beneficial effects:
according to the embodiment, the direct component of the scene primary reflection is separated from the indirect component caused by the multipath interference by combining the depth measurement results of the dot matrix illumination and the two times of floodlight illumination with different frequencies, so that the influence of the multipath interference in TOF depth calculation is inhibited, and a more accurate depth measurement value is obtained by the TOF camera when the scene with the multipath interference is imaged.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
Fig. 1 is a diagram illustrating multipath reflection, according to an example embodiment.
Fig. 2 is a flow chart illustrating a method of multipath interference mitigation in TOF depth resolution according to an exemplary embodiment.
FIG. 3 is a schematic view of TOF measurements shown according to an exemplary embodiment.
FIG. 4 is a schematic diagram of a dot matrix illumination pattern shown in accordance with an exemplary embodiment.
FIG. 5 is a schematic diagram illustrating a scene measurement with reflectivity variation according to an exemplary embodiment.
Fig. 6 is a block diagram illustrating a multipath interference suppression apparatus in TOF depth resolution according to an exemplary embodiment.
FIG. 7 is a flowchart illustrating a method of TOF depth resolution according to an exemplary embodiment
FIG. 8 is a block diagram illustrating a TOF depth solver, according to an exemplary embodiment.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It should be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
Example 1:
fig. 1 is a schematic diagram of multipath reflection according to an exemplary embodiment, and with reference to fig. 1, a multipath interference situation is illustrated by a corner model: a common TOF camera performs primary flood illumination on a scene by a light source and acquires a reflected light signal by a photosensitive receiving module to perform depth calculation, however, a direct component is generated by primary reflection after the surface of the scene is irradiated by a light beam 101 emitted by the light source, and stray light (such as 102 and 103) generated by reflection of other surfaces is irradiated and reflected to generate an indirect component, and a reflected light signal 104 received by the photosensitive receiving module is the superposition of the direct component and the indirect component. The depth value calculated by the superposition signal has a large deviation from the real depth value.
Fig. 2 is a flowchart illustrating a method for multipath interference suppression in TOF depth resolution according to an exemplary embodiment, and referring to fig. 2, an embodiment of the present invention provides a method for multipath interference suppression in TOF depth resolution, which may include:
step S11, receiving a first reflected light signal reflected by a scene collected by a TOF camera, wherein the first reflected light signal is obtained by the TOF camera through dot-matrix illumination reflection on the scene at a certain modulation frequency;
step S12, performing phase calculation on the first reflected light signal to obtain the phase of the corresponding position of the dot-matrix illumination light spot;
step S13, receiving a second reflected light signal reflected by a scene and collected by a TOF camera, wherein the second reflected light signal is obtained by performing floodlight illumination reflection on the scene by the TOF camera at two different modulation frequencies;
step S14, performing data reconstruction on the second reflected light signal, and representing the second reflected light signal as a sum of a direct component and an indirect component;
step S15, using the phase to resolve indirect components from the second reflected light signal at the position corresponding to the dot-matrix illumination spot;
step S16, carrying out two-dimensional interpolation on the indirect components to obtain indirect components of the whole scene;
step S17, removing the indirect component of the entire scene from the second reflected light signal.
According to the embodiment, the direct component of the primary reflection of the scene is separated from the indirect component caused by the multipath interference by combining the depth measurement results of the dot matrix illumination and the two floodlight illuminations with different frequencies, so that the influence of the multipath interference in the TOF depth calculation is inhibited.
The method can also comprise a preprocessing step, wherein a scene without multipath interference is constructed, and the depth correction is respectively carried out on the TOF camera under the conditions of dot matrix illumination and floodlight illumination;
specifically, as shown in fig. 3, an illumination module 301 of the TOF camera may emit a light beam 302 in a lattice pattern to perform lattice illumination on the scene, and the reflected light beam is received by a photoreception receiving module 303; lighting module 304 can flood light the scene with a beam of light having a field angle that is coincident with beam 302, and both lighting modules are not operative at the same time. A scene without multipath interference is constructed, depth correction is respectively carried out on the TOF camera under the conditions of dot matrix illumination and flood illumination, and the depth values obtained when reflected light signals of the same scene collected by the light sensing receiving module under different illumination conditions are resolved can be consistent with the real depth.
In an implementation of step S11, a TOF camera is received to acquire a first reflected light signal reflected by a scene, where the first reflected light signal is obtained by the TOF camera by dot matrix illumination reflection of the scene at a certain modulation frequency;
specifically, the modulation frequency is selected to enable the scene depth to be within the maximum measurement range of the modulation frequency, the duty ratio of the dot-matrix illumination light spots is set to enable the proportion of the indirect component and the direct component measured at the illumination position to meet the precision requirement required by measurement, and the reflected light field generated when the diffuse reflection object is illuminated can slowly change. Fig. 4 is a schematic diagram of the dot matrix pattern used, in which the size of the illumination spot 401 is 3 × 3 pixels, and the interval 402 between adjacent spots is 12 pixels wide, that is, the area of the illumination spot occupies 1/25 of the total pattern area, for convenience of description, since the illumination spots are sparsely distributed and the energy is concentrated, the signal received by the photosensitive module can be approximately regarded as having only a direct component and no indirect component. In practical use, the illumination spots 401 and the spot intervals 402 may have other sizes, and the spot shapes are not limited to rectangular shapes, but may be circular or any other shapes.
In the specific implementation of step S12, performing phase calculation on the first reflected light signal to obtain a phase of a corresponding position of the dot-matrix illumination spot;
in particular, the phase resolution of the photoreceptive receive module is typically calculated using multiple phase shift measurements, most often with double positivesSinusoidal wave frequency mixing four-phase measurement to obtain a phase shift of 0,
π、
Time of flight received wave Si(i is 0,1,2,3), then the phase change between the outgoing light signal and the reflected light signal can be calculated by the following equation:
according to each SiThe phase interval is judged according to the relative size of the light spot, and the phase interval is adjusted to a corresponding quadrant, so that a phase value which is influenced by less multipath at the position corresponding to the dot-matrix illumination light spot is obtained.
In a specific implementation of step S13, receiving a second reflected light signal reflected by the scene and collected by the TOF camera, where the second reflected light signal is obtained by performing flood illumination reflection on the scene by the TOF camera at two different modulation frequencies;
in particular, the emitting lighting modules are respectively at two different modulation frequencies f0,f1Carrying out flood lighting on a scene twice, collecting reflected light signals reflected by the scene by a photosensitive receiving module, recording the reflected light signals as low-frequency reflected light signals and high-frequency reflected light signals, and recording the ratio of the high frequency to the low frequency as gamma as f1/f0。
In a specific implementation of step S14, performing data reconstruction on the second reflected light signal, and representing the second reflected light signal as a sum of a direct component and an indirect component;
specifically, taking the cosine form as an example, the second reflected light signal is expressed as a mode in which a direct component and an indirect component are superimposed:
the left side of the middle sign in the above formula is the actual measurement value of the receiving module, and the right side is the direct component and the indirect component decomposed according to the theoretical assumption. Where the letter superscript represents the modulation frequency of the emitted light signal, a 0 in the letter subscript represents that it belongs to a direct component, and a 1 represents that it belongs to an indirect component.
And
are all at a low frequency f0The phase decomposed under the condition is in gamma-fold relation with two modulation frequencies, and the corresponding measurement decomposed phase value is also in gamma-fold relation.
In a specific implementation of step S15, resolving an indirect component from the second reflected light signal at a position corresponding to the dot-matrix illumination spot using the phase; this step may include the following sub-steps:
step S151, fixing phase values of direct components of the low-frequency reflected light signals and the high-frequency reflected light signals in the second reflected light signals at the positions corresponding to the lattice-type lighting light spots into phase values of low-frequency and high-frequency phases corresponding to floodlight;
step S152, after the phase value is fixed, according to the principle that the decomposition error is minimum, the amplitude and the phase of the indirect component are solved from the second reflected light signal at the position corresponding to the dot-matrix illumination light spot;
in particular, the second reflected light signal is accurately subjected to direct component and indirect component decomposition, i.e. a is determined0，a1，
So that the decomposition error is minimized. By using
Are respectively shown at f0And f1The measured values for each phase when flood lighting is performed at a modulation frequency are then:
the above formula is used for calculating the pixel of the corresponding position of the dot-matrix illumination light spot, and the phase value of the direct component is calculated in the above step
Fixed as phase at f with less multipath effect for dot matrix illumination0Corresponding phase values at modulation frequency, thus the variables to be solved for
The number of variables is reduced, and the calculated amount is reduced. Extracting the amplitude a of the indirect component1And phase
For the calculation of indirect component values.
And step S153, calculating the value of the indirect component at the position corresponding to the lattice type illumination spot according to the amplitude and the phase of the calculated indirect component.
Specifically, the amplitude a of the indirect component is measured1And phase
Extracting, calculating indirect components MPI at the corresponding positions of the dot-matrix illumination light spots, and expressing the rest chord components MPI _ cos and sine components MPI _ sin as follows:
in the specific implementation of step S16, performing two-dimensional interpolation on the indirect component to obtain an indirect component of the entire scene;
specifically, as shown in fig. 5, when there is an area 501 with a changed reflectivity on the scene surface, the influence of the abrupt reflectivity change on the indirect component needs to be eliminated first; this step may include the following sub-steps:
step S161, when there is an area 501 with a changed reflectivity on the scene surface, marking as an area R, detecting to obtain the position of the area R, and estimating the relative reflectivity k of the area R;
step S162, after solving the indirect component MPI at the corresponding position of the dot-matrix illumination light spot, solving the indirect component MPI at the corresponding position of the dot-matrix illumination light spot belonging to the region RRTo MPIR′＝MPIRK, the indirect component of the whole scene at the corresponding position of the lattice type lighting light spots becomes MPI';
in particular, the reflected light measurement obtained during flood lighting is affected by the relative reflectivity
Become into
Therefore, after the direct component and the indirect component of the second reflected light signal are decomposed at the position corresponding to the dot-matrix illumination light spot and the MPI value is extracted, the MPI value of the indirect component at the position corresponding to the dot-matrix illumination light spot belonging to the R region is extractedRTo MPIR′＝MPIRAnd/k, reducing the abrupt change of the indirect component caused by the area to ensure that the abrupt change of the indirect component is approximately kept to be uniform, and obtaining the indirect component MPI' of the whole scene at the corresponding position of the dot-matrix illumination light spot.
Step S163, carrying out two-dimensional interpolation on the indirect component MPI ' at the position corresponding to the dot-matrix illumination light spot to obtain an indirect component MPI ' of the whole scene surface 'full；
Step S164, adding the influence of the relative reflectivity k back to the indirect component corresponding to the region R,
obtaining indirect component MPI of whole scenefull。
In particular, the indirect component MPI 'of the entire scene surface'fullThe indirect component which is approximately uniformly changed is obtained by carrying out two-dimensional interpolation after reducing the influence of the relative reflectivity k, and actually, the region R is still influenced by the relative reflectivity k, so the relative reflectivity k is multiplied back to the indirect component of the region R, and the data are in line with the actual situation.
In a specific implementation of step S17, the indirect component of the entire scene is removed from the second reflected light signal.
In particular, the direct component direct may be reflected by said second reflected light signal W illuminated from flood lightmAnd (3) calculating an indirect component MPI of the whole scene obtained by subtracting the interpolation: direct ═ WmMPI, which achieves the suppression of multipath interference.
Corresponding to the embodiment of the multipath interference suppression method in the TOF depth calculation, the application also provides an embodiment of a multipath interference suppression device in the TOF depth calculation.
Fig. 6 is a block diagram illustrating a multipath interference suppression apparatus in TOF depth resolution according to an exemplary embodiment. Referring to fig. 6, the apparatus includes:
a first receiving module 61, configured to receive a first reflected light signal reflected by a scene acquired by a TOF camera, where the first reflected light signal is obtained by performing dot-matrix illumination reflection on the scene by the TOF camera at a certain modulation frequency;
the first calculating module 62 is configured to perform phase calculation on the first reflected light signal to obtain a phase of a corresponding position of the dot-matrix illumination spot;
a second receiving module 63, configured to receive a second reflected light signal reflected by the scene and acquired by the TOF camera, where the second reflected light signal is obtained by performing floodlight illumination reflection on the scene at two different modulation frequencies by the TOF camera;
a first reconstruction module 64 configured to perform data reconstruction on the second reflected light signal, and represent the second reflected light signal as a sum of a direct component and an indirect component;
a first decomposition module 65 for decomposing an indirect component from the second reflected light signal at the position corresponding to the dot-matrix illumination spot by using the phase;
a first interpolation module 66, configured to perform two-dimensional interpolation on the indirect component to obtain an indirect component of the entire scene;
a first removal module 67 for removing an indirect component of the entire scene from the second reflected light signal.
With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the application. One of ordinary skill in the art can understand and implement it without inventive effort.
Correspondingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when executed by the one or more processors, cause the one or more processors to implement a multipath interference mitigation method as in the TOF depth solution described above.
Accordingly, the present application also provides a computer readable storage medium having stored thereon computer instructions, wherein the instructions, when executed by a processor, implement the multipath interference suppression method in TOF depth resolution as described above.
Example 2:
fig. 7 is a flowchart illustrating a TOF depth resolving method according to an exemplary embodiment, and referring to fig. 7, an embodiment of the present invention provides a TOF depth resolving method that may include:
step S21, receiving a first reflected light signal reflected by a scene collected by a TOF camera, wherein the first reflected light signal is obtained by the TOF camera through dot-matrix illumination reflection on the scene at a certain modulation frequency;
step S22, performing phase calculation on the first reflected light signal to obtain the phase of the corresponding position of the dot-matrix illumination light spot;
step S23, receiving a second reflected light signal reflected by a scene collected by a TOF camera, wherein the second reflected light signal is obtained by performing floodlight illumination reflection on the scene by the TOF camera at two different modulation frequencies;
a step S24 of performing data reconstruction on the second reflected light signal, and representing the second reflected light signal as a sum of a direct component and an indirect component;
step S25, using the phase to resolve indirect components from the second reflected light signal at the position corresponding to the dot-matrix illumination spot;
step S26, carrying out two-dimensional interpolation on the indirect components to obtain the indirect components of the whole scene;
step S27, removing the indirect component of the entire scene from the second reflected light signal to obtain a direct component;
in step S28, the depth value for suppressing the multipath interference is calculated from the direct component.
According to the embodiment, the direct component of the primary reflection of the scene is separated from the indirect component caused by the multipath interference by combining the depth measurement results of the dot matrix illumination and the flood illumination with different frequencies twice, so that the influence of the multipath interference in TOF depth calculation is inhibited, and a more accurate depth measurement value is obtained by the TOF camera when the scene with the multipath interference is imaged.
Step S21-step S27 referring to step S11-step S17 of embodiment 1,
corresponding to the embodiment of the TOF depth calculating method, the application also provides an embodiment of a TOF depth calculating device.
FIG. 8 is a block diagram illustrating a TOF depth solver, according to an exemplary embodiment. Referring to fig. 8, the apparatus includes:
a third receiving module 81, configured to receive a first reflected light signal reflected by a scene acquired by a TOF camera, where the first reflected light signal is obtained by performing dot-matrix illumination reflection on the scene by the TOF camera at a certain modulation frequency;
the second resolving module 82 is configured to perform phase resolving on the first reflected light signal to obtain a phase of a corresponding position of the dot-matrix illumination spot;
a fourth receiving module 83, configured to receive a second reflected light signal reflected by the scene and acquired by the TOF camera, where the second reflected light signal is obtained by performing floodlighting reflection on the scene at two different modulation frequencies by the TOF camera;
a second reconstruction module 84 for performing data reconstruction on the second reflected light signal, representing the second reflected light signal as a sum of a direct component and an indirect component;
a second decomposition module 85, configured to decompose an indirect component from the second reflected light signal at a position corresponding to the dot-matrix illumination spot by using the phase;
a second interpolation module 86, configured to perform two-dimensional interpolation on the indirect component to obtain an indirect component of the entire scene;
a second removing module 87, configured to remove an indirect component of the entire scene from the second reflected light signal to obtain a direct component;
and a third calculating module 88 for calculating the depth value for suppressing the multi-path interference from the direct component.
Correspondingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when executed by the one or more processors, cause the one or more processors to implement a TOF depth solution method as described above.
Accordingly, the present application also provides a computer readable storage medium having stored thereon computer instructions, wherein the instructions, when executed by a processor, implement the TOF depth resolving method as described above.
Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the present disclosure. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It will be understood that the present application is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.
Claims (8)
1. A multipath interference suppression method in TOF depth calculation is characterized by comprising the following steps:
receiving a first reflected light signal reflected by a scene collected by a TOF camera, wherein the first reflected light signal is obtained by performing dot matrix illumination reflection on the scene by the TOF camera at a certain modulation frequency;
performing phase calculation on the first reflected light signal to obtain the phase of the corresponding position of the dot-matrix illumination light spot;
receiving a second reflected light signal reflected by a scene collected by a TOF camera, wherein the second reflected light signal is obtained by performing floodlight illumination reflection on the scene by the TOF camera at two different modulation frequencies;
performing data reconstruction on the second reflected light signal, representing the second reflected light signal as a sum of a direct component and an indirect component;
resolving an indirect component from the second reflected light signal at a position corresponding to the dot-matrix illumination spot using the phase;
carrying out two-dimensional interpolation on the indirect components to obtain indirect components of the whole scene;
removing an indirect component of the entire scene from the second reflected light signal;
wherein, resolving an indirect component from the second reflected light signal using the phase at the position corresponding to the dot-matrix illumination spot comprises:
fixing phase values of direct components of the low-frequency reflected light signals and the high-frequency reflected light signals in the second reflected light signals at the positions corresponding to the lattice-type lighting light spots respectively into phase values of low frequencies and high frequencies used by floodlight corresponding to the phases;
after the phase value is fixed, solving the amplitude and the phase of the indirect component from the second reflected light signal at the position corresponding to the dot-matrix illumination light spot according to the principle of minimum decomposition error;
and calculating the value of the indirect component at the position corresponding to the lattice type illumination light spot according to the calculated amplitude and phase of the indirect component.
2. A TOF depth resolving method, comprising:
receiving a first reflected light signal reflected by a scene and acquired by a TOF camera, wherein the first reflected light signal is obtained by performing dot matrix illumination reflection on the scene by the TOF camera at a certain modulation frequency;
performing phase calculation on the first reflected light signal to obtain the phase of the corresponding position of the dot-matrix illumination light spot;
receiving a second reflected light signal reflected by a scene collected by a TOF camera, wherein the second reflected light signal is obtained by performing floodlight illumination reflection on the scene by the TOF camera at two different modulation frequencies;
performing data reconstruction on the second reflected light signal, representing the second reflected light signal as a sum of a direct component and an indirect component;
resolving an indirect component from the second reflected light signal at a position corresponding to the dot-matrix illumination spot using the phase;
carrying out two-dimensional interpolation on the indirect components to obtain indirect components of the whole scene;
removing the indirect component of the whole scene from the second reflected light signal to obtain a direct component;
resolving a depth value for suppressing the multipath interference from the direct component;
wherein, resolving an indirect component from the second reflected light signal using the phase at the position corresponding to the dot-matrix illumination spot comprises:
fixing phase values of direct components of the low-frequency reflected light signals and the high-frequency reflected light signals in the second reflected light signals at the positions corresponding to the lattice-type lighting light spots respectively into phase values of low frequencies and high frequencies used by floodlight corresponding to the phases;
after the phase value is fixed, solving the amplitude and the phase of the indirect component from the second reflected light signal at the position corresponding to the dot-matrix illumination light spot according to the principle of minimum decomposition error;
and calculating the value of the indirect component at the position corresponding to the lattice type illumination light spot according to the calculated amplitude and phase of the indirect component.
3. The method of claim 1 or 2, wherein the two-dimensional interpolation of the indirect components to obtain indirect components of the entire scene comprises:
when a region R with changed reflectivity exists in the scene surface, detecting to obtain the position of the region R, and estimating the relative reflectivity k of the region R;
solving the indirect component MPI at the corresponding position of the dot-matrix illumination light spot belonging to the region R
Become into
The indirect component of the whole scene at the corresponding position of the lattice type illumination light spot becomes
；
For indirect component at corresponding position of the dot-matrix illumination light spot
Two-dimensional interpolation is carried out to obtain indirect components of the whole scene surface
；
The influence of the relative reflectivity k is added back to the indirect component corresponding to the region R,
obtaining indirect components of the whole scene
。
4. A multipath interference suppression device in TOF depth calculation is characterized by comprising:
the system comprises a first receiving module, a second receiving module and a control module, wherein the first receiving module is used for receiving a first reflected light signal reflected by a scene collected by a TOF camera, and the first reflected light signal is obtained by performing dot-matrix illumination reflection on the scene by the TOF camera at a certain modulation frequency;
the first resolving module is used for performing phase resolving on the first reflected light signal to obtain the phase of the corresponding position of the dot-matrix illumination light spot;
the second receiving module is used for receiving a second reflected light signal reflected by the scene collected by the TOF camera, and the second reflected light signal is obtained by performing floodlight illumination reflection on the scene by the TOF camera at two different modulation frequencies;
a first reconstruction module configured to perform data reconstruction on the second reflected light signal, and represent the second reflected light signal as a sum of a direct component and an indirect component;
the first decomposition module is used for decomposing indirect components from the second reflected light signals by utilizing the phases at the positions corresponding to the dot-matrix illumination light spots;
the first interpolation module is used for carrying out two-dimensional interpolation on the indirect components to obtain the indirect components of the whole scene;
a first removal module for removing an indirect component of the entire scene from the second reflected light signal;
wherein, resolving an indirect component from the second reflected light signal using the phase at the position corresponding to the dot-matrix illumination spot comprises:
fixing phase values of direct components of low-frequency reflected light signals and high-frequency reflected light signals in the second reflected light signals at positions corresponding to the dot-matrix lighting light spots into phase values of low frequencies and high frequencies used by floodlight corresponding to the phases respectively;
after the phase value is fixed, solving the amplitude and the phase of the indirect component from the second reflected light signal at the position corresponding to the dot-matrix illumination light spot according to the principle of minimum decomposition error;
and calculating the value of the indirect component at the position corresponding to the lattice type illumination light spot according to the calculated amplitude and phase of the indirect component.
5. A TOF depth resolving apparatus comprising:
the third receiving module is used for receiving a first reflected light signal reflected by a scene collected by a TOF camera, wherein the first reflected light signal is obtained by performing dot-matrix illumination reflection on the scene by the TOF camera at a certain modulation frequency;
the second resolving module is used for performing phase resolving on the first reflected light signal to obtain the phase of the corresponding position of the dot-matrix illumination light spot;
the fourth receiving module is used for receiving a second reflected light signal reflected by a scene and collected by the TOF camera, and the second reflected light signal is obtained by performing floodlight illumination reflection on the scene by the TOF camera at two different modulation frequencies;
a second reconstruction module for performing data reconstruction on the second reflected light signal and representing the second reflected light signal as a sum of a direct component and an indirect component;
the second decomposition module is used for decomposing indirect components from the second reflected light signals by utilizing the phases at the positions corresponding to the dot-matrix illumination light spots;
the second interpolation module is used for carrying out two-dimensional interpolation on the indirect components to obtain the indirect components of the whole scene;
a second removing module, configured to remove an indirect component of the entire scene from the second reflected light signal to obtain a direct component;
the third resolving module is used for resolving the depth value for restraining the multipath interference from the direct component;
wherein, resolving an indirect component from the second reflected light signal using the phase at the position corresponding to the dot-matrix illumination spot comprises:
fixing phase values of direct components of the low-frequency reflected light signals and the high-frequency reflected light signals in the second reflected light signals at the positions corresponding to the lattice-type lighting light spots respectively into phase values of low frequencies and high frequencies used by floodlight corresponding to the phases;
after the phase value is fixed, resolving the amplitude and the phase of the indirect component from the second reflected light signal at the position corresponding to the dot-matrix illumination light spot according to the principle that the resolution error is minimum;
and calculating the value of the indirect component at the position corresponding to the lattice type illumination light spot according to the calculated amplitude and phase of the indirect component.
6. The apparatus according to claim 4 or 5, wherein the two-dimensional interpolation of the indirect components to obtain indirect components of the whole scene comprises:
when a region R with changed reflectivity exists in the scene surface, detecting to obtain the position of the region R, and estimating the relative reflectivity k of the region R;
solving the indirect component MPI at the corresponding position of the dot-matrix illumination light spot belonging to the region R
Become into
The indirect component of the whole scene at the corresponding position of the lattice type illumination light spot becomes
；
For indirect component at the corresponding position of the dot-matrix illumination light spot
Two-dimensional interpolation is carried out to obtain indirect components of the whole scene surface
；
The influence of the relative reflectivity k is added back to the indirect component corresponding to the region R,
obtaining indirect components of the whole scene
。
7. An electronic device, comprising:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the method of claim 1 or 2.
8. A computer-readable storage medium having stored thereon computer instructions, which when executed by a processor, perform the steps of the method of claim 1 or 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111227054.8A CN113945951B (en) | 2021-10-21 | 2021-10-21 | Multipath interference suppression method in TOF (time of flight) depth calculation, TOF depth calculation method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111227054.8A CN113945951B (en) | 2021-10-21 | 2021-10-21 | Multipath interference suppression method in TOF (time of flight) depth calculation, TOF depth calculation method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113945951A CN113945951A (en) | 2022-01-18 |
| CN113945951B true CN113945951B (en) | 2022-07-08 |
Family
ID=79331821
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111227054.8A Active CN113945951B (en) | 2021-10-21 | 2021-10-21 | Multipath interference suppression method in TOF (time of flight) depth calculation, TOF depth calculation method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113945951B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10382736B1 (en) * | 2018-02-09 | 2019-08-13 | Infineon Technologies Ag | Two frequency time-of-flight three-dimensional image sensor and method of measuring object depth |
| CN111239729A (en) * | 2020-01-17 | 2020-06-05 | 西安交通大学 | Speckle and floodlight projection fused ToF depth sensor and distance measuring method thereof |
| CN111366941A (en) * | 2020-04-20 | 2020-07-03 | 深圳奥比中光科技有限公司 | TOF depth measuring device and method |
| CN111710024A (en) * | 2020-03-23 | 2020-09-25 | 上海数迹智能科技有限公司 | Method for eliminating multipath interference in ToF camera based on multi-frequency and multi-illumination |
| WO2020248335A1 (en) * | 2019-06-14 | 2020-12-17 | 深圳奥比中光科技有限公司 | Time depth camera and multi-frequency modulation and demodulation-based noise-reduction distance measurement method |
| CN112698309A (en) * | 2020-12-23 | 2021-04-23 | 宁波飞芯电子科技有限公司 | Multi-path inhibition method in TOF distance detection |
| CN112824934A (en) * | 2019-11-20 | 2021-05-21 | 深圳市光鉴科技有限公司 | TOF multi-path interference removal method, system, equipment and medium based on modulated light field |
| CN112824935A (en) * | 2019-11-20 | 2021-05-21 | 深圳市光鉴科技有限公司 | Depth imaging system, method, device and medium based on modulated light field |
| CN112859049A (en) * | 2020-12-23 | 2021-05-28 | 宁波飞芯电子科技有限公司 | Multi-path inhibition method and ranging system in TOF (time of flight) distance detection |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9753128B2 (en) * | 2010-07-23 | 2017-09-05 | Heptagon Micro Optics Pte. Ltd. | Multi-path compensation using multiple modulation frequencies in time of flight sensor |
| EP2955544B1 (en) * | 2014-06-11 | 2020-06-17 | Sony Depthsensing Solutions N.V. | A TOF camera system and a method for measuring a distance with the system |
| US10191154B2 (en) * | 2016-02-11 | 2019-01-29 | Massachusetts Institute Of Technology | Methods and apparatus for time-of-flight imaging |
| US20180278910A1 (en) * | 2017-03-22 | 2018-09-27 | Microsoft Technology Licensing, Llc | Correction of multipath interference in time of flight camera depth imaging measurements |
| CN111971578A (en) * | 2018-03-29 | 2020-11-20 | 松下半导体解决方案株式会社 | Distance information acquisition device, multipath detection device, and multipath detection method |
| US10663567B2 (en) * | 2018-05-04 | 2020-05-26 | Microsoft Technology Licensing, Llc | Field calibration of a structured light range-sensor |
| CN109991584A (en) * | 2019-03-14 | 2019-07-09 | 深圳奥比中光科技有限公司 | A kind of jamproof distance measurement method and depth camera |
| EP3835720B1 (en) * | 2019-12-10 | 2023-08-09 | Melexis Technologies NV | Method for multipath error compensation and multipath error-compensated indirect time of flight range calculation apparatus |
-
2021
- 2021-10-21 CN CN202111227054.8A patent/CN113945951B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10382736B1 (en) * | 2018-02-09 | 2019-08-13 | Infineon Technologies Ag | Two frequency time-of-flight three-dimensional image sensor and method of measuring object depth |
| WO2020248335A1 (en) * | 2019-06-14 | 2020-12-17 | 深圳奥比中光科技有限公司 | Time depth camera and multi-frequency modulation and demodulation-based noise-reduction distance measurement method |
| CN112824934A (en) * | 2019-11-20 | 2021-05-21 | 深圳市光鉴科技有限公司 | TOF multi-path interference removal method, system, equipment and medium based on modulated light field |
| CN112824935A (en) * | 2019-11-20 | 2021-05-21 | 深圳市光鉴科技有限公司 | Depth imaging system, method, device and medium based on modulated light field |
| CN111239729A (en) * | 2020-01-17 | 2020-06-05 | 西安交通大学 | Speckle and floodlight projection fused ToF depth sensor and distance measuring method thereof |
| CN111710024A (en) * | 2020-03-23 | 2020-09-25 | 上海数迹智能科技有限公司 | Method for eliminating multipath interference in ToF camera based on multi-frequency and multi-illumination |
| CN111366941A (en) * | 2020-04-20 | 2020-07-03 | 深圳奥比中光科技有限公司 | TOF depth measuring device and method |
| CN112698309A (en) * | 2020-12-23 | 2021-04-23 | 宁波飞芯电子科技有限公司 | Multi-path inhibition method in TOF distance detection |
| CN112859049A (en) * | 2020-12-23 | 2021-05-28 | 宁波飞芯电子科技有限公司 | Multi-path inhibition method and ranging system in TOF (time of flight) distance detection |
Non-Patent Citations (2)
| Title |
|---|
| Image stitching for Airborne Staring LIDAR by vertical range-gated registration;Heng Su等;《Proceedings of 2011 International Conference on Electronics and Optoelectronics》;20110908;全文 * |
| ToF 相机的有效深度数据提取与校正算法研究;乔欣等;《智能科学与技术学报》;20200331;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113945951A (en) | 2022-01-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10234561B2 (en) | Specular reflection removal in time-of-flight camera apparatus | |
| US9874638B2 (en) | Time of flight camera system which resolves direct and multi-path radiation components | |
| US10584963B2 (en) | System and methods for shape measurement using dual frequency fringe pattern | |
| US9879985B2 (en) | Simultaneous multiple view surface geometry acquisition using structured light and mirrors | |
| US9270974B2 (en) | Calibration between depth and color sensors for depth cameras | |
| CN104137150A (en) | Motion compensation in range imaging | |
| JP2021524908A (en) | Methods and systems for optimizing depth imaging | |
| Whyte et al. | Resolving multiple propagation paths in time of flight range cameras using direct and global separation methods | |
| EP3129803B1 (en) | Signal harmonic error cancellation method and apparatus | |
| Dobson et al. | Fast, large-scale, particle image velocimetry-based estimations of river surface velocity | |
| Istenič et al. | Automatic scale estimation of structure from motion based 3D models using laser scalers in underwater scenarios | |
| US6803997B2 (en) | Gridlocking and correlation methods and arrangements | |
| US20210263137A1 (en) | Phase noise and methods of correction in multi-frequency mode lidar | |
| Gerg et al. | GPU acceleration for synthetic aperture sonar image reconstruction | |
| CN112882030B (en) | InSAR imaging interference integrated processing method | |
| Ti et al. | Simultaneous time-of-flight sensing and photometric stereo with a single tof sensor | |
| CN113945951B (en) | Multipath interference suppression method in TOF (time of flight) depth calculation, TOF depth calculation method and device | |
| Maier-Hein et al. | Accounting for anisotropic noise in fine registration of time-of-flight range data with high-resolution surface data | |
| WO2019069787A1 (en) | Radar image processing device, radar image processing method, and radar image processing program | |
| US8818124B1 (en) | Methods, apparatus, and systems for super resolution of LIDAR data sets | |
| Mangelson et al. | Robust visual fiducials for skin-to-skin relative ship pose estimation | |
| JP2019046268A (en) | Image processing apparatus and program | |
| Lavieri et al. | Image-based measurement system for regular waves in an offshore basin | |
| Bonnett et al. | Data-driven image registration for coherent change detection of synthetic aperture sonar imagery | |
| WO2020104416A1 (en) | Method and system of decomposition of composite targets on elements of a radar target signature with a super-resolution, using the total signal spectrum |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |