Classifications

• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
  • G01S17/88—Lidar systems specially adapted for specific applications
  • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
  • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
  • G01S7/491—Details of non-pulse systems
  • G01S7/4911—Transmitters
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
  • G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
  • G01S17/06—Systems determining position data of a target
  • G01S17/08—Systems determining position data of a target for measuring distance only
  • G01S17/10—Systems determining position data of a target for measuring distance only using transmission of interrupted, pulse-modulated waves
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
  • G01S17/88—Lidar systems specially adapted for specific applications
  • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
  • G01S17/894—Three-dimensional [3D] imaging with simultaneous measurement of time-of-flight at a two-dimensional [2D] array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
  • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
  • G01S7/483—Details of pulse systems
  • G01S7/486—Receivers
  • G01S7/487—Extracting wanted echo signals, e.g. pulse detection
  • G01S7/4876—Extracting wanted echo signals, e.g. pulse detection by removing unwanted signals
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
  • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
  • G01S7/491—Details of non-pulse systems
  • G01S7/493—Extracting wanted echo signals
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
  • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
  • G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
  • G01S17/06—Systems determining position data of a target
  • G01S17/46—Indirect determination of position data

Definitions

• the invention relates (i) to a method for measuring a distance between an object of a scene and a Time-Of-Flight camera system and (ii) to the Time-Of-Flight camera system associated therewith.
• the invention relates to the problem of direct and indirect reflections of light within a scene and to the errors of depth measurements induced by these multiple reflections. It should be understood that by scene is meant all the surfaces surrounding the object onto which a light beam could be directly or indirectly reflected.
• Time-Of-Flight technology is a promising technology for depth perception.
• the well-known basic operational principle of a standard TOF camera system 3 is illustrated in Figure 1 .
• the TOF camera system 3 captures 3D images of a scene 15 by analysing the time of flight of light from a dedicated illumination unit 18 to an object.
• TOF camera system 3 includes a camera, for instance a matrix of pixels 1 , and data processing means 4.
• the scene 15 is actively illuminated with a modulated light 16 at a predetermined wavelength using the dedicated illumination unit 18, for instance with some light pulses of at least one predetermined frequency.
• the modulated light is reflected back from objects within the scene.
• a lens 2 collects the reflected light 17 and forms an image of the objects onto the imaging sensor 1 of the camera.
• a standard TOF camera system 9 is represented, comprising an illumination unit 8 for illuminating a scene 24 in multiple directions, a TOF sensor 6 for detecting the reflections of the emitted light and processing means 7 for processing the data acquired by the TOF sensor 6.
• the light captured by the sensor 6 can originate from both the direct path 25 and the secondary reflection 26, the depth map 27, representing the depth associated to each point of the scene, measured is thereby erroneous.
• several methods have been implemented to recover the direct component of impinging light. For instance, multiple frequency approaches have been performed, by acquiring a set of depth measurements with different modulation frequencies, but the resolution obtained remains low.
• Another approach uses a set of different spatial patterns which are generated by, for example, a Digital Light Processing (DLP) projector.
• DLP Digital Light Processing
• the direct and indirect components can be separated, as the depth acquired from the black part of the pattern is created by only indirect signal originating from multipath.
• the different patterns are chosen in such a way that each part of the scene is captured in a black situation. Edge-effects are countered by defining the patterns with sufficient overlaps.
• the creation of these different patterns is expensive.
• a solution remains to be proposed in order to retrieve only the direct component of the reflected light in the most cost-effective way, in order to perform more accurate measurements of the distances between objects of a scene and the TOF camera system.
• the present invention relates to a method for measuring a distance between an object of a scene and a Time-Of-Flight camera system, and providing a depth map of the object, the Time-Of-Flight camera system comprising an illumination unit, an imaging sensor having a matrix of pixels and image processing means, the method being characterized by the following steps:
• the method when processing the data, could comprise a step of identifying, for instance on an intermediary depth maps but not only, peaks associated to elementary areas on which only indirect incident light beams can impinge. By identifying such peaks, the data could be processed for eliminating the influence of indirect light beams and obtaining a final accurate depth map of the scene.
• the present invention also relates to a Time-Of-Flight (TOF) camera system for measuring a distance between an object of a scene and the TOF camera system, and providing a depth map of the object, the TOF camera system comprising:
• an illumination unit for illuminating the scene with a modulated light
• an imaging sensor having a matrix of pixels for receiving onto the pixels of the matrix of the sensor the beams reflected by the scene
• - image processing means for receiving from the imaging sensor, data corresponding to the reflected beams and for processing the said corresponding data; characterized in that it further comprises: - patterning means for modifying in a discrete way the illumination of said illumination unit in order to illuminate elementary areas of the scene with different incident intensities, respectively, for distinguishing the direct incident light beams from the indirect incident light beams and for eliminating the influence of the indirect light beams in the depth map of the object in processing the said corresponding data.
• the modification of the illumination is performed by masking the illumination unit in a discrete way.
• the patterning means could be for instance a mask avoiding direct incident light beams to impinge on some elementary areas of the scene.
• the patterning means could comprise a series of identical pattern groups for enabling an easier processing of the data.
• Figure 1 illustrates the basic operational principle of a TOF camera system
• FIG. 1 illustrates the multipath phenomenon
• FIG 3 illustrates a TOF camera system according to an embodiment of the invention
• Figure 4 illustrates an example of patterning means
• Figure 5 illustrates an intermediary depth map of a scene and the associated pixels of the TOF camera system
• Figure 6 illustrates a scene and a light pattern projected on the scene comprising two different spatial zones.
• FIG. 3 illustrates a TOF camera system 10 according to an embodiment of the invention.
• the Time-Of-Flight camera system 10 comprises an illumination unit 20 for illuminating a scene 24 with a modulated light.
• the light emitted by this illumination unit 20 is arranged for being suitable for measuring distances using the Time-Of-Flight technology.
• the illumination unit 20 may be arranged for emitting light pulses with an appropriate pulse width. Indeed, when using pulses, the pulse width of each light pulse determines the camera range. For instance, for a pulse width of 50 ns, the range is limited to 7.5 m.
• the illumination unit is arranged for emitting multi-directional light, as represented by the plurality of emitted light rays 25, 26 and 28 represented in Figure 3.
• the TOF camera system further comprises an imaging sensor 21 typically comprising a matrix array of pixels, for receiving and detecting the reflected beams and forming an image of the scene 24.
• an imaging sensor 21 typically comprising a matrix array of pixels, for receiving and detecting the reflected beams and forming an image of the scene 24.
• an imaging sensor 21 typically comprising a matrix array of pixels, for receiving and detecting the reflected beams and forming an image of the scene 24.
• an imaging sensor 21 typically comprising a matrix array of pixels, for receiving and detecting the reflected beams and forming an image of the scene 24.
• an imaging sensor 21 typically comprising a matrix array of pixels, for receiving and detecting the reflected beams and forming an image of the scene 24.
• the TOF camera system 10 further comprises patterning means 30 for creating a light pattern on the scene 24.
• the light pattern may be a native laser speckle pattern obtained directly from laser light interferences, or be obtained from patterning means which may be placed in front of the illumination unit 20, or a combination of both the laser speckle and a patterning means 30.
• the patterning means 30 the light emitted by the illumination unit 20 passes through these patterning means, causing the light to be modified and to form a light pattern on the scene, with delimited elementary areas 31 , 32 of different intensities.
• the light emitted by the illumination unit 20 may be blocked or its intensity may be reduced on given areas of the patterning means 30 and not blocked on other areas, resulting in the creation of areas with low light intensity 31 and high light intensity 32, respectively, on the scene.
• these areas have been represented by thick lines 31 and 32, but it should be understood that the light pattern created on the scene is not a solid or a physical pattern attached on the scene 24, but the result of light effects originated from the patterning means 30 placed in front of the illumination unit 20.
• the light pattern is projected on the scene by the illumination unit 20.
• the patterning means 30 can be filtering means, a mask, a grid or any means enabling to modify in a discrete way the illumination.
• the patterning means should provide a spatially periodic light pattern 31 , 32, 45, 46 on the scene, for retrieving easily the areas 31 where only secondary reflections are measured.
• the patterning means 30 could also comprise a series of identical pattern groups 50 as represented in Figure 4 or a series of different pattern groups that can be sequentially used in time in synchrony with a multiple of the TOF camera frame rate.
• the TOF camera system 10 further comprises processing means 5 for determining the time of flight of light emitted by the illumination 20, and thereby, the distance between an object of the scene 24 and the imaging sensor 21 .
• the processing means 30 are arranged for receiving data from the pixels of the imaging sensor 21 and for processing them for eliminating the influence of the indirect light beams in the depth map of the object. The method for determining this distance, and a final and accurate depth map of the object, will be described in the following paragraphs.
• the time of flight can be calculated in a separate processing unit which may be coupled to the TOF sensor 21 or may directly be integrated into the TOF sensor itself.
• the processing means 5 have been represented coupled to the illumination unit 20, but the invention is not limited thereto.
• Time-Of-Flight camera system 10 comprising an illumination unit 20, an imaging sensor 21 having a matrix of pixels 22, 23 and image processing means 30, will be now described, by referring to Figure 3, Figure 4, Figure 5 and Figure 6.
• the method comprises the step of modifying in a discrete way the illumination of the illumination unit 20 in order to illuminate elementary areas 31 , 32 of the scene with different incident intensities, respectively, for distinguishing the direct incident light beam 25 from the indirect incident light beam 26, 28.
• This modification can be performed for instance by creating a light pattern on a scene, the light pattern comprising delimited elementary areas with high and low light intensity.
• This light pattern can be created by placing the patterning means 30 in front of the illumination unit 20, and thus, projecting a light pattern on the scene.
• the pixels of the sensor 21 receive the beams reflected by these elementary areas 31 , 32 and provide the image processing means 30 with corresponding data.
• the light pattern projected on the scene can be retrieved on the intermediary depth map 29. This is illustrated by Figure 3 and Figure 5.
• Figure 3 and Figure 5 By comparing the depth maps 27 of Figure 2 and 29 of Figure 3, without and with the use of the patterning means, respectively, one can notice the apparition of peaks 33. These peaks 33 correspond to areas 31 of the scene 24 where only secondary reflections are measured. Indeed, in these areas, without secondary reflection, pixels of the imaging sensor 21 should not measured light, or with a small intensity because, by definition, these areas 31 are associated to areas of the patterning means 30 where the light is blocked or where the light intensity is reduced.
• the light measured on pixel 23 is more dominated by the secondary reflection 28, whereas the light measured on pixel 22 corresponds to both direct and indirect components 25, 26.
• the fact of identifying, for instance on an intermediary depth map 29, elementary areas 31 on which no direct incident light beam can impinge can be used for eliminating the influence of indirect light beams and obtaining a final and accurate depth map of the object.
• the complex data obtained by the time of flight measurement of light dominated by the indirect components in pixel 23 can for example be subtracted from the complex data obtained by both direct and indirect components in pixel 22 to form a new complex data NC. If the indirect component contributions to the complex data are equal in pixel 22 and 23, the resulting complex data NC only contains information from the direct component. Even if pixel 23 still received a small direct component due to the limited contrast of the patterning means 30, the resulting complex data NC will have a smaller amplitude but will have the correct phase representing the time of flight of the direct component.
• Figure 5 illustrates an intermediary depth map of an object of the scene 29 and the associated pixels of the TOF camera system.
• Pixels 40 measure only indirect components and are associated to higher depth and peaks 33, whereas pixels 41 measure both direct and indirect components, and are associated to areas 34 of the depth map 29.
• the identification of the pixels corresponding to the areas 31 can also be done using a signal intensity map, where these pixels will have lower intensity due to the missing direct components.
• Confidence maps or noise maps can also be used to identify the pixels associated with the areas 31 .
• the complex value measured by the pixel 22 can be subtracted from the complex value measured by the pixel 23 to form a new complex value NC.
• an indirect component function can be constructed by the samples taken on the areas 31 . This indirect component function can then be interpolated for all pixels having both direct and indirect components and subtracted from these pixels, leaving only direct components.
• the value associated to indirect components is a continuous function which can be easily sampled by all areas 31 , because the indirect components originate from a Lambertian reflectance of the scene 24.
• the scene 40 of Figure 6 comprises for instance a first wall 43 with a door and a second wall 41 , on which a cupboard 42, with a given depth, is mounted.
• a cupboard 42 with a given depth
• indirect reflections originating from the reflection of the cupboard 42 or from the wall 43 do not lead to similar measurements.
• different spatial zones 45, 46 of light pattern on the scene 40 can be determined.
• different shapes have been used in Figure 6, but it should be understood that both light sub-patterns 45 and 46 originate from the same patterning means 30 placed in front of the illumination unit 20 of the TOF camera system 10.
• the scene is now first segmented using the depth data available or any additional data useful for segmenting the scene.
• an continuous function can be associated with the indirect components, which can be sampled by the areas 31 belonging to each segment respectively.
• This indirection component function linked to each segment can then be used to compensate for the unwanted indirect components present in the pixels with both direct and indirect components.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Computer Networks & Wireless Communication (AREA)
• General Physics & Mathematics (AREA)
• Radar, Positioning & Navigation (AREA)
• Remote Sensing (AREA)
• Electromagnetism (AREA)
• Optical Radar Systems And Details Thereof (AREA)
• Measurement Of Optical Distance (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=50927987

Family Applications (1)

Country Status (7)

Cited By (1)

Families Citing this family (43)

Citations (3)

Family Cites Families (18)

• 2014
  • 2014-06-11 EP EP14171985.6A patent/EP2955544B1/en active Active
  • 2014-07-25 BE BE2014/0578A patent/BE1022486B1/fr not_active IP Right Cessation
• 2015
  • 2015-06-11 US US15/317,367 patent/US10901090B2/en active Active
  • 2015-06-11 CN CN201580036607.8A patent/CN106662651B/zh active Active
  • 2015-06-11 KR KR1020177000302A patent/KR102432765B1/ko active Active
  • 2015-06-11 WO PCT/EP2015/063015 patent/WO2015189311A1/en not_active Ceased
  • 2015-06-11 JP JP2016572252A patent/JP2017517737A/ja active Pending
• 2020
  • 2020-11-26 JP JP2020195625A patent/JP7191921B2/ja active Active

Patent Citations (3)

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events