WO2022241942A1 - 一种深度相机及深度计算方法 - Google Patents

一种深度相机及深度计算方法 Download PDF

Info

Publication number
WO2022241942A1
WO2022241942A1 PCT/CN2021/107947 CN2021107947W WO2022241942A1 WO 2022241942 A1 WO2022241942 A1 WO 2022241942A1 CN 2021107947 W CN2021107947 W CN 2021107947W WO 2022241942 A1 WO2022241942 A1 WO 2022241942A1
Authority
WO
WIPO (PCT)
Prior art keywords
depth
frequency
processing circuit
target object
pixel
Prior art date
Application number
PCT/CN2021/107947
Other languages
English (en)
French (fr)
Inventor
孙瑞
孙飞
武万多
黄源浩
肖振中
Original Assignee
奥比中光科技集团股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 奥比中光科技集团股份有限公司 filed Critical 奥比中光科技集团股份有限公司
Publication of WO2022241942A1 publication Critical patent/WO2022241942A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
    • G01S17/894—3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar

Definitions

  • the present application belongs to the field of image processing, and in particular relates to a depth camera and a depth calculation method.
  • TOF uses time of flight to achieve 3D imaging, and its working distance gradually increases with the decrease of the light source modulation frequency, such as the farthest of 100MHz
  • the working distance is 1.5m
  • the farthest working distance of 60MHz is 2.5m
  • the furthest working distance of 20MHz is 7.5m.
  • the accuracy of TOF three-dimensional imaging gradually deteriorates as the frequency decreases.
  • TOF system design often adopts a high-low frequency mixed working mode (ie "dual-frequency measurement"), such as 100MHz&20MHz, its working distance reaches 7.5m, and its accuracy will be higher than Ranging accuracy of 20MHz is used alone.
  • a high-low frequency mixed working mode such as 100MHz&20MHz
  • the high-frequency and low-frequency dual-frequency measurement is performed in a "serial" manner in terms of signal timing, such as collecting high-frequency modulation and demodulation signals first, and then collecting low-frequency modulation and demodulation signals; or collecting low-frequency modulation and demodulation signals first. Modulation signal, and then collect high-frequency modulation and demodulation signal. After the high-frequency and low-frequency dual-frequency signal data is collected, “dual-frequency fusion" is performed, and finally the "de-mixing" algorithm is used to complete the final depth measurement.
  • the embodiment of the present application provides a depth camera and a depth calculation method, which can improve the obvious "motion artifact” phenomenon in the current high and low frequency modulation and demodulation signals, and can reduce system power consumption at the same time.
  • the embodiment of the present application provides a depth camera, including:
  • an emitter configured to emit a light beam to a target, and the emitter includes a first light source and a second light source;
  • a collector configured to sample the light beam reflected by the target, and the collector includes a first pixel and a second pixel;
  • a processing circuit including a first processing circuit and a second processing circuit, the first processing circuit is electrically connected to the first light source and the first pixel respectively, and the second processing circuit is respectively connected to the second light source and the first pixel The second pixel is electrically connected to;
  • the first processing circuit is used to control the first light source to emit a light beam of the first frequency to the target object, and at the same time control the first pixel to sample the light beam of the first frequency reflected by the target object and outputting a first sampling signal, and calculating a first depth of the target object according to the first sampling signal;
  • the second processing circuit is used to synchronously control the second light source to emit a light beam of a second frequency to the target object, and at the same time control the second pixel to sample the light beam of the second frequency reflected by the target object and output a second sampling signal, and calculate a second depth of the target object according to the second sampling signal;
  • the processing circuit is further configured to fuse the first depth and the second depth to obtain a third depth of the target object.
  • the processing circuit is specifically configured to:
  • the fourth depth and the fifth depth are fused according to a preset fusion formula to obtain a third depth of the target object.
  • the preset conditions are:
  • said fourth depth is equal to said fifth depth
  • the absolute value difference between the fourth depth and the fifth depth is the smallest.
  • processing circuit is also used for:
  • f 1 is the first frequency
  • f 2 is the second frequency
  • f max is the greatest common divisor of the first frequency and the second frequency
  • a 1 and A 2 are the first pixel and The magnitude of the signal received by the second pixel.
  • the preset fusion formula is:
  • D 1 is the fourth depth
  • D 2 is the fifth depth
  • M is the weight coefficient
  • the depth camera further includes an RGB sensor
  • the RGB sensor collects the RGB image of the target; the processing circuit is also used to fuse the first depth and the second depth to obtain the third depth of the target, including: processing the first depth image generating a first high-resolution depth image with the RGB image, the first depth image including the first depth; processing a second depth image and the RGB image to generate a second high-resolution depth image, the second The depth image includes the second depth; fusing the first high-resolution depth image and the second high-resolution depth image to obtain a third high-resolution image of the target, the third high-resolution image includes the Describe the third depth.
  • an embodiment of the present application provides a depth calculation method, which is applied to a depth camera, and the depth camera includes: an emitter, the emitter includes a first light source and a second light source; a collector, the collector It includes a first pixel and a second pixel; a processing circuit, the processing circuit includes a first processing circuit and a second processing circuit, and the first processing circuit is electrically connected to the first light source and the first pixel respectively, so The second processing circuit is electrically connected to the second light source and the second pixel respectively;
  • the methods include:
  • the first processing circuit controls the first light source to emit a light beam of the first frequency to the target, and at the same time controls the first pixel to sample the light beam of the first frequency reflected by the target and output a first sample signal, calculating a first depth of the target object according to the first sampling signal;
  • the second processing circuit synchronously controls the second light source to emit a light beam of the second frequency to the target object, and at the same time controls the second pixel to sample the light beam of the second frequency reflected by the target object and output second sampling signal, calculating a second depth of the target object according to the second sampling signal;
  • the processing circuit fuses the first depth and the second depth to obtain a third depth of the target object.
  • the processing circuit fuses the first depth and the second depth to obtain the third depth of the target, including:
  • the fourth depth and the fifth depth are fused according to a preset fusion formula to obtain a third depth of the target object.
  • an embodiment of the present application provides a photosensitive chip, including a memory, a processor, and a computer program stored in the memory and operable on the processor, and the processor implements the computer program when executing the computer program.
  • an embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the depth calculation method as described in the second aspect are implemented .
  • the embodiments of the present application have the following beneficial effects: the present application uses the first processing circuit to control the first light source to emit a light beam of the first frequency to the target object, and at the same time control the first pixel to sample the first pixel after being reflected by the target object.
  • the second processing circuit synchronously controls the second light source to emit a light beam with a second frequency to the target object, and at the same time controls the second pixel sampling through The beam of the second frequency reflected by the target object outputs a second sampling signal, and the second depth of the target object is calculated according to the second sampling signal; the processing circuit is also used to fuse the first depth and the second depth to obtain the third depth of the target object depth. Simultaneous exposure of optical signals of different frequencies is realized. In terms of timing, the collection of optical signals of different frequencies is expressed as a "parallel" working logic.
  • time interval measures are beneficial to improve the obvious "motion artifact” phenomenon in the current high and low frequency modulation and demodulation signals, and can reduce system power consumption at the same time.
  • Fig. 1 is a schematic structural diagram of a depth calculation system provided by an embodiment of the present application
  • Fig. 2a is a schematic diagram of a pixel array provided by an embodiment of the present application.
  • Fig. 2b is a schematic diagram of another pixel array provided by an embodiment of the present application.
  • FIG. 3 is a schematic structural diagram of a processing circuit provided by an embodiment of the present application.
  • Fig. 4 is a flowchart of steps of a depth calculation method provided by an embodiment of the present application.
  • references to "an embodiment of this application” or “some embodiments” or the like described in the specification of the application mean that a specific feature, structure or characteristic described in connection with the embodiment is included in one or more embodiments of the application.
  • the phrases “in other embodiments”, “an embodiment of the present application”, “other embodiments of the present application”, etc. appearing in different places in this specification do not necessarily all refer to the same embodiment, but rather means “one or more but not all embodiments” unless specifically emphasized otherwise.
  • the terms “including”, “comprising”, “having” and variations thereof mean “including but not limited to”, unless specifically stated otherwise.
  • Fig. 1 is a schematic structural diagram of a depth camera provided by an embodiment of the present application.
  • the depth camera is a TOF depth camera, including a transmitter 11, a collector 12, and a processing circuit 13.
  • the distance between the transmitter 11 and the collector 12 is only It is about a few millimeters, which is much smaller than the distance between the target object 20 and the depth camera 10 .
  • the emitter 11 is used to emit light beams to the target object
  • the collector 12 receives at least part of the light beam reflected by the target object and outputs a sampling signal
  • the processing circuit 13 controls the emitter 11 and the collector 12 simultaneously, and according to the emitted light beam 30 and
  • the time difference or phase difference between the reflected light beams 40 is used to calculate the depth of the target object 20.
  • the embodiment of the present application deals with the background The method of beams of light, interference light, etc. is not limited.
  • emitter 11 includes light source 111 , emitting optics 112 , and driver 113 .
  • the light source 111 can be a light source such as a light-emitting diode, an edge emitting laser, and a vertical cavity surface emitting laser (Vertical Cavity Surface Emitting Laser, VCSEL), and can also be formed by generating multiple VCSEL light sources on a single semiconductor substrate.
  • VCSEL array light source chip, the light beam emitted by the light source can be pulse wave, continuous modulation wave, etc.
  • the selected light source 111 has a wavelength of 780nm to 2526nm, for example, 980nm or 1550nm.
  • the transmitter transmits light beams of various frequencies to the target object 20 , and the embodiment of the present application does not specifically limit the number of types of frequencies emitted by the transmitter.
  • the transmitter includes two light sources 111, which are respectively a first light source and a second light source.
  • the first light source emits a modulated light beam with a first frequency
  • the second light source emits a modulated light beam with a first frequency when the first light source emits a modulated light beam with a first frequency.
  • a modulated light beam of a second frequency is emitted, the first frequency being greater than the second frequency.
  • the transmitter 11 includes three kinds of light sources 111, which are respectively a first light source, a second light source, and a third light source.
  • a modulated beam of a frequency emits a modulated beam of a second frequency while the third light source emits a modulated beam of a third frequency while the first light source emits a modulated beam of the first frequency, and the first frequency is greater than the second frequency and greater than the third frequency .
  • the transmitting optical element 112 includes an optical diffraction element and a lens element, wherein the lens element receives the light beam emitted by the light source and converges it to the optical diffraction element, and the optical diffraction element receives the light beam converged by the lens element and sends it to the target object.
  • Cast beam The emitting optical element can modulate the light emitting area and area of the emitter 11, so that the light beam can be spatially modulated into an ideal surface illumination mode, so that the light beam illumination area and the field of view of the imaging system of the collector 12 overlap as much as possible, and the light beam can be maximized.
  • the utilization rate improves the detection accuracy.
  • the optical diffraction element is used to diffract the beam emitted by the transmitter to form a spot beam, such as a regularly arranged spot beam and an irregularly arranged spot beam. Compared with the flood light, the signal-to-noise ratio calculated by the spot beam is better high.
  • the lens element is used to adjust the beam emitted by the transmitter to achieve functions such as focusing and collimation.
  • the number of lens elements can be designed according to specific conditions; the optical diffraction element and lens element can be independent components or integrated components. There is no limitation here.
  • the driver 113 may be a driver chip, the driver chip may be an integrated circuit (Integrated Circuit, IC) chip, and the IC chip is controlled by the processing circuit 130.
  • the light source 111 Under the control of the IC chip, the light source 111 simultaneously emits a plurality of emission beams of different frequencies, and the emission beams are projected onto the target through the emission optical elements to form illumination spots.
  • the transmitter includes a plurality of drivers 113, for example: includes two drivers 113, respectively a first driver and a second driver, the first driver can be a first IC chip, and the second driver can be a second IC chip .
  • the first IC chip Under the control of the first IC chip (the first IC chip is controlled by the processing circuit 130 ), the first light source emits a light beam of the first frequency, and the light beam of the first frequency is projected onto the target through the emitting optical element to form an illumination spot.
  • the second IC chip synchronously controls the second light source (the second IC chip is controlled by the processing circuit 130 ) to emit a light beam of the second frequency, and the light beam of the second frequency is projected onto the target through the emitting optical element to form an illumination spot.
  • the transmitter includes three drivers 113, which are respectively a first driver, a second driver, and a third driver.
  • the first driver may be a first IC chip
  • the second driver may be a second IC chip
  • the second driver may be a second IC chip.
  • the third driver may be a third IC chip.
  • the first light source emits a light beam of the first frequency
  • the light beam of the first frequency is projected onto the target through the emitting optical element to form an illumination spot.
  • the second IC chip synchronously controls the second light source (the second IC chip is controlled by the processing circuit) to emit light beams of the second frequency outward, and the light beams of the second frequency are projected onto the target through the emitting optical element to form illumination spots.
  • the third IC chip synchronously controls the third light source (the third IC chip is controlled by the processing circuit 130 ) to emit a light beam of a third frequency, and the light beam of the third frequency is projected onto the target through the emitting optical element to form an illumination spot.
  • the collector 12 includes an image sensor 121, a filter element 122, and a receiving optical element 123.
  • the spot beams of various frequencies emitted by the transmitter form a reflected beam after passing through the target object, and the receiving optical element 123 takes the target object
  • the reflected spot beams of various frequencies are imaged onto the image sensor 121, and the filter element 122 is used to filter background noise.
  • the image sensor 121 is an image sensor specially used for optical time-of-flight TOF measurement, and the image sensor 121 may also be integrated into a TOF photosensitive chip specially used for optical time-of-flight TOF measurement.
  • the image sensor 121 may be a charge coupled device (charge coupled device, CCD), a complementary metal-oxide-semiconductor transistor (complementary metal-oxide-semiconductor transistor, CMOS), an avalanche diode (avalanche diode, AD), a single photon avalanche diode (single photon avalanche) diode, SPAD), etc.
  • the size of the array represents the resolution of the depth camera, such as 320 ⁇ 240, etc.
  • the pixels of the image sensor 121 can also be in the form of single point or line array.
  • the image sensor is connected with a signal amplifier, a time-to-digital converter (time-to-digital converter, TDC), a digital-to-analog converter (digital to analog converter, DAC), etc. circuit (not shown in the figure).
  • TDC time-to-digital converter
  • DAC digital to analog converter
  • circuits can be integrated with the pixels as a part of the image sensor or as a part of the processing circuit 130 , which will be collectively regarded as a part of the processing circuit 130 for the convenience of description later.
  • the image sensor 121 includes multiple types of pixels, for example, includes two types of pixels, three types of pixels, and the like.
  • the reflected light beams of various frequencies reflected by the target object 20 are filtered by the receiving optical element 123 and the filter element 122 and then received by each type of pixel in the image sensor 121, and each type of pixel is used to sample the corresponding reflected light beam and output sample signal.
  • each type of pixel in image sensor 121 responds to a single photon in the reflected light beam and outputs a photon signal indicative of the corresponding arrival time of the received photon at each pixel.
  • each type of pixel in the image sensor 121 is used to store and read or discharge charge signals generated by incident photons under the control of corresponding electrodes. The above-mentioned photon signal or charge signal is a specific embodiment of the sampling signal.
  • the image sensor 121 includes two types of pixels, namely first pixels and second pixels, the first pixels are used to collect the reflected light beam of the first frequency and output the first sampling signal, and the second pixels are used to collect the second frequency of the reflected light beam and output a second sampling signal.
  • the first pixels and the second pixels are arranged alternately, which may be that the first pixels and the second pixels in the same row or the same column are arranged alternately, or that the first pixels and the second pixels are alternately arranged in rows or alternate columns.
  • Figure 2a and Figure 2b for the form of the pixel array.
  • Figure 2a is a schematic diagram of a pixel array form provided by an embodiment of the present application. A represents the first pixel, and B represents the second pixel.
  • the image sensor 121 may also be configured to include two pixel arrays, namely a first pixel array and a second pixel array, wherein the first pixel array includes a plurality of first pixels, and the second pixel array includes a plurality of second pixel.
  • each type of pixel includes a plurality of taps for storing and reading or discharging charge signals generated by incident photons
  • the first pixel A includes three taps, respectively
  • the first tap, the second tap, and the third tap of course, in other embodiments, the number of taps included in each type of pixel may also be 4 or more, which is not limited to the above situation.
  • the first tap, the second tap, and the third tap are turned on in turn and only one tap is turned on at the same time, the first tap and the second tap Integral sampling is performed on the reflected light beam of the first frequency, and the third tap performs integral sampling on ambient light, interference light, etc., and outputs the collected first sampling signal data.
  • the second pixel B may also include three taps, which are respectively the fourth tap, the fifth tap, and the sixth tap.
  • the fourth tap and The fifth tap performs integral sampling on the signal of the reflected beam of the second frequency
  • the sixth tap performs integral sampling on ambient light, interference light, etc., and outputs the sampled second sampling signal data.
  • the image sensor 121 includes three types of pixels, namely first pixels, second pixels and third pixels.
  • the reflected beam of the first frequency reflected by the target object 20 is filtered by the receiving optical element 123 and the filter element 122 and then received by the first type of pixels in the image sensor 121 to output a first sampling signal.
  • the reflected beam of the second frequency reflected by the target object 20 is filtered by the receiving optical element 123 and the filter element 122 and then received by the second type of pixels in the image sensor 121 to output the second sampling data.
  • the reflected beam of the third frequency reflected by the object 20 is filtered by the receiving optical element 123 and the filter element 122 and then received by the third type of pixels in the image sensor 121 to output the third sampling data.
  • the receiving optical element 123 includes one or more optical lenses for collecting the light beam reflected from the target and imaging it on the image sensor 121 .
  • the filter element 122 is a filter, and the filter needs to be a narrow-band filter matching the wavelength of the light source 111 to suppress ambient light noise in other bands.
  • the processing circuit 13 may be an independent dedicated circuit, such as an independent circuit with computing capability of the depth camera itself. It may also include a general processing circuit. For example, when the depth camera is integrated into a smart terminal such as a mobile phone, a TV, or a computer, the processing circuit in the terminal may be at least a part of the processing circuit. Please refer to FIG. 3.
  • FIG. 3 is a schematic structural diagram of a processing circuit provided by an embodiment of the present application.
  • the independent dedicated circuit may include, but is not limited to, a processor 131, a memory 132, and stored in the memory 132 and can be stored in the processor. 131 computer program running on it.
  • the processor 131 executes the computer program, it can synchronously provide the periodic modulation signal required when the light source emits a variety of different frequency beams, so that the light source emits a pulsed beam or a continuous modulated wave beam to the target under the control of the modulation signal; It is used to provide the demodulation signal of each pixel of the image sensor at the same time, so that each pixel can collect a variety of different sampling signal data generated by a variety of reflected light beams of different frequencies reflected back by the target under the control of the demodulation signal, and can also be used for According to a variety of different sampling signal data collected by each pixel in the image sensor, calculate the depth of the target object using the reflected beam of each frequency, and use the reflected beam of multiple frequencies to measure the depth of the target object to obtain the depth of the target object Accurate depth. Please refer to the method steps in the following embodiments of the depth calculation method for the specific details of realizing the above functions when the processor executes the computer program.
  • the processing circuit includes a first processing circuit and a second processing circuit, the first processing circuit is electrically connected to the first light source and the first pixel respectively, and the second processing circuit is electrically connected to the second light source and the second pixel respectively , the first processing circuit and the second processing circuit may be independent circuits with computing capabilities, or may be integrated into smart terminals such as mobile phones, televisions, computers, etc., and the processing circuit in the terminal may serve as at least a part of the processing circuit,
  • the first processing circuit and the second processing circuit may include, but are not limited to, a processor 131 , a memory 132 , and a computer program stored in the memory 132 and executable on the processor 131 .
  • the processor 131 in the first processing circuit executes the computer program, it can provide the periodic modulation signal required when the first light source emits a light beam of the first frequency, so that the first light source emits pulses to the target under the control of the modulation signal light beam or continuous modulation wave light beam; it is also used to provide the demodulation signal of the first pixel at the same time, so that the first pixel samples the light beam of the first frequency reflected back by the target under the control of the demodulation signal to output the first sampling signal, and also It can be used to calculate the first depth of the target object according to the first sampling signal.
  • the processor 131 in the second processing circuit executes the computer program, it can provide the second light source to emit the second frequency while the first processing circuit provides the periodic modulation signal required when the first light source emits the emission beam of the first frequency.
  • the periodic modulation signal required when emitting the beam so that the second light source emits a pulsed beam or a continuous modulated wave beam to the target under the control of the modulation signal; it is also used to provide the demodulation signal of the second pixel at the same time, so that the second Under the control of the demodulation signal, the pixels sample the beam of the second frequency reflected by the target to output a second sampling signal, which can also be used to calculate the second depth of the target according to the second sampling signal.
  • the processor 131 in the processing circuit executes the computer program, it can also fuse the first depth of the target object measured by the light beam of the first frequency and the second depth of the target object measured by the light beam of the second frequency to obtain the accurate depth of the target object (third depth).
  • the processing circuit fuses the depths of the corresponding target objects at different frequencies measured by adjacent two types of pixels in each row (column), and then the accurate depth of the fused target object can be obtained.
  • the processing circuit fuses the depths of the corresponding target objects at different frequencies measured by the adjacent two types of pixels in each column, so as to obtain the accurate depth of the fused target object.
  • the fusion method of the embodiment of the present application is not limited to this, for example: in Figure 2a, it is also possible to select adjacent 4 pixels as a group for depth fusion, each group includes two first pixels and two second pixels pixels.
  • each group includes two first pixels and two second pixels pixels.
  • the processing circuit may further include a first processing circuit, a second processing circuit and a third processing circuit, the first processing circuit is electrically connected to the first light source and the first pixel respectively, and the second processing circuit is respectively connected to The second light source is electrically connected to the second pixel, and the third processing circuit is electrically connected to the third light source and the third pixel respectively.
  • the first processing circuit, the second processing circuit and the third processing circuit may be independent circuits with computing power, or It can be integrated into smart terminals such as mobile phones, televisions, computers, etc., and the processing circuit in the terminal can be used as at least a part of the processing circuit.
  • the first processing circuit and the second processing circuit can include, but are not limited to, a processor 131, The memory 132 and computer programs stored in the memory 132 and executable on the processor 131 .
  • the processor 131 in the processing circuit executes the computer program, the method steps in the following embodiments of the depth calculation method are realized.
  • the processor 131 can be a central processing unit (central processing unit, CPU), and the processor 131 can also be other general-purpose processors, a digital signal processor (digital signal processor, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC) ), off-the-shelf programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • a general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.
  • the memory 132 may be an internal storage unit of the processing circuit 13 in some embodiments, such as a hard disk or memory of the processing circuit 13 .
  • the memory 132 may also be an external storage device of the processing circuit 13 in other embodiments, such as a plug-in hard disk equipped on the processing circuit 13, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, flash card (flash card, FD), etc.
  • the memory 132 may also include both an internal storage unit of the processing circuit 13 and an external storage device.
  • the memory 132 is used to store an operating system, application programs, boot loader (bootloader, BL), data, and other programs, such as program codes of computer programs.
  • the memory 132 can also be used to temporarily store data that has been output or will be output.
  • FIG. 1 is only an exemplary description, and cannot be construed as a specific limitation to the present application.
  • it can also include input and output devices, network access devices, and RGB (red green blue) sensors.
  • the RGB sensor is used to collect the RGB image of the target object. The images are fused to obtain a higher resolution depth map.
  • this application provides a depth camera, which synchronously controls multiple light sources in the transmitter to emit beams of different frequencies through the processing circuit, and at the same time controls the collector to sample beams of multiple frequencies reflected by the target to obtain the next frequency of each frequency.
  • the sampling signal data of each frequency calculate the depth of the target when using the beam of each frequency to measure the depth of the target, and use the beam of multiple frequencies to measure the depth of the target to obtain the accurate depth of the target.
  • the working mode is called “parallel", which is beneficial to improve the obvious "motion artifact” phenomenon when using the "serial” working mode.
  • the power consumption of the TOF depth camera can be reduced.
  • FIG. 4 shows a flowchart of the steps of a depth calculation method provided by an embodiment of the present application.
  • the method in FIG. 4 can be executed by the processing circuit in FIG. 1 .
  • the method in Fig. 4 is based on the fusion calculation of the depth of the target object based on two modulation frequencies.
  • the method specifically includes S201 to S203.
  • S201 Control the first light source to emit a light beam of the first frequency to the target, and simultaneously control the first pixel to sample the light beam of the first frequency reflected by the target and output the first sampling signal, and calculate the first sampling signal of the target according to the first sampling signal one depth.
  • the first processing circuit in the processing circuit controls the first light source in the transmitter to emit an emission light beam with a first frequency f 1 to the target object.
  • the first processing circuit in the processing circuit simultaneously controls the first pixel in the collector to sample the reflected light beam of the first frequency reflected by the target and output a first sampling signal.
  • the first processing circuit calculates a first depth (including time-of-flight) value d 1 of the target object according to the first sampling signal.
  • the first pixel of the image sensor includes 4 taps, which are set to be activated at different times within a single cycle time, such as 0 ⁇ T/2, T/2 ⁇ T, T/4 ⁇ 3T/4, 3T respectively /4 to 5T/4 are activated to collect the light beams reflected by the target, and respectively obtain the sampling signal values I 1 , I 2 , I 3 and I 4 .
  • the processing circuit can use the four-step phase shift method, and the first depth value of the target object will be calculated by the following formula:
  • the first processing circuit modulates the beam emitted by the light source into a pulsed beam, and the modulation period is T 1 , and the first processing circuit simultaneously regulates the first pixel to collect the reflected beam.
  • the first pixel of the image sensor includes 2 taps, one of the 2 taps is activated synchronously with the first light source, and receives the light beam within 0 ⁇ T/2 time, and the other receives light within T/2 ⁇ T time
  • Light beams generate sampling signals I 1 and I 2 respectively, based on which the first depth value of the target object can be calculated as:
  • the first processing circuit modulates the beam emitted by the light source into a pulsed beam, and the modulation period is T 1 , and the first processing circuit simultaneously regulates the first pixel to collect the reflected beam.
  • the first pixel of the image sensor includes 3 taps, which are set to be activated at different times within a single cycle time, and respectively collect the background light signal I 0 within 0 ⁇ T/3 time, and in T/3 ⁇ 2T/ 3 Collect the optical signal I 1 , and collect the optical signal I 2 at 2T/3 ⁇ T.
  • collect the light signal I 1 within 0 ⁇ T/3 collect the light signal I 2 from T/3 ⁇ 2T/3, and collect the background light signal I 0 from 2T/3 ⁇ T, based on which the first depth of the target object can be calculated
  • Values are:
  • the light beam emitted by the modulated light source is a pulsed light beam
  • the modulation period is T 1 .
  • the first processing circuit also includes a TDC circuit and a histogram circuit, the TDC circuit receives the photon signal for determining the time-of-flight of the photon from emission to collection, and generates a time code representing the time-of-flight information, and uses the time code to find the time code in the histogram circuit The corresponding time bin, so that the photon count value in the time bin is increased by "1".
  • the histogram of the photon count corresponding to the photon signal can be obtained statistically, and the first position of the beam can be determined according to the peak position of the histogram.
  • a flight time t 1 based on which the first depth value of the target object can be calculated as:
  • S202 Synchronously control the second light source to emit a light beam of the second frequency to the target, and simultaneously control the second pixel to sample the light beam of the second frequency reflected by the target and output a second sampling signal, and calculate the target's weight according to the second sampling signal second depth.
  • the second processing circuit in the processing circuit synchronously controls the second light source in the transmitter to emit the second frequency f2 to the target when the first processing circuit controls the first light source to emit a light beam of the first frequency emission beam.
  • the first frequency is greater than the second frequency.
  • the second processing circuit in the processing circuit simultaneously controls the second pixel in the collector to sample the beam of the second frequency reflected by the target and output a second sampling signal. Then, the second processing circuit calculates the second depth d 2 of the target object according to the second sampling signal.
  • the second processing circuit modulates and demodulates the transmitted optical signal and calculates the depth in the same way as the first processing circuit, and details can be referred to step S201, and will not be repeated here.
  • the processing circuit includes not only the first processing circuit and the second processing circuit, but also other processing circuits, for example, a third processing circuit.
  • the third processing circuit synchronously controls the third light source to emit the emission beam of the third frequency to the target object, and simultaneously controls the third pixel to sample the emission beam reflected by the target object
  • the light beam with the third frequency outputs a third sampling signal, and the third depth of the target is calculated according to the third sampling signal.
  • the method for calculating the depth of the target object is the same as the method for calculating the first depth in S201 and the method for calculating the second depth in S202 , which will not be repeated here.
  • S203 Merge the first depth and the second depth to obtain a third depth of the target object.
  • the method specifically includes: steps S2031 and S2032.
  • S2031 According to preset conditions, based on the first depth and the second depth, determine a fourth depth for measuring the target object by using the reflected light beam of the first frequency and a fifth depth for measuring the target object by using the reflected light beam of the second frequency.
  • d max1 c/2f 1
  • the first processing circuit cannot calculate the accurate depth value of the target object at this time, and a distance aliasing phenomenon occurs, so that the first processing circuit
  • the first depth value calculated by the circuit is less than the actual depth value of the target object, and then the fourth depth value of the target object is calculated according to the first depth value, and the fourth depth value is the depth of the target object measured by the reflected light beam of the first frequency value.
  • the preset first formula is:
  • k 1 is the number of the first period when the actual depth value of the target exceeds the first maximum depth value, and the value of k 1 is an integer constant 0, 1, 2, 3, 4, 5....
  • the possible depth values of the target are 1m, 2.5m, 4m, 5.5m m...
  • the distance between the object and the depth camera is greater than the second maximum depth value, the distance aliasing phenomenon will still occur, so that the second depth value calculated by the second processing circuit is smaller than the actual depth of the object value, then calculate the fifth depth value of the target object according to the second depth value, and the fifth depth value is the depth value of the target object measured by the reflected beam of the second frequency.
  • the preset second formula is:
  • k 2 is the second period number in which the actual depth value of the target exceeds the second maximum depth value, and the value of k 2 is an integer constant 0, 1, 2, 3, 4, 5....
  • the modulation f 2 60MHz
  • d max2 2.5m
  • the second depth value calculated by the second processing circuit is 1.6m
  • the possible depth values of the target are 1.6m, 4.1m, 6.6m....
  • the fourth depth value and the fifth depth value should be the same in theory, and considering that two optical signals with different frequencies are selected to detect the target system error that exists when the frequency is higher, the higher the detection accuracy is, in the present invention, the preset condition is set to be equal to D 1 and D 2 or the absolute value of the difference between D 1 and D 2 is the smallest to determine the fourth depth value and the first depth value Five depth values.
  • the fourth depth and the fifth depth are determined to be equal to the fourth depth and the fifth depth according to a preset first condition.
  • the first frequency is 100MHz
  • the first depth value d1 of the target object calculated by the first processing circuit is 1m
  • the possible depth values of the target object measured by the reflected beam of the first frequency are 1m, 2.5m, 4m, 5.5m
  • the second frequency is 60MHz
  • the second depth value d2 of the target object calculated by the second processing circuit is 1m
  • the possible depth values of the target object measured by the reflected beam of the second frequency are 1m, 3.5m, and 6m respectively , 7.5m, etc.
  • the fourth depth and the fifth depth are determined according to the preset second condition for the minimum absolute value of the difference between the fourth depth and the fifth depth. For example, assuming that the first frequency is 100MHz, and the first depth value d1 of the target object calculated by the first processing circuit is 1m, then the possible depth values of the target object measured by the reflected beam of the first frequency are 1m, 2.5m, and 4m respectively. , 5.5m, etc. Assuming that the second frequency is 60MHz, if the distance d2 of the target object calculated by the second processing circuit is 1.6m, then the possible depth values of the target object measured by the reflected beam of the second frequency are 1.6m, 4.1m, and 6.6m respectively Wait.
  • S2032 Merge the fourth depth and the fifth depth according to a preset fusion formula to obtain a third depth of the target object.
  • the weight coefficients of the fourth depth and fifth depth are mainly affected by the magnitude of the signal received by the pixel. According to the previous description, it can be known that the detection distance of high-frequency signals Small but high detection accuracy, low-frequency signal detection distance is large but low detection accuracy, then the distance from the target to the system can be known according to the amplitude of the received signal, that is, the fourth depth or the fifth depth has a higher weight in the fusion calculation .
  • the formula for calculating the weight is:
  • f max is the greatest common divisor of the first frequency and the second frequency
  • a 1 and A 2 are the amplitudes of the signals received by the first pixel and the second pixel respectively, and in some embodiments, the pixel acquisition signal can be used to output The amplitude size of the sampled signal is characterized.
  • the third depth is calculated according to a preset fusion formula, and the fusion formula is:
  • M is the weight, and the value range is a value between 0-1.
  • the embodiment of the present application uses the above-mentioned depth calculation method to synchronously control multiple light sources in the transmitter to emit beams of different frequencies through the processing circuit, and at the same time control the collector to sample the reflected beams of multiple frequencies reflected by the target object.
  • the reflected beam of each frequency in the reflected beam of each frequency is used to obtain the sampling signal of each frequency, and according to the sampling signal of each frequency, the depth of the target object is calculated using the reflected beam of each frequency, and the fusion of multiple frequencies
  • the reflected light beam measures the depth of the target and obtains the accurate depth of the target.
  • This working mode is called “parallel", which helps to improve the obvious "motion artifact” phenomenon when the current "serial” working mode is used.
  • the power consumption of the TOF depth camera can be reduced.
  • the present application also provides another depth camera, and the depth camera further includes an RGB sensor.
  • the depth camera is configured to simultaneously collect a first depth image and a second depth image of the target field of view, wherein the first depth image is a depth image generated by the TOF depth camera sampling an optical signal of a first frequency reflected back by the target, and the second A depth image includes a first depth; a second depth image is a depth image generated by a TOF camera collecting an optical signal of a second frequency reflected back by an object, and the second depth image includes a second depth.
  • the structure of the TOF depth camera is shown in Figure 1, and will not be repeated here.
  • the RGB sensor is configured to capture RGB images of the target object.
  • the processing circuit is also used to process the first depth image and the RGB image to generate a first high-resolution depth image, and is also used to process the second depth image and the RGB image to generate a second high-resolution depth image, and further fuses the first high-resolution depth image
  • the image and the second high resolution depth image generate a third high resolution image, the third high resolution image including the third depth.
  • the two light sources in the transmitter in the TOF depth camera simultaneously emit the first frequency and the second frequency of the emitted light beam, and at the same time the collector samples the reflected light beams of two different frequencies reflected by the target object, for two different frequencies
  • the first sampling signal and the second sampling signal are obtained from the reflected light beam, and the processing circuit acquires the first depth image and the second depth image of the target object according to the first sampling signal and the second sampling signal.
  • Steps S201 and S202 have explained in detail that the two light sources in the synchronous controller emit two different frequencies of emitted light beams, and at the same time control the collector to sample the reflected light beams of two different frequencies reflected by the target object, for two different frequencies
  • the reflected beams of each frequency in the reflected beams are used to obtain the sampling signals of each frequency, and according to the sampling signals of each frequency, the specific method for determining the depth of the target object when using the reflected beams of each frequency is not described here. Let me repeat.
  • the RGB sensor collects the RGB image of the target object, and fuses the RGB image with the first depth image and the second depth image respectively to obtain a depth image with higher resolution when measuring the target object using the reflected light beam of each frequency.
  • the resolutions of the first depth image and the second depth image are 50*50 respectively
  • the resolutions of the fused first high-resolution depth image and the second high-resolution depth image are 600*400 respectively.
  • the embodiment of the present application does not limit the specific method for the RGB sensor to collect the RGB image of the target.
  • the first depth image and the second depth image are respectively fused with the RGB image through a deep neural network to obtain the first high-resolution depth image and the second high-resolution depth image.
  • the depth images obtained when two reflected light beams of different frequencies are used to measure the target are illustrated as examples.
  • the TOF depth camera can also obtain the third depth image and the fourth depth image of the target at the same time.
  • the first depth image and the second depth image of the target are measured by using the reflected light beams of two different frequencies. Input the first depth image and the RGB image into the deep neural network for fusion, and output the first high-resolution depth image.
  • the deep neural network includes a feature extraction module, a feature fusion module, an encoding module, a decoding module, and a deep fusion module.
  • First use the residual network to design two feature extraction modules to perform feature extraction on the input RGB image and the first depth image respectively, and obtain the feature map of the RGB image and the feature map of the first depth image.
  • Each feature extraction module includes convolution layer, batch normalization layer, and Relu activation function layer.
  • the feature map of the extracted RGB image and the feature map of the first depth image are input into the splicing module for feature fusion to realize splicing, and the first feature map is obtained, wherein, using the splicing module to carry out feature image splicing refers to two feature images.
  • the feature map after splicing contains rich semantics after multiple convolution operations.
  • the feature map of the color image, the feature map of the first depth image, and the first feature map are input into the encoding module and the decoding module.
  • the encoding module performs feature encoding on the input feature map.
  • the encoding module includes multiple residual blocks. Each residual block performs convolution processing on the input feature map at least once, and then passes through the batch normalization layer and the Relu activation function layer. Normalize and nonlinearize the input feature map; similarly, the decoding module includes multiple residual structures, and the decoding module performs feature decoding processing on the input feature image and converts it into a feature map of multiple channels.
  • the depth fusion module fuses the feature maps of multiple channels into the first high-resolution depth image.
  • the depth fusion module includes two convolutional layers, and a batch normalization layer and an activation layer may also be included between the two convolutional layers.
  • the depth value at each pixel in the first high-resolution depth image and the second high-resolution depth image is fused, that is, the first depth and the second depth at the same pixel coordinates are fused , please refer to S203 for the specific fusion method, which will not be repeated here.
  • another depth camera provided by an embodiment of the present application, on the one hand, emits beams of different frequencies by synchronously controlling multiple light sources in the transmitter, and simultaneously controls the collector to sample a variety of different frequencies reflected by the target There is no difference in the timing of the collected signals.
  • This working mode is called “parallel”, which helps to improve the obvious "motion artifact” phenomenon when using the "serial” working mode.
  • the power consumption of the TOF depth camera can be reduced.
  • a higher-resolution depth map is obtained by fusing the low-resolution depth map with the RGB map through a deep neural network.
  • the embodiment of the present application also provides a computer-readable storage medium.
  • the computer-readable storage medium stores a computer program.
  • the steps in the above-mentioned depth calculation method embodiments can be realized.
  • the embodiment of the present application provides a computer program product.
  • the computer program product is run on the mobile terminal, the mobile terminal is executed to implement the steps in the above embodiments of the depth calculation method.
  • the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
  • all or part of the processes in the methods of the above-mentioned embodiments in the present application can be completed by instructing related hardware through computer programs, and the computer programs can be stored in computer-readable storage media, and the computer programs can be processed When executed by the controller, the steps in the above-mentioned various method embodiments can be realized.
  • the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form.
  • the computer-readable medium may at least include: any entity or device capable of carrying computer program codes to the photographing device/terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (random access memory, RAM), electrical carrier signals, telecommunication signals, and software distribution media.
  • ROM read-only memory
  • RAM random access memory
  • electrical carrier signals telecommunication signals
  • software distribution media Such as U disk, mobile hard disk, magnetic disk or optical disk, etc.
  • computer readable media may not be electrical carrier signals and telecommunication signals under legislation and patent practice.
  • a unit described as a separate component may or may not be physically separated, and a component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment of the present application.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Electromagnetism (AREA)
  • General Physics & Mathematics (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Optical Radar Systems And Details Thereof (AREA)
  • Measurement Of Optical Distance (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

一种图像处理领域的深度相机及深度计算方法。该深度相机中的处理器控制第一光源向目标物发射第一频率的光束，同时控制第一像素采样经目标物反射后的第一频率的光束并输出第一采样信号，根据第一采样信号计算目标物的第一深度（S201）；同步控制第二光源向目标物发射第二频率的光束，同时控制第二像素采样经目标物反射后的第二频率的光束并输出第二采样信号，根据第二采样信号计算目标物的第二深度（S202）；还用于融合第一深度和第二深度，得到目标物的第三深度（S203）。实现了不同频率的光信号同时曝光，有利于改善目前高低频调制解调信号时明显的"运动伪影"现象，同时可以降低系统功耗。

Description

一种深度相机及深度计算方法 技术领域
本申请属于图像处理领域，尤其涉及一种深度相机及深度计算方法。
背景技术
现有三维成像技术包括结构光、飞行时间(time of flight，TOF)、双目等，TOF采用飞行时间来实现三维成像，其工作距离随光源调制频率的降低而逐渐增加，比如100MHz的最远工作距离为1.5m，60MHz的最远工作距离为2.5m，20MHz的最远工作距离为7.5m。与之相反，TOF三维成像的精度却随之频率的降低而逐渐变差。因此，为了提高工作距离同时满足高精度的要求，TOF系统设计中往往采用高低频混合的工作模式(即“双频测量”)，比如100MHz&20MHz，其工作距离达到7.5m，而且其精度将高于单独使用20MHz的测距精度。
双频测量的工作模式满足了远距高精度的应用需求。但是在具体应用中，高频与低频双频测量在信号时序上，表现为“串行”完成，比如先采集高频调制解调信号，再采集低频调制解调信号；或者先采集低频调制解调信号，再采集高频调制解调信号。高频与低频的双频信号数据采集后，再进行“双频融合”，最后利用“解混频”算法完成最终的深度测量。
双频信号采集与计算过程中，若目标物处于移动状态时，容易产生“运动伪影”的现象。目前为了降低运动伪影的影响，在实际应用中，采用“尽可能的降低高频信号与低频信号之间的时间间隔的”措施。但是，受限于目前TOF感光芯片设计中的信号读取与传输能力，高频与低频信号之间的时间间隔最多只能降低到几个ms左右，因而导致改善“运动伪影”现象的效果不佳，或者改善能力受到限制。另外，“双频分时曝光”的工作模式，也增加了TOF三维成像系统的功耗。因此，有必要提供新的技术改善“运动伪影”现象，同时降低系统的功耗。
发明内容
本申请实施例提供一种深度相机及深度计算方法，可以改善目前高低频调制解调信号时明显的“运动伪影”现象，同时可以降低系统功耗。
第一方面，本申请实施例提供一种深度相机，包括：
发射器，用于向目标物发射光束，所述发射器包括第一光源和第二光源；
采集器，用于采样经所述目标物反射后的所述光束，所述采集器包括第一像素和第二像素；
处理电路，包括第一处理电路和第二处理电路，所述第一处理电路分别与所述第一光源和所述第一像素电连接，所述第二处理电路分别与所述第二光源和所述第二像素电连接；
所述第一处理电路用于控制所述第一光源向所述目标物发射第一频率的光束，同时控制所述第一像素采样经所述目标物反射后的所述第一频率的光束并输出第一采样信号，根据所述第一采样信号计算所述目标物的第一深度；
所述第二处理电路用于同步控制所述第二光源向所述目标物发射第二频率的光束，同时控制所述第二像素采样经所述目标物反射后的所述第二频率的光束并输出第二采样信号，根据所述第二采样信号计算所述目标物的第二深度；
所述处理电路还用于融合所述第一深度和所述第二深度，得到所述目标物的第三深度。
在第一方面一种可能实现的方式中，所述处理电路具体用于：
根据预设条件，基于所述第一深度和所述第二深度，确定利用所述第一频率的反射光束测量所述目标物的第四深度和利用所述第二频率的反射光束测量所述目标物的第五深度；
依据预设的融合公式融合所述第四深度和所述第五深度，得到所述目标物的第三深度。
其中，所述预设条件为：
所述第四深度与所述第五深度相等；或，
所述第四深度与所述第五深度的绝对值差值最小。
其中，所述处理电路还用于：
计算所述第四深度和所述第五深度相对所述第三深度的权重系数，根据所述权重系数计算所述第三深度；所述权重的计算公式为：
Figure PCTCN2021107947-appb-000001
其中，f 1为所述第一频率，f 2为所述第二频率，f max为所述第一频率和第二频率的最大公约数，A 1和A 2分别为所述第一像素和第二像素接收信号的幅值大小。
其中，所述预设融合公式为：
D＝D 1×M+D 2×(1-M)
其中，D 1为第四深度，D 2为第五深度，M为权重系数。
在一个实施例中，所述深度相机还包括RGB传感器；
所述RGB传感器采集所述目标物的RGB图像；所述处理电路还用于融合所述第一深度 和所述第二深度，得到所述目标物的第三深度，包括：处理第一深度图像和所述RGB图像生成第一高分辨率深度图像，所述第一深度图像包括所述第一深度；处理第二深度图像和所述RGB图像生成第二高分辨率深度图像，所述第二深度图像包括所述第二深度；融合所述第一高分辨率深度图像和第二高分辨率深度图像获得所述目标物的第三高分辨率图像，所述第三高分辨率图像包括所述第三深度。
第二方面，本申请实施例提供一种深度计算方法，应用于深度相机中，所述深度相机包括：发射器，所述发射器包括第一光源和第二光源；采集器，所述采集器包括第一像素和第二像素；处理电路，所述处理电路包括第一处理电路和第二处理电路，所述第一处理电路分别与所述第一光源和所述第一像素电连接，所述第二处理电路分别于所述第二光源和所述第二像素电连接；
所述方法包括：
所述第一处理电路控制所述第一光源向目标物发射第一频率的光束，同时控制所述第一像素采样经所述目标物反射后的所述第一频率的光束并输出第一采样信号，根据所述第一采样信号计算所述目标物的第一深度；
所述第二处理电路同步控制所述第二光源向所述目标物发射第二频率的光束，同时控制所述第二像素采样经所述目标物反射后的所述第二频率的光束并输出第二采样信号，根据所述第二采样信号计算所述目标物的第二深度；
所述处理电路融合所述第一深度和所述第二深度，得到所述目标物的第三深度。
在第二方面一种可实现的方式中，所述处理电路融合所述第一深度和所述第二深度，得到所述目标物的第三深度，包括：
根据预设条件，基于所述第一深度和所述第二深度，确定利用所述第一频率的反射光束测量所述目标物的第四深度和利用所述第二频率的反射光束测量所述目标物的第五深度；
依据预设的融合公式融合所述第四深度和所述第五深度，得到所述目标物的第三深度。
第三方面，本申请实施例提供一种感光芯片，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现如第二方面所述深度计算方法的步骤。
第四方面，本申请实施例提供一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序被处理器执行时实现如第二方面所述深度计算方法的步骤。
本申请实施例与现有技术相比存在的有益效果是：本申请通过第一处理电路控制第一光源向目标物发射第一频率的光束，同时控制第一像素采样经目标物反射后的第一频率的光束并输出第一采样信号，根据第一采样信号计算目标物的第一深度；第二处理电路同步控制第二光源向目标物发射第二频率的光束，同时控制第二像素采样经目标物反射后的第二频率的 光束并输出第二采样信号，根据第二采样信号计算目标物的第二深度；处理电路还用于融合第一深度和第二深度，得到目标物的第三深度。实现了不同频率的光信号同时曝光，在时序上，采集不同频率的光信号表现为“并行”的工作逻辑，相对于“串行”的工作逻辑中通过降低高频信号与低频信号之间的时间间隔的措施，有利于改善目前高低频调制解调信号时明显的“运动伪影”现象，同时可以降低系统功耗。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本申请一实施例提供的深度计算系统的结构示意图；
图2a是本申请一实施例提供的一种像素阵列形式示意图；
图2b是本申请一实施例提供的另一种像素阵列形式示意图；
图3是本申请一实施例提供的处理电路的结构示意图；
图4是本申请一实施例提供的一种深度计算方法步骤流程图；
具体实施方式
以下描述中，为了说明而不是为了限定，提出了诸如特定系统结构、技术之类的具体细节，以便透彻理解本申请实施例。然而，本领域的技术人员应当清楚，在没有这些具体细节的其它实施例中也可以实现本申请。在其它情况中，省略对众所周知的系统、装置、电路以及方法的详细说明，以免不必要的细节妨碍本申请的描述。
应当理解，当在本申请说明书和所附权利要求书中使用时，术语“包括”指示所描述特征、整体、步骤、操作、元素和/或组件的存在，但并不排除一个或多个其它特征、整体、步骤、操作、元素、组件和/或其集合的存在或添加。
还应当理解，在本申请说明书和所附权利要求书中使用的术语“和/或”是指相关联列出的项中的一个或多个的任何组合以及所有可能组合，并且包括这些组合。
在本申请说明书中描述的参考“本申请实施例”或“一些实施例”等意味着在本申请的一个或多个实施例中包括结合该实施例描述的特定特征、结构或特点。由此，在本说明书中的不同之处出现的语句“在另一些实施例中”、“本申请一实施例”、“本申请其他实施例”等不是必然都参考相同的实施例，而是意味着“一个或多个但不是所有的实施例”，除非是 以其他方式另外特别强调。术语“包括”、“包含”、“具有”及它们的变形都意味着“包括但不限于”，除非是以其他方式另外特别强调。
另外，在本申请说明书和所附权利要求书的描述中，术语“第一”、“第二”等仅用于区分描述，而不能理解为指示或暗示相对重要性。
为了说明本申请的技术方案，下面通过具体实施例来进行说明。
图1是本申请一实施例提供的深度相机的结构示意图，该深度相机为TOF深度相机，包括发射器11、采集器12、处理电路13，发射器11和采集器12之间的距离仅为几毫米左右，远小于目标物20与深度相机10之间的距离。其中，发射器11用于向目标物发射光束，采集器12接收至少部分经目标物反射后的光束并输出采样信号，处理电路13同时控制发射器11和采集器12，并根据发射光束30与反射光束40之间的时间差或相位差，计算目标物20的深度，一般地，除了经目标物反射回的反射光束之外还有一些背景光、干扰光等光束，本申请实施例对处理背景光、干扰光等光束的方法不作限制。
在一些实施例中，发射器11包括光源111、发射光学元件112、和驱动器113。其中，光源111可以是发光二级管、边发射激光器、和垂直腔面发射激光器(Vertical Cavity Surface Emitting Laser，VCSEL)等光源，也可以是在单块半导体基底上生成多个VCSEL光源以形成的VCSEL阵列光源芯片，光源所发射的光束可以是脉冲波、连续调制波等。由于在太阳光谱中，近红外波段的比例相较于可见光要低得多，同时硅基材质的探测器探测效率基本能够达到探测的要求，能够最大程度地降低太阳光的干扰，因此本实施例选择的光源111波长为780nm至2526nm，例如可以为980nm或者1550nm。
本申请实施例中，发射器向目标物20发射多种不同频率的光束，本申请实施例不对发射器发射频率的种类数作具体限制。作为一种实现方式，发射器包括两种光源111，分别为第一光源和第二光源，第一光源发射第一频率的调制光束，第二光源在第一光源发射第一频率的调制光束的同时发射第二频率的调制光束，第一频率大于第二频率。
作为另一实现方式，发射器11包括三种光源111，分别为第一光源、第二光源、和第三光源，第一光源发射第一频率的调制光束，第二光源在第一光源发射第一频率的调制光束的同时发射第二频率的调制光束，第三光源在第一光源发射第一频率的调制光束的同时发射第三频率的调制光束，第一频率大于第二频率大于第三频率。
在一个实施例中，发射光学元件112包括光学衍射元件和透镜元件，其中透镜元件接收光源发射的光束并将其汇聚至光学衍射元件，光学衍射元件接收经透镜元件汇聚后的光束并向目标物投射光束。发射光学元件可以对发射器11的光发射区域以及面积进行调制，从而将光束在空间调制成理想的面照明方式，使光束照明区域与采集器12的成像系统的视场尽量重合，最大化光束的利用率，提升探测精度。
光学衍射元件用于对发射器发射的光束进行衍射以形成斑点光束，比如规则排列的斑点光束、不规则排列的斑点光束，与泛光明相比，斑点光束所计算出的飞行时间信噪比更高。
透镜元件用于对发射器发射的光束进行调整，以实现聚焦、准直等功能，透镜元件数量可根据具体情况进行设计；光学衍射元件和透镜元件可为独立元件，亦可为一体化元件，此处不作限制。
在一个实施例中，驱动器113可以为驱动芯片，驱动芯片可以为集成电路(Integrated Circuit，IC)芯片，IC芯片由处理电路130控制。光源111在IC芯片的控制下同时向外发射多种不同频率的发射光束，发射光束经过发射光学元件投射到目标物上形成照明斑点。
作为一实现方式，发射器包括多个驱动器113，例如：包括两个驱动器113，分别为第一驱动器和第二驱动器，第一驱动器可以为第一IC芯片，第二驱动器可以为第二IC芯片。第一光源在第一IC芯片的控制下(第一IC芯片由处理电路130控制)向外发射第一频率的光束，第一频率的光束经过发射光学元件投射到目标物上形成照明斑点。第二IC芯片同步控制第二光源(第二IC芯片由处理电路130控制)向外发射第二频率的光束，第二频率的光束经过发射光学元件投射到目标物上形成照明斑点。
作为另一实现方式，发射器包括三个驱动器113，分别为第一驱动器、第二驱动器、和第三驱动器，第一驱动器可以为第一IC芯片，第二驱动器可以为第二IC芯片，第三驱动器可以为第三IC芯片。第一光源在第一IC芯片的控制下(第一IC芯片由处理电路130控制)向外发射第一频率的光束，第一频率的光束经过发射光学元件投射到目标物上形成照明斑点。第二IC芯片同步控制第二光源(第二IC芯片由处理电路控制)向外发射第二频率的光束，第二频率的光束经过发射光学元件投射到目标物上形成照明斑点。第三IC芯片同步控制第三光源(第三IC芯片由处理电路130控制)向外发射第三频率的光束，第三频率的光束经过发射光学元件投射到目标物上形成照明斑点。
在一些实施例中，采集器12包括图像传感器121、过滤元件122和接收光学元件123等，发射器发射的多种不同频率的斑点光束经过目标物后形成反射光束，接收光学元件123将目标物反射的多种不同频率的斑点光束成像到图像传感器121上，过滤元件122用于过滤背景噪音。
在一些实施例中，图像传感器121是一种专门用于光飞行时间TOF测量的图像传感器，图像传感器121也可以集成到一种专门用于光飞行时间TOF测量的TOF感光芯片中。
图像传感器121可以是电荷耦合元件(charge coupled device，CCD)，互补金属氧化物半导体(complementary metal-oxide-semiconductor transistor，CMOS)，雪崩二极管(avalanche diode，AD)、单光子雪崩二极管(single photon avalanche diode，SPAD)等组成的像素阵列，阵列大小代表着该深度相机的分辨率，比如320×240等，图像传感器121的像素也可以是单 点、线阵等形式。一般地，与图像传感器连接有信号放大器、时数转换器(time-to-digital converter，TDC)、数模转换器(digital to analog converter，DAC)等器件中的一种或多种组成的读出电路(图中未示出)。这些电路既可以与像素整合在一起，作为图像传感器的一部分，也可以作为处理电路130的一部分，后面为便于描述，将统一视作处理电路130的一部分。
在一个实施例中，图像传感器121包括多类像素，例如：包括两类像素、三类像素等。经目标物20反射的多种频率的反射光束经过接收光学元件123和过滤元件122滤光后被图像传感器121中的每类像素所接收，每类像素用于对相应的反射光束进行采样并输出采样信号。在一个实施例中，图像传感器121中的每类像素均对反射光束中的单个光子进行响应并输出指示所接收光子在每个像素处相应到达时间的光子信号。在一个实施例中，图像传感器121中的每类像素均用于在相应电极的控制下存储并读取或者排出由入射光子产生的电荷信号。以上所述的光子信号或电荷信号均为采样信号的一种具体实施例。
作为一实现方式，图像传感器121包括两类像素，分别为第一像素和第二像素，第一像素用于采集第一频率的反射光束并输出第一采样信号，第二像素用于采集第二频率的反射光束并输出第二采样信号。在一个实施例中，第一像素和第二像素是交替排列的，可以是在同一行或同一列内的第一像素和第二像素交替排列，也可以是第一像素和第二像素行交替或列交替的排列形式。像素阵列的形式请参考图2a和图2b，图2a是本申请一实施例提供的一种像素阵列形式示意图，A表示第一像素，B表示第二像素，图2b是本申请一实施例提供的另一种像素阵列形式示意图。当然，像素阵列的形式不限于此，本申请对此不作限制。在一个实施例中，图像传感器121也可以被配置为包括两个像素阵列，分别为第一像素阵列、第二像素阵列，其中第一像素阵列包括多个第一像素，第二像素阵列包括多个第二像素。
作为一实现方式，在图2a或图2b中，每类像素均包括多个抽头，用于存储并读取或者排出由入射光子产生的电荷信号，比如第一像素A包含三个抽头，分别为第一抽头、第二抽头、和第三抽头，当然，在其它实施例中，每类像素包含的抽头数也可以为4个及以上，并不限于上述情形。在一次曝光时间内(通常设置一次曝光时间为发射光束的一个周期长度T)，第一抽头、第二抽头、和第三抽头依次开启且同一时刻只有一个抽头开启，第一抽头和第二抽头对第一频率的反射光束进行积分采样，第三抽头对环境光、干扰光等进行积分采样，并输出采集的第一采样信号数据。同样的，第二像素B也可以包含三个抽头，分别为第四抽头、第五抽头、和第六抽头，在第一类像素对第一频率的反射光束进行处理的同时，第四抽头和第五抽头对第二频率的反射光束的信号进行积分采样，第六抽头对环境光、干扰光等进行积分采样，并输出采样的第二采样信号数据。
作为另一实现方式，图像传感器121包括三类像素，分别为第一像素、第二像素和第三 像素。经目标物20反射的第一频率的反射光束经过接收光学元件123和过滤元件122滤光后被图像传感器121中的第一类像素所接收并输出第一采样信号。经目标物20反射的第二频率的反射光束经过接收光学元件123和过滤元件122滤光后被图像传感器121中的第二类像素所接收并输出第二采样数据。经目标物20反射的第三频率的反射光束经过接收光学元件123和过滤元件122滤光后被图像传感器121中的第三类像素所接收并输出第三采样数据。
在一个实施例中，接收光学元件123包括一个或多个光学透镜，用于收集来自目标物反射的光束，并在图像传感器121上成像。
在一个实施例中，过滤元件122为过滤片，滤光片需选择与光源111的波长相匹配的窄带滤光片，用于抑制其余波段的环境光噪声。
在一个实施例中，处理电路13可以是独立的专用电路，比如深度相机自身具有计算能力的独立电路。也可以包含通用处理电路，比如当该深度相机被集成到如手机、电视、电脑等智能终端中去，终端中的处理电路可以作为该处理电路的至少一部分。请参考图3，图3是本申请一实施例提供的处理电路的结构示意图，该独立的专用电路可包括，但不仅限于，处理器131、存储器132以及存储在存储器132中并可在处理器131上运行的计算机程序。处理器131执行计算机程序时，可以同步提供光源发射多种不同频率的发射光束时所需的周期性调制信号，使光源在调制信号的控制下向目标物发射脉冲光束或连续调制波光束；还用于同时提供图像传感器各像素的解调信号，使各像素在解调信号的控制下采集目标反射回的多种不同频率的反射光束而产生的多种不同的采样信号数据，还可以用于根据图像传感器中各像素采集到的多种不同的采样信号数据，分别计算利用每种频率的反射光束测量目标物的深度，融合利用多种频率的反射光束测量目标物的深度，得到目标物的准确深度。处理器执行计算机程序时实现上述功能的具体细节请参照下述深度计算方法实施例中的方法步骤。
在一个实施例中，处理电路包括第一处理电路和第二处理电路，第一处理电路分别与第一光源和第一像素电连接，第二处理电路分别与第二光源和第二像素电连接，第一处理电路和第二处理电路可以是具有计算能力的独立电路，也可以被集成到如手机、电视、电脑等智能终端中去，终端中的处理电路可以作为该处理电路的至少一部分，第一处理电路和第二处理电路可包括，但不仅限于，处理器131、存储器132以及存储在存储器132中并可在处理器131上运行的计算机程序。
第一处理电路中的处理器131执行计算机程序时，可以提供第一光源发射第一频率的发射光束时所需的周期性调制信号，使第一光源在调制信号的控制下向目标物发射脉冲光束或连续调制波光束；还用于同时提供第一像素的解调信号，使第一像素在解调信号的控制下采样经目标反射回的第一频率的光束而输出第一采样信号，还可以用于根据第一采样信号计算目标物的第一深度。
第二处理电路中的处理器131执行计算机程序时，可以在第一处理电路提供第一光源发射第一频率的发射光束时所需的周期性调制信号的同时，提供第二光源发射第二频率的发射光束时所需的周期性调制信号，使第二光源在调制信号的控制下向目标物发射脉冲光束或连续调制波光束；还用于同时提供第二像素的解调信号，使第二像素在解调信号的控制下采样经目标反射回的第二频率的光束而输出第二采样信号，还可以用于根据第二采样信号计算目标物的第二深度。
处理电路中的处理器131执行计算机程序时，还可以融合利用第一频率的光束测量目标物的第一深度和利用第二频率的光束测量目标物的第二深度，得到目标物的准确深度(第三深度)。例如：在图2a中，处理电路对每一行(列)中相邻的两类像素测量的不同频率下对应的目标物的深度进行融合，即可得到融合后的目标物的准确深度。在图2b中，处理电路对每一列相邻的两类像素测量的不同频率下对应的目标物的深度进行融合，即可得到融合后的目标物的准确深度。当然，本申请实施例的融合方式并不限于此，例如：在图2a中，也可以选择相邻的4个像素为一组进行深度融合，每组包括两个第一像素和两个第二像素。处理电路中的处理器执行计算机程序时实现上述功能的具体细节请参照下述深度计算方法实施例中的方法步骤。
当然，在一些实施例中，处理电路还可以包括第一处理电路、第二处理电路和第三处理电路，第一处理电路分别与第一光源和第一像素电连接，第二处理电路分别与第二光源和第二像素电连接，第三处理电路分别与第三光源和第三像素电连接，第一处理电路、第二处理电路和第三处理电路可以是具有计算能力的独立电路，也可以被集成到如手机、电视、电脑等智能终端中去，终端中的处理电路可以作为该处理电路的至少一部分，第一处理电路和第二处理电路可包括，但不仅限于，处理器131、存储器132以及存储在存储器132中并可在处理器131上运行的计算机程序。处理电路中的处理器131执行计算机程序时，实现下述深度计算方法实施例中的方法步骤。
处理器131可以是中央处理单元(central processing unit，CPU)，该处理器131还可以是其他通用处理器、数字信号处理器(digital signal processor，DSP)、专用集成电路(application specific integrated circuit，ASIC)、现成可编程门阵列(field-programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
存储器132在一些实施例中可以是处理电路13的内部存储单元，例如处理电路13的硬盘或内存。存储器132在另一些实施例中也可以是处理电路13的外部存储设备，例如处理电路13上配备的插接式硬盘，智能存储卡(smart media card，SMC)，安全数字(secure digital，SD)卡，闪存卡(flash card，FD)等。进一步地，存储器132还可以既包括处理电路13的 内部存储单元也包括外部存储设备。存储器132用于存储操作系统、应用程序、引导装载程序(bootloader，BL)、数据以及其他程序等，例如计算机程序的程序代码等。存储器132还可以用于暂时地存储已经输出或者将要输出的数据。
本申请文件不限定深度相机的具体构成，深度相机可以包括比图1所示示例更多或更少的部件，或者组合某些部件，或者不同的部件。图1仅为示例性描述，不能解释为对本申请的具体限制。例如：还可以包括输入输出设备、网络接入设备、和RGB(red green blue)传感器等，RGB传感器用于采集目标物的RGB图像，处理电路将RGB图像与深度相机得到的低分辨率的深度图像进行融合，得到更高分辨率的深度图。
综上，本申请提供深度相机，通过处理电路同步控制发射器中的多种光源发射不同频率的发射光束，同时控制采集器采样经目标物反射的多种频率的光束，得到每种频率的下的采样信号数据，根据每种频率的采样信号数据，计算利用每种频率的光束测量目标物时的深度，融合利用多种频率的光束测量目标物的深度，得到目标物的准确深度，这种工作模式称为“并行”，有利于改善目前采用“串行”工作模式时比较明显的“运动伪影”现象。同时可以降低TOF深度相机的功耗。
图4示出了本申请一实施例提供的一种深度计算方法步骤流程图，图4中的方法可以被图1中的处理电路执行。图4中的方法是基于两种调制频率对目标物的深度进行融合计算。该方法具体包括S201至S203。
S201:控制第一光源向目标物发射第一频率的光束，同时控制第一像素采样经目标物反射后的第一频率的光束并输出第一采样信号，根据第一采样信号计算目标物的第一深度。
具体的，首先，处理电路中的第一处理电路控制发射器中的第一光源向目标物发射第一频率为f 1的发射光束。处理电路中的第一处理电路同时控制采集器中的第一像素采样经目标物反射后的第一频率的反射光束并输出第一采样信号。然后，第一处理电路根据第一采样信号计算目标物的第一深度(包含飞行时间)值d 1。
在一个实施例中，第一处理电路调制光源发射的光束被正弦波或方波调制，调制的周期为T 1，T 1＝1/f 1，第一处理电路同时调控第一像素采集反射光束。假设图像传感器的第一像素中包括4个抽头，被设置成单个周期时间内不同的时间上激活，比如分别在0～T/2、T/2～T、T/4～3T/4、3T/4～5T/4时间内被激活以采集经目标物反射的光束，分别得到采样信号值I 1、I 2、I 3以及I 4。基于此，处理电路可以根据四步相移法，目标物的第一深度值将由下式计算出：
Figure PCTCN2021107947-appb-000002
其中，c为光速。
在一个实施例中，第一处理电路调制光源发射的光束为脉冲光束，调制的周期为T 1，第 一处理电路同时调控第一像素采集反射光束。假设图像传感器的第一像素中包括2个抽头，2个抽头中的一个与第一光源同步被激活，并在0～T/2时间内接收光束，另一个在T/2～T时间内接收光束，分别产生采样信号I 1、I 2，据此可以计算出目标物体的第一深度值为：
Figure PCTCN2021107947-appb-000003
在一个实施例中，第一处理电路调制光源发射的光束为脉冲光束，调制的周期为T 1，第一处理电路同时调控第一像素采集反射光束。假设图像传感器的第一像素中包括3个抽头，被设置成单个周期时间内不同的时间上激活，并分别在0～T/3时间内采集背景光信号I 0，在T/3～2T/3采集光信号I 1，在2T/3～T采集光信号I 2。或者0～T/3时间内采集光信号I 1，T/3～2T/3采集光信号I 2，2T/3～T采集背景光信号I 0，据此可以计算出目标物体的第一深度值为：
Figure PCTCN2021107947-appb-000004
在一个实施例中，调制光源发射的光束为脉冲光束，调制的周期为T 1。假设图像传感器的第一像素为单光子探测器，例如SPAD，每个像素对反射光束中的单个光子进行响应并输出指示所接收光子在每个SPAD处相应到达时间的光子信号(第一采样信号)，第一处理电路还包括TDC电路和直方图电路，TDC电路接收光子信号用于确定光子从发射到采集的飞行时间，并生成表征飞行时间信息的时间码，利用时间码寻找直方图电路中对应的时间bin，使得该时间bin内的光子计数值加“1”，当进行大量重复脉冲探测后，可以统计得到光子信号对应的光子计数的直方图，根据直方图的峰值位置确定光束的第一飞行时间t 1，据此可以计算出目标物体的第一深度值为：
Figure PCTCN2021107947-appb-000005
以上仅是为了更好的说明如何进行深度计算，而列举的几种可能调制和深度计算方式，其他可能的调制方式和深度计算公式也同样适用于本申请提出的融合深度计算方法。
S202:同步控制第二光源向目标物发射第二频率的光束，同时控制第二像素采样经目标物反射后的第二频率的光束并输出第二采样信号，根据第二采样信号计算目标物的第二深度。
具体的，首先，处理电路中的第二处理电路在第一处理电路控制第一光源发射第一频率的发射光束时，同步控制发射器中的第二光源向目标物发射第二频率为f 2的发射光束。优选地，第一频率大于第二频率。处理电路中的第二处理电路同时控制采集器中的第二像素采样经目标物反射后的第二频率的光束并输出第二采样信号。然后，第二处理电路根据第二采样信号计算目标物的第二深度d 2。
在本发明中，第二处理电路对发射光信号的调制解调以及深度计算方式与第一处理电路 均相同，具体可参考步骤S201，在此不再重复说明。
在其它实施例中，处理电路不止包括第一处理电路和第二处理电路，还包括其它处理电路，例如：包括第三处理电路。第三处理电路在第一处理电路控制第一光源发射第一频率的发射光束时，同步控制第三光源向目标物发射第三频率的发射光束，同时控制第三像素采样经目标物反射后的第三频率的光束并输出第三采样信号，根据第三采样信号计算目标物的第三深度。
计算该目标物的深度方法与S201中计算第一深度和S202中计算第二深度的方法相同，此处就不再赘述。
S203:融合第一深度和第二深度，得到目标物的第三深度。
具体的，基于第一深度和第二深度，确定利用第一频率的反射光束测量目标物的第四深度和利用第二频率的反射光束测量目标物的第五深度。该方法具体包括：步骤S2031和S2032。
S2031：根据预设条件，基于第一深度和第二深度，确定利用第一频率的反射光束测量目标物的第四深度和利用第二频率的反射光束测量目标物的第五深度。
具体的，调制第一光源发射第一频率为f 1的光束时，对应的第一处理电路能计算出的第一最大深度值为d max1，其中d max1＝c/2f 1，当目标物距离深度相机的距离大于第一最大深度值时，由于发射光信号具有周期性，此时第一处理电路无法计算出目标物的准确深度值，而产生了一种距离混叠现象，使得第一处理电路计算出的第一深度值小于目标物的实际深度值，则根据第一深度值计算目标物的第四深度值，所述第四深度值为利用第一频率的反射光束测量目标物的深度值。其中，预设第一公式为：
D 1＝d 1+d max1×k 1
其中，k 1为目标实际深度值超过第一最大深度值的第一周期数，k 1取值为整数常数0、1、2、3、4、5…。例如调制f1＝100MHz时，d max1＝1.5m，第一处理电路计算出的第一深度值为1m时，考虑到距离混叠现象，目标物的可能深度值为1m、2.5m、4m、5.5m…。
同样的，调制第二光源发射第二频率为f 2的光束时，对应的第二处理电路能计算出的第二最大深度值为d max2，其中d max1＝c/2f 2，由于第一频率大于第二频率，则第一最大深度值小于第二最大深度值。对于测量的同一个目标物，当目标物距离深度相机的距离大于第二最大深度值时，仍会产生距离混叠现象，使得第二处理电路计算出的第二深度值小于目标物的实际深度值，则根据第二深度值计算目标物的第五深度值，第五深度值为利用第二频率的反射光束测量目标物的深度值。其中，预设第二公式为：
D 2＝d 2+d max2×k 2
其中，k 2为目标实际深度值超过第二最大深度值的第二周期数，k 2取值为整数常数0、1、2、3、4、5…。例如调制f 2＝60MHz时，d max2＝2.5m，第二处理电路计算出的第二深度值为1.6m 时，考虑到距离混叠现象，目标物的可能深度值为1.6m、4.1m、6.6m…。
对于本发明方案来说，选择两种不同频率的光信号探测目标物的深度，则理论上第四深度值与第五深度值应该相同，而考虑到选择两种不同频率的光信号探测目标物时存在的系统误差，比如频率越高探测精度越高，在本发明中，设置预设条件为D 1与D 2相等或者D 1与D 2差值绝对值最小时确定第四深度值和第五深度值。
作为一种实现方式，根据预设第一条件为第四深度与第五深度相等确定第四深度和第五深度。例如第一频率为100MHz，第一处理电路计算出的目标物的第一深度值d 1为1m，则利用第一频率的反射光束测量目标物的可能深度值分别为1m、2.5m、4m、5.5m…。假设第二频率为60MHz，若第二处理电路计算出的目标物的第二深度值d 2为1m，则利用第二频率的反射光束测量目标物的可能深度值分别为1m、3.5m、6m、7.5m等。根据第一预设条件即可确定求得利用第一频率的反射光束测量目标物的第四深度D 1＝1m，求得利用第二频率的反射光束测量的目标物的第五深度D 2＝1m。
作为另一种实现方式，根据预设第二条件为第四深度与第五深度的差值绝对值最小确定第四深度和第五深度。例如假设第一频率为100MHz，第一处理电路计算出的目标物的第一深度值d 1为1m，则利用第一频率的反射光束测量目标物的可能深度值分别为1m、2.5m、4m、5.5m等。假设第二频率为60MHz，若第二处理电路计算出的目标物的距离d 2为1.6m，则利用第二频率的反射光束测量目标物的可能深度值分别为1.6m、4.1m、6.6m等。由此可知，根据第二预设条件可以确定利用第一频率的反射光束测量目标物的第四深度D 1＝4m，利用第二频率的反射光束测量的目标物的第五深度，D 2＝4.1m。
以上数据仅用于举例说明，不作为对本方案的限制。
S2032：依据预设的融合公式融合第四深度和第五深度，得到目标物的第三深度。
在进行融合计算时，还需要计算出第四深度和第五深度相对第三深度的权重系数，权重系数主要受到像素接收信号幅值大小的影响，根据前面的说明可以知道，高频信号探测距离小但探测精度高，低频信号探测距离大但探测精度低，则根据接收信号幅值大小可以知道目标物距离系统的远近，即第四深度还是第五深度在融合计算时所占的权重更高。在一个实施例中，权重的计算公式为：
Figure PCTCN2021107947-appb-000006
其中，f max为第一频率和第二频率的最大公约数，A 1和A 2分别为第一像素和第二像素接收信号的幅值大小，在一些实施例中，可以用像素采集信号输出的采样信号的幅值大小表征。
具体的，依据预设的融合公式计算第三深度，所述融合公式为：
D＝D 1×M+D 2×(1-M)
其中，M为权重，取值范围为0-1之间的数值。
可以理解的是，在一些其他实施例中，还可以采用其他形式的融合公式计算第三深度。
本申请实施例利用上述深度计算方法，通过处理电路同步控制发射器中的多种光源发射不同频率的发射光束，同时控制采集器采样经目标物反射的多种频率的反射光束，针对多种不同频率的反射光束中的每种频率的反射光束，得到每种频率的采样信号，根据每种频率的采样信号，计算利用每种频率的反射光束测量目标物时的深度，融合利用多种频率的反射光束测量目标物时的深度，得到目标物的准确深度，这种工作模式称为“并行”，有利于改善目前采用“串行”工作模式时比较明显的“运动伪影”现象。同时可以降低TOF深度相机的功耗。
本申请还提供另一种深度相机，所述深度相机还包括RGB传感器。深度相机被配置为同时采集目标视场的第一深度图像和第二深度图像，其中，第一深度图像为TOF深度相机采样经目标物反射回的第一频率的光信号生成的深度图像，第一深度图像包括第一深度；第二深度图像为TOF相机采集经目标物反射回的第二频率的光信号生成的深度图像，第二深度图像包括第二深度。TOF深度相机的结构如图1所示，在此不再赘述。RGB传感器被配置为采集目标物的RGB图像。处理电路还用于处理第一深度图像和RGB图像生成第一高分辨率深度图像，还用于处理第二深度图像和RGB图像生成第二高分辨率深度图像，进一步融合第一高分辨率深度图像和第二高分辨率深度图像生成第三高分辨率图像，第三高分辨率图像包括第三深度。
具体的，TOF深度相机中的发射器中的两种光源同时发射第一频率和第二频率的发射光束，同时采集器采样经目标物反射的两种不同频率的反射光束，针对两种不同频率的反射光束中，得到第一采样信号和第二采样信号，处理电路根据第一采样信号和第二采样信号获取目标物的第一深度图像和第二深度图像。步骤S201和步骤S202中已经详细说明了同步控制器中的两种光源发射两种不同频率的发射光束，同时控制采集器采样经目标物反射的两种不同频率的反射光束，针对两种不同频率的反射光束中的每种频率的反射光束，得到每种频率的采样信号，根据每种频率的采样信号，确定利用每种频率的反射光束测量目标物时的深度的具体方法，此处就不再赘述。
RGB传感器采集目标物的RGB图像，将RGB图像分别与第一深度图像和第二深度图像进行融合，得到利用每种频率的反射光束测量目标物时更高分辨率的深度图像。比如，第一深度图像和第二深度图像的分辨率分别为50*50，经过融合后的第一高分辨率深度图像和第二高分辨率深度图像的分辨率分别为600*400。本申请实施例不对RGB传感器采集目标物的 RGB图像的具体方法作出限制。
本申请一实施例中，通过深度神经网络对第一深度图像和第二深度图像分别与RGB图像进行融合获得第一高分辨率深度图像和第二高分辨率深度。本申请实施例以利用两种不同频率的反射光束测量目标物时得到的深度图像进行举例说明，可以理解的是，TOF深度相机也可以同时获得目标物的第三深度图像、第四深度图像。针对两种频率的反射光束时，利用两种不同频率的反射光束测量目标物的第一深度图像和第二深度图像。将第一深度图像和RGB图像输入到深度神经网络中进行融合，输出第一高分辨率深度图像。
在一个实施例中，深度神经网络包括特征提取模块、特征融合模块、编码模块、解码模块、深度融合模块。首先，利用残差网络设计两个特征提取模块分别对输入的RGB图像、第一深度图像进行特征提取，得到RGB图像的特征图与第一深度图像的特征图，每个特征提取模块包括卷积层、批归一化层以及Relu激活函数层。将提取出的RGB图像的特征图与第一深度图像的特征图输入拼接模块进行特征融合以实现拼接，得到第一特征图，其中，利用拼接模块进行特征图像拼接是指将两幅特征图像进行通道维度上的拼接，拼接后的特征图包含了经过多次卷积操作的丰富语义。将彩色图像的特征图、第一深度图像的特征图以及第一特征图输入编码模块和解码模块。编码模块对输入的特征图进行特征编码，编码模块包括多个残差块，每个残差块对输入的特征图进行至少一次的卷积处理，再经过批归一化层以及Relu激活函数层对输入的特征图进行归一化和非线性化处理；同样的，解码模块包括多个残差结构，解码模块对输入的特征图像进行特征解码处理后转化为多个通道的特征图。最后，深度融合模块将多个通道的特征图进行融合成为第一高分辨率深度图像。在一个实施例中，深度融合模块包括两个卷积层，在两个卷积层之间还可以包括一个批归一化层和激活层。
同理，将第二深度图像和RGB图像输入到深度神经网络中进行融合，输出第二高分辨率深度图像。经过与RGB图像的融合处理后，融合第一高分辨率深度图像与第二高分辨率深度图像中每个像素点处的深度值，即将相同像素坐标处的第一深度与第二深度进行融合，具体融合方法请参照S203，此处不再赘述。
综上，本申请一实施例提供的另一种深度相机，一方面，通过同步控制发射器中的多种光源发射不同频率的发射光束，同时控制采集器采样经目标物反射的多种不同频率的反射光束，采集的信号在时序上不存在差异，这种工作模式称为“并行”，有利于改善目前采用“串行”工作模式时比较明显的“运动伪影”现象。同时可以降低TOF深度相机的功耗。另一方面，通过将低分辨率的深度图通过深度神经网络与RGB图融合，得到更高分辨率的深度图。
本申请实施例还提供了一种计算机可读存储介质，计算机可读存储介质存储有计算机程序，计算机程序被处理器执行时实现可实现上述各个深度计算方法实施例中的步骤。
本申请实施例提供了一种计算机程序产品，当计算机程序产品在移动终端上运行时，使 得移动终端执行时实现上述各个深度计算方法实施例中的步骤。
集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请实现上述实施例方法中的全部或部分流程，可以通过计算机程序来指令相关的硬件来完成，的计算机程序可存储于计算机可读存储介质中，该计算机程序在被处理器执行时，可实现上述各个方法实施例的步骤。其中，计算机程序包括计算机程序代码，计算机程序代码可以为源代码形式、对象代码形式、可执行文件或某些中间形式等。计算机可读介质至少可以包括：能够将计算机程序代码携带到拍照装置/终端设备的任何实体或装置、记录介质、计算机存储器、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、电载波信号、电信信号以及软件分发介质。例如U盘、移动硬盘、磁碟或者光盘等。在某些司法管辖区，根据立法和专利实践，计算机可读介质不可以是电载波信号和电信信号。
在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述或记载的部分，可以参见其它实施例的相关描述。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本申请实施例方案的目的。
以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围，均应包含在本申请的保护范围之内。

Claims (10)

  1. 一种深度相机，其特征在于，包括：
    发射器，用于向目标物发射光束，所述发射器包括第一光源和第二光源；
    采集器，用于采样经所述目标物反射后的所述光束，所述采集器包括第一像素和第二像素；
    处理电路，包括第一处理电路和第二处理电路，所述第一处理电路分别与所述第一光源和所述第一像素电连接，所述第二处理电路分别与所述第二光源和所述第二像素电连接；
    所述第一处理电路用于控制所述第一光源向所述目标物发射第一频率的光束，同时控制所述第一像素采样经所述目标物反射后的所述第一频率的光束并输出第一采样信号，根据所述第一采样信号计算所述目标物的第一深度；
    所述第二处理电路用于同步控制所述第二光源向所述目标物发射第二频率的光束，同时控制所述第二像素采样经所述目标物反射后的所述第二频率的光束并输出第二采样信号，根据所述第二采样信号计算所述目标物的第二深度；
    所述处理电路还用于融合所述第一深度和所述第二深度，得到所述目标物的第三深度。
  2. 根据权利要求1所述的深度相机，其特征在于，所述处理电路具体用于：
    根据预设条件，基于所述第一深度和所述第二深度，确定利用所述第一频率的反射光束测量所述目标物的第四深度和利用所述第二频率的反射光束测量所述目标物的第五深度；
    依据预设的融合公式融合所述第四深度和所述第五深度，得到所述目标物的第三深度。
  3. 根据权利要求2所述的深度相机，其特征在于，所述预设条件为：
    所述第四深度与所述第五深度相等；或，
    所述第四深度与所述第五深度的绝对值差值最小。
  4. 根据权利要求2所述的深度相机，其特征在于，所述处理电路还用于：
    计算所述第四深度和所述第五深度相对所述第三深度的权重系数，根据所述权重系数计算所述目标物的第三深度；所述权重系数的计算公式为：
    Figure PCTCN2021107947-appb-100001
    其中，f 1为所述第一频率，f 2为所述第二频率，f max为所述第一频率和第二频率的最大公约数，A 1和A 2分别为所述第一像素和第二像素接收信号的幅值大小。
  5. 根据权利要求4所述的深度相机，所述预设融合公式为：
    D＝D 1×M+D 2×(1-M)
    其中，D 1为第四深度，D 2为第五深度，M为权重系数。
  6. 根据权利要求1所述的深度相机，其特征在于，还包括RGB传感器；
    所述RGB传感器采集所述目标物的RGB图像；
    所述处理电路还用于融合所述第一深度和所述第二深度，得到所述目标物的第三深度，包括：
    处理第一深度图像和所述RGB图像生成第一高分辨率深度图像，所述第一深度图像包括所述第一深度；
    处理第二深度图像和所述RGB图像生成第二高分辨率深度图像，所述第二深度图像包括所述第二深度；
    融合所述第一高分辨率深度图像和第二高分辨率深度图像获得所述目标物的第三高分辨率图像，所述第三高分辨率图像包括所述第三深度。
  7. 一种深度计算方法，应用于深度相机中，其特征在于，所述深度相机包括：发射器，所述发射器包括第一光源和第二光源；采集器，所述采集器包括第一像素和第二像素；处理电路，所述处理电路包括第一处理电路和第二处理电路，所述第一处理电路分别与所述第一光源和所述第一像素电连接，所述第二处理电路分别于所述第二光源和所述第二像素电连接；
    所述方法包括：
    所述第一处理电路控制所述第一光源向目标物发射第一频率的光束，同时控制所述第一像素采样经所述目标物反射后的所述第一频率的光束并输出第一采样信号，根据所述第一采样信号计算所述目标物的第一深度；
    所述第二处理电路同步控制所述第二光源向所述目标物发射第二频率的光束，同时控制所述第二像素采样经所述目标物反射后的所述第二频率的光束并输出第二采样信号，根据所 述第二采样信号计算所述目标物的第二深度；
    所述处理电路融合所述第一深度和所述第二深度，得到所述目标物的第三深度。
  8. 根据权利要求7所述的深度计算方法，其特征在于，所述处理电路融合所述第一深度和所述第二深度，得到所述目标物的第三深度，包括：
    根据预设条件，基于所述第一深度和所述第二深度，确定利用所述第一频率的反射光束测量所述目标物的第四深度和利用所述第二频率的反射光束测量所述目标物的第五深度；
    依据预设的融合公式融合所述第四深度和所述第五深度，得到所述目标物的第三深度。
  9. 一种感光芯片，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，其特征在于，所述处理器执行所述计算机程序时实现如权利要求7至8任一项所述深度计算方法的步骤。
  10. 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现如权利要求7至8任一项所述深度计算方法的步骤。
PCT/CN2021/107947 2021-05-17 2021-07-22 一种深度相机及深度计算方法 WO2022241942A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110535030.2 2021-05-17
CN202110535030.2A CN113325439B (zh) 2021-05-17 2021-05-17 一种深度相机及深度计算方法

Publications (1)

Publication Number Publication Date
WO2022241942A1 true WO2022241942A1 (zh) 2022-11-24

Family

ID=77415758

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/107947 WO2022241942A1 (zh) 2021-05-17 2021-07-22 一种深度相机及深度计算方法

Country Status (2)

Country Link
CN (1) CN113325439B (zh)
WO (1) WO2022241942A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023091179A (ja) * 2021-12-20 2023-06-30 ソニーセミコンダクタソリューションズ株式会社 測距装置、センサ装置
CN114859365A (zh) * 2022-07-05 2022-08-05 武汉市聚芯微电子有限责任公司 一种深度测量方法及装置、存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140168369A1 (en) * 2012-12-19 2014-06-19 Randall T. Crane Single frequency time of flight de-aliasing
DE102017105142A1 (de) * 2017-03-10 2018-09-13 Infineon Technologies Ag Laufzeit-Bilderzeugungsvorrichtungen und ein Verfahren zum Anpassen einer Referenzfrequenz
KR20200039420A (ko) * 2018-10-05 2020-04-16 엘지이노텍 주식회사 깊이 정보를 획득하는 방법 및 카메라 모듈
CN111538024A (zh) * 2020-03-24 2020-08-14 深圳奥比中光科技有限公司 一种滤波ToF深度测量方法及装置
CN111722241A (zh) * 2020-05-18 2020-09-29 深圳奥锐达科技有限公司 一种多线扫描距离测量系统、方法及电子设备
CN111766596A (zh) * 2020-06-04 2020-10-13 深圳奥锐达科技有限公司 一种距离测量方法、系统及计算机可读存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140168369A1 (en) * 2012-12-19 2014-06-19 Randall T. Crane Single frequency time of flight de-aliasing
DE102017105142A1 (de) * 2017-03-10 2018-09-13 Infineon Technologies Ag Laufzeit-Bilderzeugungsvorrichtungen und ein Verfahren zum Anpassen einer Referenzfrequenz
KR20200039420A (ko) * 2018-10-05 2020-04-16 엘지이노텍 주식회사 깊이 정보를 획득하는 방법 및 카메라 모듈
CN111538024A (zh) * 2020-03-24 2020-08-14 深圳奥比中光科技有限公司 一种滤波ToF深度测量方法及装置
CN111722241A (zh) * 2020-05-18 2020-09-29 深圳奥锐达科技有限公司 一种多线扫描距离测量系统、方法及电子设备
CN111766596A (zh) * 2020-06-04 2020-10-13 深圳奥锐达科技有限公司 一种距离测量方法、系统及计算机可读存储介质

Also Published As

Publication number Publication date
CN113325439A (zh) 2021-08-31
CN113325439B (zh) 2023-04-07

Similar Documents

Publication Publication Date Title
CN110596722B (zh) 直方图可调的飞行时间距离测量系统及测量方法
CN110596721B (zh) 双重共享tdc电路的飞行时间距离测量系统及测量方法
WO2021008209A1 (zh) 深度测量装置及距离测量方法
CN110596725B (zh) 基于插值的飞行时间测量方法及测量系统
US11624835B2 (en) Processing of LIDAR images
US9316735B2 (en) Proximity detection apparatus and associated methods having single photon avalanche diodes for determining a quality metric based upon the number of events
CN110596723B (zh) 动态直方图绘制飞行时间距离测量方法及测量系统
CN109343070A (zh) 时间飞行深度相机
WO2021051480A1 (zh) 一种动态直方图绘制飞行时间距离测量方法及测量系统
CN111123289B (zh) 一种深度测量装置及测量方法
WO2022241942A1 (zh) 一种深度相机及深度计算方法
CN110221272B (zh) 时间飞行深度相机及抗干扰的距离测量方法
KR20120105169A (ko) 복수의 거리 픽셀들을 포함하는 3차원 이미지 센서의 구동 방법
CN111045029A (zh) 一种融合的深度测量装置及测量方法
CN112363180A (zh) 一种成像测距传感器、方法、系统及存储介质
CN111708039A (zh) 一种深度测量装置、方法及电子设备
WO2022011974A1 (zh) 一种距离测量系统、方法及计算机可读存储介质
CN109283508A (zh) 飞行时间计算方法
CN113504542B (zh) 距离测量系统及计算被测物反射率的方法、装置、设备
CN111965659A (zh) 一种距离测量系统、方法及计算机可读存储介质
WO2023279621A1 (zh) 一种itof测距系统及计算被测物反射率的方法
CN114829970A (zh) 飞行时间成像电路、飞行时间成像系统和飞行时间成像方法
WO2023279755A1 (zh) 一种屏蔽测距系统模糊距离值的方法、装置及设备
US20240168161A1 (en) Ranging device, signal processing method thereof, and ranging system
WO2020223982A1 (zh) 时间飞行深度相机及抗干扰的距离测量方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21940391

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21940391

Country of ref document: EP

Kind code of ref document: A1