WO2022110947A1 - 电子装置的控制方法、电子装置及计算机可读存储介质 - Google Patents

电子装置的控制方法、电子装置及计算机可读存储介质 Download PDF

Info

Publication number
WO2022110947A1
WO2022110947A1 PCT/CN2021/115382 CN2021115382W WO2022110947A1 WO 2022110947 A1 WO2022110947 A1 WO 2022110947A1 CN 2021115382 W CN2021115382 W CN 2021115382W WO 2022110947 A1 WO2022110947 A1 WO 2022110947A1
Authority
WO
WIPO (PCT)
Prior art keywords
scene
resolution
processing
photon
depth
Prior art date
Application number
PCT/CN2021/115382
Other languages
English (en)
French (fr)
Inventor
黄毅鑫
Original Assignee
Oppo广东移动通信有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Oppo广东移动通信有限公司 filed Critical Oppo广东移动通信有限公司
Publication of WO2022110947A1 publication Critical patent/WO2022110947A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
    • G01S17/894—3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/491—Details of non-pulse systems
    • G01S7/4912—Receivers
    • G01S7/4913—Circuits for detection, sampling, integration or read-out
    • G01S7/4914—Circuits for detection, sampling, integration or read-out of detector arrays, e.g. charge-transfer gates
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/40—Image enhancement or restoration by the use of histogram techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • the present application relates to the field of ranging, and more particularly, to a control method of an electronic device, an electronic device, and a non-volatile computer-readable storage medium.
  • a ranging system is configured on these electronic devices for ranging, such as a direct time of flight ranging system (Directed Time of Flight, dTOF).
  • dTOF direct time of flight ranging system
  • dTOF direct time of flight
  • High-resolution depth images can be used in applications such as virtual reality (VR) and augmented reality (AR) to enhance the VR and AR application experience.
  • VR virtual reality
  • AR augmented reality
  • Embodiments of the present application provide a control method for an electronic device, an electronic device, and a non-volatile computer-readable storage medium.
  • the control method of an electronic device includes: receiving a first photon signal.
  • the first photon signal is processed to obtain an intensity image of the scene, the intensity image having a first resolution.
  • a second photon signal is received.
  • the second photon signal is processed to obtain a first depth image of the scene, the depth image having a second resolution, the first resolution being greater than the second resolution.
  • the intensity image and the first depth map are fused to obtain a second depth image of the scene, where the second depth image has a third resolution, and the third resolution is greater than the second resolution and less than or equal to the first resolution.
  • the electronic device of the embodiment of the present application includes: a receiver and a processing circuit.
  • the receiver is used for receiving the first photon signal and the second photon signal.
  • the processing circuit is connected with the receiver, and the processing circuit is used for: processing the first photon signal to obtain an intensity image of the scene, the intensity image having a first resolution; processing the second photon signal to obtain a first depth image of the scene, the depth image having the first resolution. two resolutions, the first resolution is greater than the second resolution; and the intensity image and the first depth map are fused to obtain a second depth image of the scene, the second depth image has a third resolution, and the third resolution is greater than The second resolution is less than or equal to the first resolution.
  • the non-volatile computer-readable storage medium of the embodiment of the present application contains a computer program.
  • the computer program when executed by the one or more processing circuits, causes the processing circuits to implement the following control method: receiving the first photon signal.
  • the first photon signal is processed to obtain an intensity image of the scene, the intensity image having a first resolution.
  • a second photon signal is received.
  • the second photon signal is processed to obtain a first depth image of the scene, the depth image having a second resolution, the first resolution being greater than the second resolution.
  • FIG. 1 is a schematic flowchart of a control method of an electronic device according to some embodiments of the present application
  • FIG. 2 is a schematic structural diagram of a ranging system in an electronic device according to some embodiments of the present application.
  • FIG. 3 is a schematic diagram of a scene in which a ranging system performs ranging in an electronic device according to some embodiments of the present application;
  • FIG. 4 is a schematic structural diagram of an electronic device according to some embodiments of the present application.
  • Figure 5 is a statistical graph of count values with respect to time and counts of photon signals for certain embodiments of the present application.
  • FIG. 6 is a schematic flowchart of a control method of an electronic device according to some embodiments of the present application.
  • FIG. 7 is a schematic flowchart of a control method of an electronic device according to some embodiments of the present application.
  • FIG. 8 is a schematic diagram of the SPAD architecture of some embodiments of the present application.
  • FIG. 9 is a schematic diagram of another architecture of the SPAD according to some embodiments of the present application.
  • FIG. 10 is a schematic flowchart of a control method of an electronic device according to some embodiments of the present application.
  • FIG. 11 is a schematic diagram of fusion of an intensity image and a first depth image according to some embodiments of the present application.
  • FIG. 12 is a schematic diagram of a connection state between a computer-readable storage medium and a processor according to some embodiments of the present application.
  • control method of the electronic device 1000 includes:
  • control method further comprises: transmitting light pulses to the scene at a predetermined measurement period.
  • 02 Process the first photon signal to obtain an intensity image of the scene 40, including:
  • the processing of the second photon signal to obtain the first depth image of the scene 40 includes:
  • 041 record the number of times the receiver 20 receives the photon in each time unit
  • the recording of the number of times the receiver 20 accepts the photons in each time unit includes:
  • the number of times the receiver 20 receives the photon in each time unit is recorded, wherein, when the receiver 20 does not receive the photon, the receiver 20 receives the photon in each time unit.
  • the number of photons is recorded as zero.
  • the number of times the receiver 20 receives the photon in the time unit is increased by one, so as to obtain the number of times that the receiver 20 receives the photon.
  • step 05 performing fusion processing on the intensity image and the first depth map to obtain a second depth image of the scene, including:
  • the electronic device 1000 includes a receiver 20 and a processing circuit 30 connected to the receiver 20 .
  • the receiver 20 is used for receiving the first photon signal and the second photon signal.
  • the processing circuit 30 is configured to: process the first photon signal to obtain an intensity image of the scene 40 , the intensity image has a first resolution; process the second photon signal to obtain a first depth of the scene 40 image, the depth image has a second resolution, the first resolution is greater than the second resolution; and the intensity image and the first depth map are fused to obtain the first depth image of the scene 40
  • the electronic device 1000 further includes a transmitter 10 for transmitting light pulses to the scene 40 with a predetermined measurement period.
  • the processing circuit 30 includes a counting circuit 31 and a first processing sub-circuit 32 .
  • the counting circuit 31 is used to record the number of times the receiver 20 receives photons; and the first processing sub-circuit 32 is used to obtain the intensity information of the scene 40 according to the times, and process the intensity information to obtain Intensity image of the scene 40 .
  • one or more of the processing circuits 30 include a counting circuit 31 , a time-to-digital conversion circuit 33 , a histogram circuit 34 and a second processing sub-circuit 35 .
  • the circuit 31 is used to record the number of times the receiver 20 receives photons; the time-to-digital conversion circuit 33 is used to record the time units corresponding to the photons received by the receiver 20; The number of times the receiver 20 receives photons outputs a histogram; the second processing sub-circuit 35 is used for post-processing the histogram to obtain the peak time corresponding to the highest peak of the histogram, and according to the peak time , the starting moment of emitting the photon signal, and the speed of light to obtain depth information of each reflection point in the scene 40 , and to obtain a first depth image of the scene 40 according to the depth information.
  • the counting circuit 31 and the time-to-digital conversion circuit 33 are used to record the number of times the receiver 20 receives photons in each time unit every time an optical pulse is emitted. , wherein, when the receiver 20 does not receive the photons, the number of times the receiver 20 receives photons in each time unit is recorded as zero, and when the receiver 20 receives all the photons in a certain time unit When the photons are received, the number of times the receiver 20 receives photons in the time unit is increased by one, so as to obtain the number of times the receiver 20 receives photons in each time unit after the light pulses are emitted multiple times.
  • the receiver 20 includes a SPAD array 21 , the SPAD array 21 includes a plurality of SPADs, the counting circuits 31 are multiple, and the time-to-digital conversion circuit 33 is multiple, the histogram circuit 34 is multiple, each SPAD is connected to one of the counting circuits 31 and one of the time-to-digital conversion circuits 33, and a predetermined number of the time-to-digital conversion circuits 33 share one circuit.
  • the histogram circuit 34 is described.
  • the receiver 20 includes a SPAD array 21 , the SPAD array 21 includes a plurality of SPADs, a plurality of the counting circuits 31 , and the time-to-digital conversion There are multiple circuits 33 , multiple histogram circuits 34 , each of the SPADs is connected to one of the counting circuits 31 , and a predetermined number of the SPADs share one of the time-to-digital conversion circuits 33 .
  • the digital conversion circuit 33 is connected to one of the histogram circuits 34 .
  • the processing circuit 30 further includes a third processing sub-circuit 36 , and the third processing sub-circuit 36 is used for: acquiring the intensity image in the the first pixel of ; obtain the depth of the second pixel corresponding to the first pixel in the first depth image according to the position of the first pixel in the intensity image; and the depth to obtain a second depth image of the scene 40 .
  • the non-volatile computer-readable storage medium 400 of the embodiment of the present application includes a computer program 401, and when the computer program 401 is executed by one or more processing circuits 30, one or more of the described
  • the processing circuit 30 performs: receiving a first photon signal; processing the first photon signal to obtain an intensity image of the scene, the intensity image having a first resolution; receiving a second photon signal; processing the second photon signal to obtain a first depth image of the scene, the depth image has a second resolution, the first resolution is greater than the second resolution; and the intensity image and the first depth map are fused to A second depth image of the scene is acquired, the second depth image has a third resolution, and the third resolution is greater than the second resolution and less than or equal to the first resolution.
  • computer program 401 when executed by one or more processing circuits 30, causes one or more of said processing circuits 30 to perform: emitting light pulses to the scene at predetermined measurement periods.
  • the one or more processing circuits 30 when the computer program 401 is executed by one or more processing circuits 30, the one or more processing circuits 30 are caused to execute: record the photons received by the receiver. obtaining intensity information of the scene according to the times; and processing the intensity information to obtain an intensity image of the scene.
  • the one or more processing circuits 30 when the computer program 401 is executed by one or more processing circuits 30, the one or more processing circuits 30 are caused to execute: record the reception in each time unit the number of times the receiver receives the photons; output a histogram according to the number of times the receiver receives photons in each time unit; perform post-processing on the histogram to obtain the peak time corresponding to the highest peak of the histogram; according to the obtaining depth information of each reflection point in the scene at the peak time, the start time of emitting the photon signal, and the speed of light; and obtaining the first depth image of the scene according to the depth information.
  • the computer program 401 when executed by one or more processing circuits 30, causes the one or more processing circuits 30 to execute: each time a light pulse is emitted, record the receiver within each time unit The number of times the photon is received, wherein when the receiver does not receive the photon, the number of times the receiver receives the photon in each time unit is recorded as zero, and when the receiver receives the photon in a certain time unit When the photon is detected, the number of times the receiver receives the photon in the time unit is increased by one, so as to obtain the number of times the receiver receives the photon in each time unit after the light pulses are emitted multiple times.
  • the one or more processing circuits 30 when the computer program 401 is executed by one or more processing circuits 30 , the one or more processing circuits 30 are caused to execute: acquiring the first intensity image in the intensity image. a pixel; obtaining the depth of a second pixel in the first depth image corresponding to the first pixel according to the position of the first pixel in the intensity image; and fusing the first pixel and the the depth to obtain a second depth image of the scene.
  • an embodiment of the present application provides a control method of an electronic device 1000 , and the control method of the electronic device 1000 includes:
  • the electronic device 1000 includes a ranging system 100 .
  • the ranging system 100 includes a transmitter 10 , a receiver 20 and a processing circuit 30 .
  • the control method of the electronic device 1000 can be applied to the ranging system 100 , and specifically, the receiver 20 can be used to execute the methods in 01 and 03 .
  • Processing circuit 30 may be used to perform the methods of 02, 04 and 05. That is, the receiver 20 can be used to receive the first photon signal and the second photon signal.
  • the processing circuit 30 may be configured to: process the first photon signal to obtain an intensity image of the scene, the intensity image has a first resolution; process the second photon signal to obtain a first depth image of the scene, the depth image has a second resolution, the the first resolution is greater than the second resolution; and the intensity image and the first depth map are fused to obtain a second depth image of the scene, the second depth image has a third resolution, and the third resolution is greater than the second resolution , and less than or equal to the first resolution.
  • the electronic device 1000 can be, but is not limited to, a mobile phone, a tablet computer, a notebook computer, a smart watch, a game console, a head-mounted display device, etc.
  • a ranging system 100 is often provided to realize ranging, thereby obtaining a depth image in the scene.
  • the ranging system 100 may be a dTOF ranging system, and the measurement technology specifically used by the dTOF ranging system is Time-Correlated Single Photon Counting (TCSPC).
  • TCSPC Time-Correlated Single Photon Counting
  • TCSPC is an effective method for detecting single-photon signals, which can give the time when photons are detected, and can establish the distribution function of photons with time, that is, establish a histogram about time and the number of times the receiver 20 receives photons, and then The distance information of each emission point in the scene 40 is obtained by performing data processing through the histogram.
  • the transmitter 10 when the distance between the electronic device 1000 and each reflection point in the scene 40 needs to be measured, the transmitter 10 emits light pulses to each emission point in the scene 40 with a predetermined measurement period. Specifically, the transmitter 10 uses A surface emitting light source is projected in the scene 40 .
  • a dot-matrix light source is used to project on the scene 40. Since the present application needs to acquire an intensity image with a first resolution, a surface-emitting light source that supports detection at a higher resolution is used.
  • the mainstream surface emitting light source is mainly composed of vertical cavity surface emitting laser (Vertical-Cavity Surface-Emitting Laser, VCSEL) and diffuser (diffuser).
  • a diffuser is installed on the VCSEL, and the multiple parallel laser beams emitted by the VCSEL are diffused into an entire surface light source and projected on the scene 40; in the related art, the quasi-diameter plus diffractive optical element (Diffractive Optical Element, DOE) mode is adopted, and the DOE
  • the laser beam emitted by the VCSEL can be replicated as multiple laser beams projected in a spatial pattern.
  • the use of a surface-emitting light source enables the receiver 10 to receive more photon signals, and the photon signal intensity is sufficient, so as to obtain an intensity image with a first resolution.
  • the receiver 20 includes a single photon avalanche diode (Single Photon Avalanche Diode, SPAD) array 21 .
  • SPAD can correspond to a single photon, that is, when in the working state, as long as a single photon is incident on the active region, a saturated current signal can be generated and transmitted to the corresponding processing circuit.
  • SPAD arrays can accurately detect and record the temporal and spatial information of photons. Compared with other technologies, SPAD can obtain more efficient and accurate imaging in low-light environments.
  • the SPAD array 21 counts the number and time of avalanche breakdown of a single photon in a predetermined measurement period mainly through multiple measurements. It can be considered that in a time unit, when the optical power and the configuration of the SPAD array 21 are completely consistent, the number of times the SPAD is triggered by a photon is proportional to the emissivity of the scene 40 to the detection light. That is, the higher the light intensity reflected by each reflection point in the scene 40 is, the more times the SPAD is triggered by the avalanche breakdown. Therefore, the light intensity distribution of the scene 40 can be obtained by counting the number of times the SPAD is triggered and processed by the processing circuit, so as to obtain the intensity image of the scene 40 with the first resolution.
  • the processing circuit 30 of the embodiment of the present application further includes a counting circuit 31 and a first processing sub-circuit 32 .
  • 02 Process the first photon signal to obtain an intensity image of the scene, including:
  • 023 Process the intensity information to obtain an intensity image of the scene.
  • the counting circuit 31 is used to execute the method of 021
  • the first processing sub-circuit 32 is used to execute the method of 022 and 023 . That is, the counting circuit 31 is used to record the number of times the receiver 10 receives photons.
  • the first processing subcircuit 32 obtains the intensity information of the scene 40 according to the number of times and processes the intensity information to obtain the intensity image of the scene 40 .
  • the ranging system 100 transmits a light pulse to the emission point in the scene 40 , the light pulse is reflected to the receiver 20 through the reflection point, and the receiver 20 transmits the photon signal to the counting circuit 31 , the counting circuit 31 starts to record the number of received photons.
  • the distance from the ranging system 100 to each reflection point in the scene 40 is 10m
  • the distance of a photon from emission to reflection back to the receiver is 20m
  • the time it takes is about 67ns, so if ranging If the ranging range of the system 100 is 10m, a preset cycle time of the histogram statistics must be greater than 67ns.
  • a preset cycle time of the histogram statistics is at least 40ns.
  • the value range of the preset period is [40ns, 100ns]. If the preset period is less than 40ns, photons may not be counted, which reduces the ranging accuracy. If the preset period is greater than 100 ns, the time for one count may be long, resulting in low statistical efficiency, resulting in increased ranging duration and low ranging sensitivity. Therefore, the value range of the preset period is [40ns, 100ns], which can not only count photons, ensure the ranging accuracy, but also shorten the ranging duration.
  • the counting circuit 31 counts the count value of one photon count for each measurement, that is, counts the number of photons received by the receiver 20 . In a measurement, if a photon is detected, the number of times the receiver 30 receives a photon is incremented by 1, and if no photon is detected, the number of times the receiver 20 receives a photon is incremented by 0. After many times of measurement and accumulation of the number of received photons, the counting circuit 31 can finally obtain the counting information shown in the figure c in FIG. 5 .
  • the first subcircuit 32 acquires the intensity information of the scene 40 according to the count information, and acquires an intensity image of the scene 40 according to the intensity information, and the intensity image has a first resolution.
  • the processing circuit 30 of the embodiment of the present application further includes a time-to-digital conversion circuit 33 , a histogram circuit 34 , and a second processing sub-circuit 35 .
  • 04 Process the second photon signal to obtain a first depth image of the scene, including:
  • 041 Record the number of photons received by the receiver in each time unit
  • the counting circuit 31 and the time-to-digital conversion circuit 33 are used to execute the method in 041
  • the histogram circuit 34 is used to execute the method in 042
  • the second processing subcircuit 35 For performing the methods in 043, 044 and 045. That is, the counting circuit 31 is used to record the number of times the receiver 20 receives photons.
  • the time-to-digital conversion circuit 33 is used to record the time unit corresponding to the photons received by the receiver 20 .
  • the histogram circuit 34 is configured to output a histogram according to the number of photons received by the receiver 20 in each time unit.
  • the second processing sub-circuit 35 is used for post-processing the histogram to obtain the peak time corresponding to the highest peak of the histogram; obtain the depth information of each reflection point in the scene according to the peak time, the start time of the emitted photon signal, and the speed of light; and acquiring a first depth image of the scene according to the depth information.
  • the ranging system 100 starts timing when it emits light pulses to the emission points in the scene 40 , and records the moment of the photons in the light pulses emitted by the reflection points in the scene 40 , so that a single photon can be obtained
  • the flight time ⁇ can be obtained by TCSPC technology.
  • a histogram can be built with respect to time and the number of times the receiver 20 receives photons (count values of counts of photon signals) according to the TCSPC technique. The histogram is the integration of the distribution functions of photons over time with multiple preset periods, and can reflect the distribution relationship over time of the number of times the receiver 20 receives photons after one or more times of emitting light beams.
  • the time interval between the two dotted lines shown in Figure b and Figure c in Figure 5 is a time unit.
  • the counting circuit 31 counts the count value of one photon count for each measurement, that is, the number of photons received by the receiver 20 . In a measurement, if a photon is detected, the number of photons received in each time unit by the counting circuit 31 increases by 1, and if no photon is detected, the number of photons received in each time unit increases by 0.
  • the time-to-digital conversion circuit 33 can finally establish the waveform information shown in the figure c in FIG. 5 in the form of a histogram.
  • the histogram circuit 34 can perform smooth filtering on the count value of each unit in the histogram to obtain waveform information about the change of the count value with time.
  • the time unit where the highest peak is located can be reflected in multiple measurements, and the number of photons detected in this time unit is the largest. Therefore, the peak time corresponding to the highest peak of the histogram can more accurately reflect the time when the receiver 20 receives the photon.
  • the second processing sub-circuit 35 can obtain the flight time ⁇ of the photon according to the peak time and the starting time of the emitted photon signal, and then calculate the distance between the ranging system 100 and each reflection point in the scene 40 according to the flight time ⁇ and the speed of light C. distance S, so as to obtain the depth information of each reflection point in the scene 40 . Finally, a first depth image of the scene 40 is obtained according to the depth information, where the first depth image has a second resolution, and the second resolution is smaller than the first resolution.
  • the receiver includes a SPAD array 21, the SPAD array 21 includes a plurality of SPADs, a plurality of counting circuits, a plurality of time-to-digital conversion circuits, and a plurality of histogram circuits, each Each SPAD is connected to a counting circuit and a time-to-digital conversion circuit, and a predetermined number of time-to-digital conversion circuits share a histogram circuit.
  • Each pixel is designed with a corresponding counter and a time-to-digital conversion circuit to record the time of receiving photons.
  • a histogram circuit is shared by n pixels, and a depth information is obtained by calculation.
  • m histogram circuits can be designed, and finally an intensity image of m*n resolution and a first depth image of m resolution can be obtained.
  • the SPAD array 21 includes a plurality of SPADs, a plurality of counting circuits, a plurality of time-to-digital conversion circuits, a plurality of histogram circuits, and each SPAD is connected to a counting circuit, A predetermined number of SPADs share a time-to-digital conversion circuit, and each time-to-digital conversion circuit is connected to a histogram circuit. Compared with connecting one time-to-digital conversion circuit to each SPAD, sharing one TDC for a predetermined number of SPADs can further reduce the volume of the processing circuit 30 .
  • the processing circuit 30 further includes a third processing sub-circuit 36 .
  • step 05 performing fusion processing on the intensity image and the first depth map to obtain a second depth image of the scene, including:
  • the third processing sub-circuit 36 is used to execute the methods in 051 , 052 and 053 . That is, the third processing sub-circuit 36 is used to obtain the first pixel in the intensity image; obtain the depth of the second pixel corresponding to the first pixel in the first depth image according to the position of the first pixel in the intensity image; and The first pixel and depth are fused to obtain a second depth image of the scene.
  • each pixel in the intensity image has the first resolution, but the distance information is roughly obtained through machine learning, and the distance between objects in the image is not accurate enough.
  • the distance information in the first depth image is obtained by the second processing circuit 35, and the distance information is relatively accurate, but the resolution of each pixel in the image is not high.
  • the third processing sub-circuit 36 obtains the second depth image with the third resolution by using the fusion algorithm to restore. Wherein, the third resolution is greater than the second resolution and less than or equal to the first resolution.
  • Each pixel in the second depth image has a third resolution and the distance information of each pixel is relatively accurate, so that a higher resolution depth image is obtained.
  • an embodiment of the present application further provides a non-volatile computer-readable storage medium 400 including a computer program 401 .
  • the computer program 401 When executed by one or more processing circuits 30, the computer program 401 causes the processing circuits to perform the control method of any of the above-described embodiments.
  • the processing circuits 30 are caused to perform the following control methods:
  • the processing circuits 30 are caused to perform the following control methods:
  • 041 Record the number of photons received by the receiver in each time unit

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Electromagnetism (AREA)
  • Optical Radar Systems And Details Thereof (AREA)

Abstract

提供了一种控制方法、电子装置(1000)及可读存储介质(400)。控制方法包括接收第一光子信号(01)；处理第一光子信号获取场景的第一分辨率的强度图像。(02)；接收第二光子信号(03)；处理第二光子信号获取场景的第二分辨率的第一深度图像，第一分辨率大于第二分辨率(04)；对强度图像及第一深度图进行融合处理获取场景的第三分辨率的第二深度图像，第三分辨率大于第二分辨率，且小于等于第一分辨率(05)。

Description

电子装置的控制方法、电子装置及计算机可读存储介质
优先权信息
本申请请求2020年11月25日向中国国家知识产权局提交的、专利申请号为202011346135.5的专利申请优先权和权益，并且通过参展将其全文并入此处。
技术领域
本申请涉及测距领域，更具体而言，涉及一种电子装置的控制方法、电子装置及非易失性计算机可读存储介质。
背景技术
随着技术的发展，手机、平板电脑等电子装置具备了越来越多的功能，例如虚拟现实和增强现实等等。为了实现这些功能，这些电子装置上都会配置测距系统以用于测距，如直接飞行时间测距系统(Directed Time of Flight，dTOF)。为了获取更好的深度图像信息，(Directed Time of Flight，dTOF)技术被引入到电子设备的成像系统中。高分辨率的深度图像可在虚拟现实VR和增强现实AR等应用中应用，提升VR和AR应用体验。
发明内容
本申请实施方式提供一种电子装置的控制方法、电子装置及非易失性计算机可读存储介质。
本申请实施方式的电子装置的控制方法包括：接收第一光子信号。处理第一光子信号以获取场景的强度图像，强度图像具有第一分辨率。接收第二光子信号。处理第二光子信号以获取场景的第一深度图像，深度图像具有第二分辨率，第一分辨率大于第二分辨率。对强度图像及第一深度图进行融合处理以获取场景的第二深度图像，第二深度图像具有第三分辨率，第三分辨率大于第二分辨率，且小于等于第一分辨率。
本申请实施方式的电子装置包括：接收器及处理电路。接收器用于接收第一光子信号及第二光子信号。处理电路与接收器连接，处理电路用于：处理第一光子信号以获取场景的强度图像，强度图像具有第一分辨率；处理第二光子信号以获取场景的第一深度图像，深度图像具有第二分辨率，第一分辨率大于第二分辨率；及对强度图像及第一深度图进行融合处理以获取场景的第二深度图像，第二深度图像具有第三分辨率，第三分辨率大于第二分辨率，且小于等于第一分辨率。
本申请实施方式的非易失性计算机可读存储介质包含有计算机程序。当计算机程序被一个或多个处理电路执行时，使得处理电路实现如下控制方法：接收第一光子信号。处理第一光子信号以获取场景的强度图像，强度图像具有第一分辨率。接收第二光子信号。处理第二光子信号以获取场景的第一深度图像，深度图像具有第二分辨率，第一分辨率大于第二分辨率。及对强度图像及第一深度图进行融合处理以获取场景的第二深度图像，第二深度图像具有第三分辨率，第三分辨率大于第二分辨率，且小于等第一分辨率。
本申请的附加方面和优点将在下面的描述中部分给出，部分将从下面的描述中变得明显，或通过本申请的实践了解到。
附图说明
本申请上述的和/或附加的方面和优点从下面结合附图对实施例的描述中将变得明显和容易理解，其中：
图1是本申请某些实施方式电子装置的控制方法的流程示意图；
图2是本申请某些实施方式的电子装置中测距系统的结构示意图；
图3是本申请某些实施方式的电子装置中测距系统进行测距的场景示意图；
图4是本申请某些实施方式的电子装置的结构示意图；
图5是本申请某些实施方式的关于时间和光子信号的计数的计数值的统计图；
图6是本申请某些实施方式的电子装置的控制方法的流程示意图；
图7是本申请某些实施方式的电子装置的控制方法的流程示意图；
图8是本申请某些实施方式的SPAD架构示意图；
图9本申请某些实施方式的SPAD另一架构示意图；
图10是本申请某些实施方式的电子装置的控制方法的流程示意图；
图11是本申请某些实施方式的强度图像及第一深度图像融合示意图；
图12是本申请某些实施方式的计算机可读存储介质和处理器的连接状态示意图。
具体实施方式
下面详细描述本申请的实施例，所述实施例的示例在附图中示出，其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的，旨在用于解释本申请，而不能理解为对本申请的限制。
下文的公开提供了许多不同的实施例或例子用来实现本申请的不同结构。为了简化本申请的公开，下文中对特定例子的部件和设置进行描述。当然，它们仅仅为示例，并且目的不在于限制本申请。此外，本申请可以在不同例子中重复参考数字和/或字母。这种重复是为了简化和清楚的目的，其本身不指示所讨论各种实施例和/或设置之间的关系。此外，本申请提供了的各种特定的工艺和材料的例子，但是本领域普通技术人员可以意识到其他工艺的可应用于性和/或其他材料的使用。
请参阅图1及图4，本申请实施方式的电子装置1000的控制方法包括：
01：接收第一光子信号；
02：处理所述第一光子信号以获取场景40的强度图像，所述强度图像具有第一分辨率；
03：接收第二光子信号；
04：处理所述第二光子信号以获取所述场景40的第一深度图像，所述深度图像具有第二分辨率，所述第一分辨率大于所述第二分辨率；及
05：对所述强度图像及所述第一深度图进行融合处理以获取所述场景40的第二深度图像，所述第二深度图像具有第三分辨率，所述第三分辨率大于所述第二分辨率，且小于等于所述第一分辨率。
在某些实施方式中，控制方法还包括：以预定的测量周期向场景发射光脉冲。
请参阅图3及图6，在某些实施方式中，02：处理所述第一光子信号以获取场景40的强度图像，包括：
021：记录所述接收器10接收光子的次数；
022：根据所述次数获取所述场景的强度信息；及
023：处理所述强度信息以获取所述场景40的强度图像。
请参阅图3及图7，在某些实施方式中，04：所述处理所述第二光子信号以获取所述场景40的第一深度图像，包括：
041：记录各时间单元内所述接收器20接收所述光子的次数；
042：根据各时间单元内所述接收器20接收光子的次数输出直方图；
043：对所述直方图进行后处理，获取所述直方图的最高峰值对应的峰值时刻；
044：根据所述峰值时刻、发射所述光子信号的起始时刻、及光速获取场景内各反射点的深度信息；及
045：根据所述深度信息获取所述场景40的第一深度图像。
请参阅图3，在某些实施方式中，041：所述记录各时间单元内所述接收器20接受所述光子的次数，包括：
每发射一次光脉冲，记录一次各时间单元内所述接收器20接收所述光子的次数，其中，当所述接收器20未接收所述光子时，则各时间单元内所述接收器20接收光子的次数均记录为零，当所述接收器20在某个时间单元内接收到所述光子时，则在时间单元内所述接收器20接收光子的次数加一，以得到发射多次所述光脉冲后各时间单元内所述接收器20接收光子的次数。
请参阅图10，在某些实施方式中，05：所述对所述强度图像及所述第一深度图进行融合处理以获取所述场景的第二深度图像，包括：
051：获取所述强度图像中的第一像素；
052：根据所述第一像素在所述强度图像中的位置，获取所述第一深度图像中的与所述第一像素对应的第二像素的深度；及
053：融合所述第一像素及所述深度以获取所述场景40的第二深度图像。
请参阅图1至图4，在某些实施方式中，电子装置1000包括接收器20及与所述接收器20连接的处理电路30。所述接收器20用于接收第一光子信号及第二光子信号。所述处理电路30用于：处理所述第一光子信号以获取场景40的强度图像，所述强度图像具有第一分辨率；处理所述第二光子信号以获取所述场景40的第一深度图像，所述深度图像具有第二分辨率，所述第一分辨率大于所述第二分辨率；及对所述强度图像及所述第一深度图进行融合处理以获取所述场景40的第二深度图像，所述第二深度图像具有第三分辨率，所述第三分辨率大于所述第二分辨率，且小于等于所述第一分辨率。
请参阅图3及图4，在某些实施方式中，电子装置1000还包括发射器10，所述发射器10用于以预定的测量周期向所述场景40发射光脉冲。
请参阅图2、图3及图6，在某些实施方式中，所述处理电,30包括计数电路31和第一处理子电路32。所述计数电路31用于记录所述接收器20接收光子的次数；及所述第一处理子电路32用于根据所述次数获取所述场景40的强度信息、及处理所述强度信息以获取所述场景40的强度图像。
请参阅图2及图3，在某些实施方式中，一个或多个所述处理电路30包括计数电路31、时间数字转换电路33、直方图电路34和第二处理子电路35，所述计数电路31用于记录所述接收器20接收光子的次数；所述时间数字转换电路33用于记录所述接收器20接收光子对应的时间单元；所述直方图电路34用于根据各时间单元内所述接收器20接收光子的次数输出直方图；所述第二处理子电路35用于对所述直方图进行后处理，获取所述直方图的最高峰值对应的峰值时刻、根据所述峰值时刻、发射所述光子信号的起始时刻、及光速获取场景40内各反射点的深度信息、及根据所述深度信息获取所述场景40的第一深度图像。
请参阅图2，在某些实施方式中，所述计数电路31及所述时间数字转换电路33，用于：每发射一次光脉冲，记录一次各时间单元内所述接收器20接收光子的次数，其中，当所述接收器20未接收所述光子时，则各时间单元内所述接收器20接收光子的次数均记录为零，当所述接收器20在某个时间单元内接收到所述光子时，则在所述时间单元内所述接收器20接收光子的次数加一，以得到发射多次所述光脉冲后各时间单元内所述接收器20接收光子的次数。
请参阅图2及图8，在某些实施方式中，所述接收,20包括SPAD阵列21，所述SPAD阵列21包括多个SPAD，所述计数电路31为多个，所述时间数字转换电路33为多个，所述直方图电路34为多个，每个所述SPAD连接一个所述计数电路31及一个所述时间数字转换电路33，预定数量的所述时间数字转换电路33共用一个所述直方图电路34。
请参阅图2及图9，在某些实施方式中，所述接收器20包括SPAD阵列21，所述SPAD阵列21包括个多个SPAD，所述计数电路31为多个，所述时间数字转换电路33为多个，所述直方图电路34为多个，每个所述SPAD连接一个所述计数电路31，预定数量的所述SPAD共用一个所述时间数字转换电路33，每个所述时间数字转换电路33连接一个所述直方图电路34。
请参阅图2、图3及图10，在某些实施方式中，所述处理电路30还包括第三处理子电,36，所述第三处理子电路36用于：获取所述强度图像中的第一像素；根据所述第一像素在所述强度图像中的位置，获取所述第一深度图像中的与所述第一像素对应的第二像素的深度；及融合所述第一像素及所述深度以获取所述场景40的第二深度图像。
请参阅图1及图12，本申请实施方式的非易失性计算机可读存储介质400包括计算机程序401，当计算机程序401被一个或多个处理电路30执行时，使得一个或多个所述处理电路30执行：接收第一光子信号；处理所述第一光子信号以获取场景的强度图像，所述强度图像具有第一分辨率；接收第二光子信号；处理所述第二光子信号以获取所述场景的第一深度图像，所述深度图像具有第二分辨率，所述第一分辨率大于所述第二分辨率；及对所述强度图像及所述第一深度图进行融合处理以获取所述场景的第二深度图像，所述第二深度图像具有第三分辨率，所述第三分辨率大于所述第二分辨率，且小于等于所述第一分辨率。
在某些实施方式中，当计算机程序401被一个或多个处理电路30执行时，使得一个或多个所述处 理电路30执行：以预定的测量周期向所述场景发射光脉冲。
请参阅图6及图12，在某些实施方式中，当计算机程序401被一个或多个处理电路30执行时，使得一个或多个所述处理电路30执行：记录所述接收器接收光子的次数；根据所述次数获取所述场景的强度信息；及处理所述强度信息以获取所述场景的强度图像。
请参阅图7及图12，在某些实施方式中，当计算机程序401被一个或多个处理电路30执行时，使得一个或多个所述处理电路30执行：记录各时间单元内所述接收器接收所述光子的次数；根据各时间单元内所述接收器接收光子的次数输出直方图；对所述直方图进行后处理，获取所述直方图的最高峰值对应的峰值时刻；根据所述峰值时刻、发射所述光子信号的起始时刻、及光速获取场景内各反射点的深度信息；及根据所述深度信息获取所述场景的第一深度图像。
在某些实施方式中，当计算机程序401被一个或多个处理电路30执行时，使得一个或多个所述处理电路30执行：每发射一次光脉冲，记录一次各时间单元内所述接收器接收光子的次数，其中，当所述接收器未接收所述光子时，则各时间单元内所述接收器接收光子的次数均记录为零，当所述接收器在某个时间单元内接收到所述光子时，则在所述时间单元内所述接收器接收光子的次数加一，以得到发射多次所述光脉冲后各时间单元内所述接收器接收光子的次数。
请参阅图10及图12，在某些实施方式中，当计算机程序401被一个或多个处理电路30执行时，使得一个或多个所述处理电路30执行：获取所述强度图像中的第一像素；根据所述第一像素在所述强度图像中的位置，获取所述第一深度图像中的与所述第一像素对应的第二像素的深度；及融合所述第一像素及所述深度以获取所述场景的第二深度图像。
请参阅图1及图4，本申请实施方式提供一种电子装置1000的控制方法，该电子装置1000的控制方法包括：
01：接收第一光子信号；
02：处理第一光子信号以获取场景的强度图像，强度图像具有第一分辨率；
03：接收第二光子信号；
04：处理第二光子信号以获取场景的第一深度图像，深度图像具有第二分辨率，第一分辨率大于第二分辨率；及
05：对强度图像及第一深度图进行融合处理以获取场景的第二深度图像，第二深度图像具有第三分辨率，第三分辨率大于第二分辨率，且小于等于第一分辨率。
请参阅图2，本申请实施方式的电子装置1000包括测距系统100。测距系统100包括发射器10、接收器20及处理电路30。电子装置1000的控制方法可应用于测距系统100，具体地，接收器20可用于执行01及03中的方法。处理电路30可用于执行02、04及05中的方法。即，接收器20可用于接收第一光子信号及第二光子信号。处理电路30可用于：处理第一光子信号以获取场景的强度图像，强度图像具有第一分辨率；处理第二光子信号以获取场景的第一深度图像，深度图像具有第二分辨率，所述第一分辨率大于第二分辨率；及对强度图像及第一深度图进行融合处理以获取场景的第二深度图像，第二深度图像具有第三分辨率，第三分辨率大于第二分辨率，且小于等于第一分辨率。
电子装置1000可以但不限于是手机、平板电脑、笔记本电脑、智能手表、游戏机、头显设备等，在这些电子设备中，往往具有测距系统100实现测距，从而获取场景中的深度图像。具体地，测距系统100可以是dTOF测距系统，dTOF测距系统具体用到的测量技术是时间相关单光子计数(Time-Correlated Single Photon Counting，TCSPC)。TCSPC是一种探测单光子信号的有效手段，能够给出光子被探测到的时间，并且能建立起光子随时间的分布函数，即建立关于时间和接收器20接收光子的次数的直方图，再通过该直方图进行数据处理获取场景40中各发射点的距离信息。
请参阅图3，当需要测量电子装置1000到场景40中各反射点之间的距离时，发射器10以预定的测量周期向场景40中各发射点发射光脉冲，具体地，发射器10使用面发射光源投射在场景40中。相关技术中采用点阵光源投射在场景40中，由于本申请需要获取具有第一分辨率的强度图像，因此采用支持较高分辨率探测的面发射光源。主流的面发射光源主要由垂直腔面发射激光器(Vertical-Cavity Surface-Emitting Laser，VCSEL)和扩散器(diffuser)构成。具体地，在VCSEL上安装diffuser，将VCSEL发出的多束平行激光扩散为一整面光源投射在场景40中；相关技术中采用准直径加衍射光学元件(Difractive Optical Element，DOE)模式，通过DOE可将VCSEL发出的激光束复制为多个激光束投射 在空间模式中。采用面发射光源使得接收器10能接收到更多光子信号，光子信号强度足够，从而获取具有第一分辨率的强度图像。
请参阅图2，接收器20包括单光子雪崩二极管(Single Photon Avalanche Diode，SPAD)阵列21。SPAD可以相应单个光子，即处于工作状态时，只要有单个光子入射在有源区，即可产生饱和的电流信号从而传递到相应的处理电路。SPAD阵列能够精确地检测并记录光子的时间和空间信息，相比其他技术，SPAD能够在弱光环境下获得更高效更准确的成像。
具体地，SPAD阵列21主要通过多次测量统计单个光子在预定的测量周期内雪崩击穿的次数与时间。可以认为，在一个时间单元内，在光功率及SPAD阵列21配置完全一致的情况下，SPAD被当个光子触发的次数与场景40对探测光的发射率成一定关系的正比。即被场景40中各反射点反射回的光强度越高，则SPAD被雪崩击穿触发的次数也越多。由此，统计SPAD被触发的次数通过处理电路处理就可以得到场景40的光强分布，从而获取场景40具有第一分辨率的强度图像。
请继续参阅图2，本申请实施方式的处理电路30还包括计数电路31及第一处理子电路32。
请参阅图6，在某些实施方式中，02：处理第一光子信号以获取场景的强度图像，包括：
021：记录接收器10接收光子的次数；
022：根据次数获取场景的强度信息；及
023：处理强度信息以获取场景的强度图像。
请参阅图2及图6，在某些实施方式中，计数电路31用于执行021中的方法，第一处理子电路32用于执行022及023中的方法。即，计数电路31用于记录接收器10接收光子的次数。第一处理子电路32根据次数获取场景40的强度信息及处理强度信息以获取场景40的强度图像。
请参阅图3及图5，具体地，测距系统100在向场景40中的发射点发射光脉冲时，光脉冲经反射点反射到接收器20，接收器20将光子信号传送至计数电路31，计数电路31开始记录接收到的光子的次数。假设测距系统100到场景40中各反射点的距离为10m，考虑光速，则一个光子从发射到反射回到接收器一个来回的距离为20m，所花的时间约为67ns，因此如果测距系统100的测距范围为10m，则直方图统计的一个预设周期时间必须大于67ns。考虑目前消费类电子产品的需求，直方图统计的一个预设周期时间至少为40ns。本申请的实施方式中，预设周期的取值范围为[40ns，100ns]。若预设周期小于40ns，则可能统计不到光子，降低测距精度。若预设周期大于100ns，则一次统计的时间可能较长，导致统计效率较低，导致测距时长增加，测距灵敏度较低。因此，预设周期的取值范围为[40ns，100ns]，既能统计到光子，保证测距精度，又能缩短测距时长。请结合图5中的b图，在每次测量中，一般检测到不超过一个光子。计数电路31对每次测量进行一次光子计数的计数值统计，即统计接收器20接收到光子的次数。在一次测量中，如果探测到一个光子，则接收器30接收光子的次数增加1，如果没有探测到光子，则接收器20接收光子的次数增加0。经过多次测量并累计接收到的光子的次数，最终计数电路31能够得到图5中的c图所示的计数信息。第一子电路32根据计数信息获取场景40的强度信息，并根据该强度信息获取场景40的强度图像，该强度图像具有第一分辨率。
请参阅图2，本申请实施方式的处理电路30还包括时间数字转换电路33、直方图电路34、第二处理子电路35。
请参阅图7，在某些实施方式中，04：处理第二光子信号以获取场景的第一深度图像，包括：
041：记录各时间单元内接收器接收光子的次数；
042：根据各时间单元内接收器接收光子的次数输出直方图；
043：对直方图进行后处理，获取直方图的最高峰值对应的峰值时刻；
044：根据峰值时刻、发射光子信号的起始时刻、及光速获取场景内各反射点的深度信息；及
045：根据深度信息获取场景的第一深度图像。
请参阅图2及图7，在某些实施方式中，计数电路31及时间数字转换电路33用于执行041中的方法，直方图电路34用于执行042中的方法，第二处理子电路35用于执行043、044及045中的方法。即，计数电路31用于记录接收器20接收光子的次数。时间数字转换电路33用于记录接收器20接收光子对应的时间单元。直方图电路34用于根据各时间单元内接收器20接收光子的次数输出直方图。第二处理子电路35用于对直方图进行后处理，获取直方图的最高峰值对应的峰值时刻；根据峰值时刻、发射光子信号的起始时刻、及光速获取场景内各反射点的深度信息；及根据深度信息获取场景的第一深度 图像。
请再次参阅图5，测距系统100在向场景40中的发射点发射光脉冲时开始计时，并记录被场景40中各反射点发射回的光脉冲中的光子的时刻，即可获取单个光子从发射器10到达场景40中个反射点，再被各反射点发射至接收器20这段行程经历的飞行时间τ。由于光速C是已知的，所以可以通过计算出测距系统100到物体50之间的距离S。飞行时间τ具体可由TCSPC技术获取。根据TCSPC技术可以建立关于时间和接收器20接收到光子的次数(光子信号的计数的计数值)的直方图。直方图是多个预设周期的光子随时间的分布函数的整合，能够反映经过一次或多次发射光束后，接收器20接收到光子的次数随时间的分布关系。
为了确定检测到的光子的飞行时间τ，还需要将预设周期划分为独立的一系列的时间单元。请结合图5，图5中的b图和c图所示的两条虚线之间的时间间隔即为一个时间单元。同样的，在每次测量中，一般检测到不超过一个光子。计数电路31对每次测量进行一次光子计数的计数值统计，即接收器20接收到的光子的次数。在一次测量中，如果探测到一个光子，计数电路31则在各时间单元内接收到的光子次数增加1，如果没有探测到一个光子，则各时间单元内接收到的光子增加0。经过多次测量并累计接收到的光子的次数，最终时间数字转换电路33能够以直方图的形式建立图5中c图所示的波形信息。在直方图中可能存在一个或多个计数值大于0的时间单元。直方图电路34可以对直方图中各单元的计数值进行平滑滤波获取关于计数值随时间变化的波形信息。其中，最高峰值所在的时间单元能够反映在多次测量中，在该时间单元检测到的光子的次数最多。因此直方图的最高峰值对应的峰值时刻能够较为准确地反映接收器20接收到光子的时刻。第二处理子电路35能够根据峰值时刻及发射光子信号的起始时刻获取光子的飞行时间τ，再根据飞行时间τ和光速C通过计算出测距系统100到场景40中各反射点之间的距离S，从而获取场景40内各反射点的深度信息。最后根据该深度信息获取场景40的第一深度图像，第一深度图像具有第二分辨率，第二分辨率小于第一分辨率。
请参阅8，具体地，在一个实施例中，接收器包括SPAD阵列21，SPAD阵列21包括多个SPAD，计数电路为多个，时间数字转换电路为多个，直方图电路为多个，每个SPAD连接一个计数电路及一个时间数字转换电路，预定数量的时间数字转换电路共用一个直方图电路。每个像素都设计一个对应的计数器及一个时间数字转换电路，用于记录接收光子的时间。n个像素共享一个直方图电路，计算得到一个深度信息，同时可以设计m个直方图电路，最终可以得到一个m*n分辨率的强度图像及m分辨率的第一深度图像。
请参阅图9，在另一个实施方式中，SPAD阵列21包括个多个SPAD，计数电路为多个，时间数字转换电路为多个，直方图电路为多个，每个SPAD连接一个计数电路，预定数量的SPAD共用一个时间数字转换电路，每个时间数字转换电路连接一个直方图电路。相较于每个SPAD连接一个时间数字转换电路而言，预定数量的SPAD共用一个TDC可进一步减小处理电路30的体积。
请再次参阅图2，在某些实施方式中，处理电路30还包括第三处理子电路36。
请参阅图10，在某些实施方式中，05：对强度图像及第一深度图进行融合处理以获取场景的第二深度图像，包括：
051：获取强度图像中的第一像素；
052：根据第一像素在强度图像中的位置，获取第一深度图像中的与第一像素对应的第二像素的深度；及
053：融合第一像素及深度以获取场景的第二深度图像。
请参阅图2及图10，在某些实施方式中，第三处理子电路36用于执行051、052及053中的方法。即，第三处理子电路36用于获取强度图像中的第一像素；根据第一像素在强度图像中的位置，获取第一深度图像中的与第一像素对应的第二像素的深度；及融合第一像素及深度以获取场景的第二深度图像。
请参阅图11，强度图像中各像素均具有第一分辨率，但是距离信息是通过机器学习大致获取的，图像中各物体之间的距离不够准确。第一深度图像中距离信息通过第二处理电路35获取得到的，距离信息较准确，但是图像中各像素分辨率不高。第三处理子电路36通过融合算法采用还原得到具有第三分辨率的第二深度图像。其中，第三分辨率大于第二分辨率，且小于等于第一分辨率。第二深度图像中各像素均具有第三分辨率且各像素的距离信息较为准确，从而得到较高分辨率的深度图像。
请参阅图12，本申请实施方式还提供一种包含计算机程序401的非易失性计算机可读存储介质400。当计算机程序401被一个或多个处理电路30执行时，使得处理电路执行上述任一实施方式的控制方法。
请结合图1，例如，当计算机程序401被一个或多个处理电路30执行时，使得处理电路30执行一下控制方法：
01：接收第一光子信号；
02：处理第一光子信号以获取场景的强度图像，强度图像具有第一分辨率；
03：接收第二光子信号；
04：处理第二光子信号以获取场景的第一深度图像，深度图像具有第二分辨率，第一分辨率大于第二分辨率；及
05：对强度图像及第一深度图进行融合处理以获取场景的第二深度图像，第二深度图像具有第三分辨率，第三分辨率大于第二分辨率，且小于等于第一分辨率。
又例如，当计算机程序401被一个或多个处理电路30执行时，使得处理电路30执行一下控制方法：
01：接收第一光子信号；
021：记录接收器10接收光子的次数；
022：根据次数获取场景的强度信息；及
023：处理强度信息以获取场景的强度图像；
03：接收第二光子信号；
041：记录各时间单元内接收器接收光子的次数；
042：根据各时间单元内接收器接收光子的次数输出直方图；
043：对直方图进行后处理，获取直方图的最高峰值对应的峰值时刻；
051：获取强度图像中的第一像素；
052：根据第一像素在强度图像中的位置，获取第一深度图像中的与第一像素对应的第二像素的深度；及
053：融合第一像素及深度以获取场景的第二深度图像。
在本说明书的描述中，参考术语“实施例”、“示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本申请的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不一定指的是相同的实施例或示例。而且，描述的具体特征、结构、材料或者特点可以在任何的一个或多个实施例或示例中以合适的方式结合。
尽管已经示出和描述了本申请的实施例，本领域的普通技术人员可以理解：在不脱离本申请的原理和宗旨的情况下可以对这些实施例进行多种变化、修改、替换和变型，本申请的范围由权利要求及其等同物限定。

Claims (20)

  1. 一种电子装置的控制方法，其特征在于，包括：
    接收第一光子信号；
    处理所述第一光子信号以获取场景的强度图像，所述强度图像具有第一分辨率；
    接收第二光子信号；
    处理所述第二光子信号以获取所述场景的第一深度图像，所述深度图像具有第二分辨率，所述第一分辨率大于所述第二分辨率；及
    对所述强度图像及所述第一深度图进行融合处理以获取所述场景的第二深度图像，所述第二深度图像具有第三分辨率，所述第三分辨率大于所述第二分辨率，且小于等于所述第一分辨率。
  2. 根据权利要求1所述的控制方法，其特征在于，所述控制方法还包括：以预定的测量周期向所述场景发射光脉冲。
  3. 根据权利要求1所述的控制方法，其特征在于，所述处理所述第一光子信号以获取强度图像，包括：
    记录所述接收器接收光子的次数；
    根据所述次数获取所述场景的强度信息；及
    处理所述强度信息以获取所述场景的强度图像。
  4. 根据权利要求1所述的控制方法，其特征在于，所述处理所述第二光子信号以获取所述场景的第一深度图像，包括：
    记录各时间单元内所述接收器接收所述光子的次数；
    根据各时间单元内所述接收器接收光子的次数输出直方图；
    对所述直方图进行后处理，获取所述直方图的最高峰值对应的峰值时刻；
    根据所述峰值时刻、发射所述光子信号的起始时刻、及光速获取场景内各反射点的深度信息；及
    根据所述深度信息获取所述场景的第一深度图像。
  5. 根据权利要求4所述的控制方法，其特征在于，所述记录各时间单元内所述接收器接收所述光子的次数，包括：
    每发射一次光脉冲，记录一次各时间单元内所述接收器接收光子的次数，其中，当所述接收器未接收所述光子时，则各时间单元内所述接收器接收光子的次数均记录为零，当所述接收器在某个时间单元内接收到所述光子时，则在所述时间单元内所述接收器接收光子的次数加一，以得到发射多次所述光脉冲后各时间单元内所述接收器接收光子的次数。
  6. 根据权利要求1所述的控制方法，其特征在于，所述对所述强度图像及所述第一深度图进行融合处理以获取所述场景的第二深度图像，包括：
    获取所述强度图像中的第一像素；
    根据所述第一像素在所述强度图像中的位置，获取所述第一深度图像中的与所述第一像素对应的第二像素的深度；及
    融合所述第一像素及所述深度以获取所述场景的第二深度图像。
  7. 一种电子装置，其特征在于，包括：
    接收器，用于接收第一光子信号及第二光子信号；及
    与所述接收器连接的处理电路，所述处理电路用于：
    处理所述第一光子信号以获取场景的强度图像，所述强度图像具有第一分辨率；
    处理所述第二光子信号以获取所述场景的第一深度图像，所述深度图像具有第二分辨率，所述第一分辨率大于所述第二分辨率；及
    对所述强度图像及所述第一深度图进行融合处理以获取所述场景的第二深度图像，所述第二深度图像具有第三分辨率，所述第三分辨率大于所述第二分辨率，且小于等于所述第一分辨率。
  8. 根据权利要求7所述的电子装置，其特征在于，所述电子装置还包括发射器，所述发射器用于以预定的测量周期向所述场景发射光脉冲。
  9. 根据权利要求7所述的电子装置，其特征在于，所述处理电路包括：
    计数电路，用于记录所述接收器接收光子的次数；及
    第一处理子电路，用于根据所述次数获取所述场景的强度信息、及处理所述强度信息以获取所述场景的强度图像。
  10. 根据权利要求7所述的电子装置，其特征在于，一个或多个所述处理电路包括：
    计数电路，用于记录所述接收器接收光子的次数；
    时间数字转换电路，用于记录所述接收器接收光子对应的时间单元；
    直方图电路，用于根据各时间单元内所述接收器接收光子的次数输出直方图；
    第二处理子电路，用于对所述直方图进行后处理，获取所述直方图的最高峰值对应的峰值时刻、根据所述峰值时刻、发射所述光子信号的起始时刻、及光速获取场景内各反射点的深度信息、及根据所述深度信息获取所述场景的第一深度图像。
  11. 根据权利要求10所述的电子装置，其特征在于，所述计数电路及所述时间数字转换电路，用于：
    每发射一次光脉冲，记录一次各时间单元内所述接收器接收光子的次数，其中，当所述接收器未接收所述光子时，则各时间单元内所述接收器接收光子的次数均记录为零，当所述接收器在某个时间单元内接收到所述光子时，则在所述时间单元内所述接收器接收光子的次数加一，以得到发射多次所述光脉冲后各时间单元内所述接收器接收光子的次数。
  12. 根据权利要求10所述的电子装置，其特征在于，所述接收器包括SPAD阵列，所述SPAD阵列包括多个SPAD，所述计数电路为多个，所述时间数字转换电路为多个，所述直方图电路为多个，每个所述SPAD连接一个所述计数电路及一个所述时间数字转换电路，预定数量的所述时间数字转换电路共用一个所述直方图电路。
  13. 根据权利要求10所述的电子装置，其特征在于，所述接收器包括SPAD阵列，所述SPAD阵列包括个多个SPAD，所述计数电路为多个，所述时间数字转换电路为多个，所述直方图电路为多个，每个所述SPAD连接一个所述计数电路，预定数量的所述SPAD共用一个所述时间数字转换电路，每个所述时间数字转换电路连接一个所述直方图电路。
  14. 根据权利要求7所述的电子装置，其特征在于，所述处理电路还包括第三处理子电路，所述第三处理子电路用于：
    获取所述强度图像中的第一像素；
    根据所述第一像素在所述强度图像中的位置，获取所述第一深度图像中的与所述第一像素对应的第二像素的深度；及
    融合所述第一像素及所述深度以获取所述场景的第二深度图像。
  15. 一个或多个存储有计算机程序的非易失性计算机可读存储介质，当所述计算机程序被一个或多个处理电路执行时，使得一个或多个所述处理电路执行：
    接收第一光子信号；
    处理所述第一光子信号以获取场景的强度图像，所述强度图像具有第一分辨率；
    接收第二光子信号；
    处理所述第二光子信号以获取所述场景的第一深度图像，所述深度图像具有第二分辨率，所述第一分辨率大于所述第二分辨率；及
    对所述强度图像及所述第一深度图进行融合处理以获取所述场景的第二深度图像，所述第二深度图像具有第三分辨率，所述第三分辨率大于所述第二分辨率，且小于等于所述第一分辨率。
  16. 根据权利要求15所述的非易失性计算机可读存储介质，其特征在于，当所述计算机程序被一个或多个处理电路执行时，使得一个或多个所述处理电路执行：以预定的测量周期向所述场景发射光脉冲。
  17. 根据权利要求15所述的非易失性计算机可读存储介质，其特征在于，当所述计算机程序被一个或多个处理电路执行时，使得一个或多个所述处理电路执行：
    记录所述接收器接收光子的次数；
    根据所述次数获取所述场景的强度信息；及
    处理所述强度信息以获取所述场景的强度图像。
  18. 根据权利要求15所述的非易失性计算机可读存储介质，其特征在于，当所述计算机程序被一个或多个处理电路执行时，使得一个或多个所述处理电路执行：
    记录各时间单元内所述接收器接收所述光子的次数；
    根据各时间单元内所述接收器接收光子的次数输出直方图；
    对所述直方图进行后处理，获取所述直方图的最高峰值对应的峰值时刻；
    根据所述峰值时刻、发射所述光子信号的起始时刻、及光速获取场景内各反射点的深度信息；及
    根据所述深度信息获取所述场景的第一深度图像。
  19. 根据权利要求18所述的非易失性计算机可读存储介质，其特征在于，当所述计算机程序被一个或多个处理电路执行时，使得一个或多个所述处理电路执行：
    每发射一次光脉冲，记录一次各时间单元内所述接收器接收光子的次数，其中，当所述接收器未接收所述光子时，则各时间单元内所述接收器接收光子的次数均记录为零，当所述接收器在某个时间单元内接收到所述光子时，则在所述时间单元内所述接收器接收光子的次数加一，以得到发射多次所述光脉冲后各时间单元内所述接收器接收光子的次数。
  20. 根据权利要求15所述的非易失性计算机可读存储介质，其特征在于，当所述计算机程序被一个或多个处理电路执行时，使得一个或多个所述处理电路执行：
    获取所述强度图像中的第一像素；
    根据所述第一像素在所述强度图像中的位置，获取所述第一深度图像中的与所述第一像素对应的第二像素的深度；及
    融合所述第一像素及所述深度以获取所述场景的第二深度图像。
PCT/CN2021/115382 2020-11-25 2021-08-30 电子装置的控制方法、电子装置及计算机可读存储介质 WO2022110947A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202011346135.5 2020-11-25
CN202011346135.5A CN112346076A (zh) 2020-11-25 2020-11-25 电子装置的控制方法、电子装置及计算机可读存储介质

Publications (1)

Publication Number Publication Date
WO2022110947A1 true WO2022110947A1 (zh) 2022-06-02

Family

ID=74365888

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/115382 WO2022110947A1 (zh) 2020-11-25 2021-08-30 电子装置的控制方法、电子装置及计算机可读存储介质

Country Status (2)

Country Link
CN (1) CN112346076A (zh)
WO (1) WO2022110947A1 (zh)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112346076A (zh) * 2020-11-25 2021-02-09 Oppo（重庆）智能科技有限公司 电子装置的控制方法、电子装置及计算机可读存储介质
CN113050120A (zh) * 2021-03-24 2021-06-29 Oppo广东移动通信有限公司 成像方法、电子装置、控制装置与存储介质

Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170139041A1 (en) * 2015-11-16 2017-05-18 Stmicroelectronics (Grenoble 2) Sas Ranging device with imaging capability
CN106772430A (zh) * 2016-12-30 2017-05-31 南京理工大学 基于多分辨率小波逼近的单像素光子计数三维成像系统及方法
CN107534764A (zh) * 2015-04-30 2018-01-02 深圳市大疆创新科技有限公司 增强图像分辨率的系统及方法
CN107610077A (zh) * 2017-09-11 2018-01-19 广东欧珀移动通信有限公司 图像处理方法和装置、电子装置和计算机可读存储介质
CN108007385A (zh) * 2017-12-31 2018-05-08 中国人民解放军陆军工程大学 一种大视场弹坑表面形貌成像系统及方法
CN207779348U (zh) * 2017-12-31 2018-08-28 中国人民解放军陆军工程大学 一种大视场弹坑表面形貌成像系统
CN110333501A (zh) * 2019-07-12 2019-10-15 深圳奥比中光科技有限公司 深度测量装置及距离测量方法
CN111091592A (zh) * 2018-10-24 2020-05-01 Oppo广东移动通信有限公司 图像处理方法、图像处理装置、电子设备及可读存储介质
CN111812661A (zh) * 2020-06-22 2020-10-23 深圳奥锐达科技有限公司 一种距离测量方法及系统
CN111856497A (zh) * 2020-07-29 2020-10-30 北京深测科技有限公司 一种单光子成像方法和系统
CN112346076A (zh) * 2020-11-25 2021-02-09 Oppo（重庆）智能科技有限公司 电子装置的控制方法、电子装置及计算机可读存储介质

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102006029025A1 (de) * 2006-06-14 2007-12-27 Iris-Gmbh Infrared & Intelligent Sensors Vorrichtung und Verfahren zur Abstandsbestimmung
EP3271747B1 (en) * 2015-03-17 2022-05-25 Cornell University Depth field imaging apparatus, methods, and applications
EP3159711A1 (en) * 2015-10-23 2017-04-26 Xenomatix NV System and method for determining a distance to an object
CN108603933B (zh) * 2016-01-12 2022-07-08 三菱电机株式会社 用于融合具有不同分辨率的传感器输出的系统和方法
CN107818554B (zh) * 2016-09-12 2023-04-21 索尼公司 信息处理设备和信息处理方法
CN107462898B (zh) * 2017-08-08 2019-06-28 中国科学院西安光学精密机械研究所 基于单光子阵列的选通型漫反射绕角成像系统与方法
CN107907885B (zh) * 2017-09-28 2020-03-27 北京华航无线电测量研究所 一种基于单光子计数方法的水下目标探测装置
KR20200110451A (ko) * 2018-02-13 2020-09-23 센스 포토닉스, 인크. 고분해능 장거리 플래시 lidar를 위한 방법들 및 시스템들
US11175404B2 (en) * 2018-11-30 2021-11-16 Nxp B.V. Lidar system and method of operating the lidar system comprising a gating circuit range-gates a receiver based on a range-gating waveform
CN110458778A (zh) * 2019-08-08 2019-11-15 深圳市灵明光子科技有限公司 一种深度图像去噪方法、装置及存储介质
CN110677596A (zh) * 2019-11-04 2020-01-10 深圳市灵明光子科技有限公司 环境光调节装置、方法、图像传感器及电子装置
CN111415310B (zh) * 2020-03-26 2023-06-30 Oppo广东移动通信有限公司 一种图像处理方法及装置、存储介质
CN111708040B (zh) * 2020-06-02 2023-08-11 Oppo广东移动通信有限公司 测距装置、测距方法及电子设备

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107534764A (zh) * 2015-04-30 2018-01-02 深圳市大疆创新科技有限公司 增强图像分辨率的系统及方法
US20170139041A1 (en) * 2015-11-16 2017-05-18 Stmicroelectronics (Grenoble 2) Sas Ranging device with imaging capability
CN106772430A (zh) * 2016-12-30 2017-05-31 南京理工大学 基于多分辨率小波逼近的单像素光子计数三维成像系统及方法
CN107610077A (zh) * 2017-09-11 2018-01-19 广东欧珀移动通信有限公司 图像处理方法和装置、电子装置和计算机可读存储介质
CN108007385A (zh) * 2017-12-31 2018-05-08 中国人民解放军陆军工程大学 一种大视场弹坑表面形貌成像系统及方法
CN207779348U (zh) * 2017-12-31 2018-08-28 中国人民解放军陆军工程大学 一种大视场弹坑表面形貌成像系统
CN111091592A (zh) * 2018-10-24 2020-05-01 Oppo广东移动通信有限公司 图像处理方法、图像处理装置、电子设备及可读存储介质
CN110333501A (zh) * 2019-07-12 2019-10-15 深圳奥比中光科技有限公司 深度测量装置及距离测量方法
CN111812661A (zh) * 2020-06-22 2020-10-23 深圳奥锐达科技有限公司 一种距离测量方法及系统
CN111856497A (zh) * 2020-07-29 2020-10-30 北京深测科技有限公司 一种单光子成像方法和系统
CN112346076A (zh) * 2020-11-25 2021-02-09 Oppo（重庆）智能科技有限公司 电子装置的控制方法、电子装置及计算机可读存储介质

Also Published As

Publication number Publication date
CN112346076A (zh) 2021-02-09

Similar Documents

Publication Publication Date Title
CN110596722B (zh) 直方图可调的飞行时间距离测量系统及测量方法
US11808891B2 (en) Integrated LIDAR illumination power control
JP6483725B2 (ja) 光学的イベントを感知する方法とそのための光学的イベントセンサ、及び距離測定モバイル装置
CN110596725B (zh) 基于插值的飞行时间测量方法及测量系统
CN110596724B (zh) 动态直方图绘制飞行时间距离测量方法及测量系统
CN110596721A (zh) 双重共享tdc电路的飞行时间距离测量系统及测量方法
US10497738B2 (en) First photon correlated time-of-flight sensor
CN110596723B (zh) 动态直方图绘制飞行时间距离测量方法及测量系统
WO2022110947A1 (zh) 电子装置的控制方法、电子装置及计算机可读存储介质
US11029397B2 (en) Correlated time-of-flight sensor
US20190293796A1 (en) Lidar depth measurement systems and methods
Beer et al. SPAD-based flash LiDAR sensor with high ambient light rejection for automotive applications
Arvani et al. TDC sharing in SPAD-based direct time-of-flight 3D imaging applications
KR20220118398A (ko) 다중 리턴 기능을 가진 픽셀을 구비한 가이거 모드 애벌랜치 포토다이어드 기반의 수신기를 포함하는 lidar 시스템
US11579291B2 (en) Optical ranging system having multi-mode operation using short and long pulses
US20220206158A1 (en) Event driven shared memory pixel
JP7450809B2 (ja) Ｄｔｏｆ検知モジュール、端末デバイス、および測距方法
US11470261B2 (en) Three-dimensional distance measuring method and device
CN113009498A (zh) 一种距离测量方法、装置及系统
CN114829970A (zh) 飞行时间成像电路、飞行时间成像系统和飞行时间成像方法
US20230273304A1 (en) Efficient Fault Detection For Lidar Sensors
US20230221439A1 (en) Addressing redundant memory for lidar pixels
US20220163643A1 (en) Lidar and method of fast photon-count integration for lidar
US20230243928A1 (en) Overlapping sub-ranges with power stepping
CN115267743A (zh) 用于确定飞行时间的方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21896451

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21896451

Country of ref document: EP

Kind code of ref document: A1