WO2021120402A1

WO2021120402A1 - Fused depth measurement apparatus and measurement method

Info

Publication number: WO2021120402A1
Application number: PCT/CN2020/077862
Authority: WO
Inventors: 许星
Original assignee: 深圳奥比中光科技有限公司
Priority date: 2019-12-18
Filing date: 2020-03-04
Publication date: 2021-06-24
Also published as: CN111045029A; CN111045029B

Abstract

Disclosed is a fused depth measurement apparatus, comprising a transmitting module which is used for transmitting, to a target object, a spot pattern light beam with the amplitude being modulated in time sequence; a receiving module for receiving the spot pattern light beam reflected by the target object and forming an electrical signal; and a control and processing circuit for receiving the electrical signal and performing calculation to obtain a TOF depth image and a structured light pattern, taking a depth value in the TOF depth image as a reliable point and assigning same to a corresponding pixel position in a structured light depth image, and using the reliable point to correct the structured light depth image to finally obtain a depth image of the target object. The present application has relatively high measurement accuracy on the basis of a TOF depth value; an error of a structured light depth image caused by matching calculation is corrected on the basis of the TOF depth value to obtain a high-accuracy depth image; and in conjunction with the advantage of high resolution of structured light measurement, a high-accuracy, high-resolution, low-power-consumption and miniaturized depth measurement apparatus is achieved.

Description

Fusion depth measuring device and measuring method

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on December 18, 2019, the application number is 201911306106.3, and the invention title is "a fusion depth measurement device and measurement method", the entire content of which is incorporated by reference In this application.

Technical field

This application relates to the field of optical measurement technology, and in particular to a fusion depth measurement device and measurement method.

Background technique

The depth measurement device can be used to obtain the depth image of the object, and can further perform 3D modeling, skeleton extraction, face recognition, etc., and has a very wide range of applications in the fields of 3D measurement and human-computer interaction. The current depth measurement technology mainly includes TOF ranging technology and structured light ranging technology.

The full name of TOF is Time-of-Flight, that is, time of flight. TOF ranging technology is a technology that achieves precise ranging by measuring the round-trip flight time of light pulses between the transmitting/receiving device and the target object. It is divided into direct measurement Distance technology and indirect distance measurement technology. Among them, the indirect ranging technology measures the phase delay of the reflected light signal relative to the emitted light signal, and then calculates the flight time by the phase delay. According to the different types of modulation and demodulation, it can be divided into continuous wave (CW) modulation and decoding. Modulation method and pulse modulation (Pulse Modulated, PM) modulation and demodulation method. TOF ranging technology does not require complex image processing calculations, and the detection distance is relatively long and can maintain high accuracy.

The structured light ranging technology emits a structured light beam to a space object, then collects the structured light pattern formed by the structured light beam modulated and reflected by the object, and finally uses the triangulation method to perform depth calculation to obtain the depth data of the object. Commonly used structured light patterns include irregular spot patterns, stripe patterns, phase shift patterns and so on.

Structured light technology has very high accuracy in short-distance measurement. It performs well in low-light environments, but is easily affected in strong light environments. Relatively speaking, the anti-interference effect of TOF technology in strong light environments is better than that of structure. Light technology.

Therefore, if a solution can be provided that integrates structured light technology and TOF technology, and makes full use of the advantages of both for depth measurement, it will greatly improve the advantages of measurement.

Summary of the invention

The purpose of this application is to provide a fusion depth measurement device and measurement method to solve at least one of the above-mentioned background technical problems.

In order to achieve the foregoing objectives, the technical solutions of the embodiments of the present application are implemented as follows:

A fusion depth measurement device, including a transmitting module, a receiving module, and a control and processing circuit respectively connected with the transmitting module and the receiving module; wherein the transmitting module includes a light source array and optical elements, and the light source array is used for transmitting A light beam whose amplitude is modulated in time sequence. The optical element receives the light beam and then emits a spot-patterned light beam to a target object. The receiving module includes a TOF image sensor. The TOF image sensor includes a pixel array. The pixel array receives the light beam from the target. The spot pattern beam reflected by the object forms an electrical signal; the control and processing circuit receives the electrical signal and calculates the phase difference, uses the phase difference to calculate the TOF depth map of the target object; and, receives the electrical signal and calculates it. Structured light pattern, using the structured light pattern to calculate the structured light depth map of the target object; and, using the depth value in the TOF depth map as a reliable point, assigning values to the corresponding pixel positions in the structured light depth map, And use the reliable point to correct the structured light depth map, and finally obtain a depth image of the target object.

In some embodiments, the TOF image sensor includes at least one pixel; wherein each pixel includes two or more taps.

In some embodiments, the control and processing circuit provides the demodulated signal of each tap in each pixel of the TOF image sensor, and the taps are controlled by the demodulated signal to collect the reflected light beam generated by the reflected light from the target object. Electrical signal.

In some embodiments, the control and processing circuit includes a phase calculation module and an intensity calculation module, and the electrical signal generated by the TOF image sensor is transmitted to the phase calculation module and the intensity calculation module at the same time, and is calculated through processing. According to the phase information and intensity information corresponding to the pixel, a TOF depth map and a structured light depth map corresponding to the pixel are further obtained according to the phase information and the intensity information.

In some embodiments, the control and processing circuit further includes a calibration module, a matching module, and a correction module; wherein the TOF depth map obtained by the calibration module and the structured light depth obtained by the matching module The map is input to the correction module to establish a mapping between the TOF depth map and the structured light depth map to ensure that the TOF depth value calculated from the electrical signal generated by each pixel corresponds to the calculated structured light depth Value, assign the TOF depth value to the corresponding pixel coordinates in the structured light depth map, and take the assigned point as a reliable point, and perform correction processing on the unassigned point in the structured light depth map.

Another technical solution of this application is:

A fusion depth measurement method includes the following steps:

S1. Control the light source array to emit a light beam whose amplitude is modulated in time sequence, and the optical element emits a spot pattern light beam to a target object after receiving the light beam;

S2. Receive the spot pattern light beam reflected by the target object through the pixel array in the TOF image sensor and form an electric signal;

S3. Receive the electrical signal and calculate the phase difference, calculate the TOF depth map of the target object based on the phase difference; and, receive the electrical signal to form a structured light pattern, and use the structured light pattern to calculate the The structured light depth map of the target object; and, the depth value in the TOF depth map is used as a reliable point, assigned to the corresponding pixel position in the structured light depth map, and the structured light depth is corrected by using the reliable point Figure, finally get the depth image of the target object.

In some embodiments, in step S1, the light source array emits the spot pattern beam toward the target area, and the control and processing circuit controls the amplitude of the beam corresponding to each spot in the spot pattern beam to be in a continuous wave, square wave or pulse mode in time sequence. At least one way of modulation.

In some embodiments, in step S2, the TOF image sensor includes at least one pixel, and each pixel includes two or more taps; within a single frame period, the taps are sequentially switched in a certain order to collect the corresponding photons. Receive optical signals and convert them into electrical signals.

In some embodiments, in step S3, the electrical signal input by the pixel array is received through the control and processing circuit to perform phase calculation and intensity calculation respectively to obtain the TOF depth map and the structured light depth map of the target object, and the TOF depth value Assign values to the corresponding pixels in the structured light depth map, thereby distinguishing the points on the structured light depth map into reliable points and unreliable points, and use the reliable points to correct the unreliable points in combination with a correction algorithm , To obtain the depth image of the target object.

Another technical solution of this application is:

An electronic device comprising a housing, a screen, and a fused depth measuring device; the fused depth measuring device includes a transmitting module, a receiving module, and a control and processing circuit respectively connected to the transmitting module and the receiving module; The transmitting module includes a light source array and an optical element, the light source array is used to transmit a light beam whose amplitude is modulated in time sequence, and the optical element emits a spot pattern light beam to a target object after receiving the light beam; the receiving module includes a TOF image Sensor, the TOF image sensor includes a pixel array, the pixel array receives the spot pattern beam reflected by the target object and forms an electrical signal; the control and processing circuit receives the electrical signal and calculates the phase difference, using all The phase difference calculates the TOF depth map of the target object; and, receiving the electrical signal and calculating the structured light pattern, using the structured light pattern to calculate the structured light depth map of the target object; and, converting the TOF The depth value in the depth map is used as a reliable point, assigned to the corresponding pixel position in the structured light depth map, and the structured light depth map is corrected by using the reliable point to finally obtain a depth image of the target object; wherein, the fusion The transmitting module and the receiving module of the depth measuring device are arranged on the first plane of the electronic device, and are used to transmit a spot pattern light beam whose amplitude is modulated in time sequence to a target object and receive a spot pattern light beam reflected by the target object; The screen is installed on the second plane of the electronic device to display information such as images or text; the first plane and the second plane are the same plane or the first plane and the second plane For opposite planes.

The embodiment of the present application provides a fusion depth measurement device, including a transmitting module for transmitting a spot pattern beam whose amplitude is time-modulated to a target object; a receiving module for receiving the spot pattern beam reflected by the target object and forming an electrical signal; controlling and The processing circuit receives the electrical signal and calculates the TOF depth map and the structured light pattern, uses the depth value in the TOF depth map as a reliable point, assigns the value to the corresponding pixel position in the structured light depth map, and uses the The structured light depth map is calibrated reliably, and the depth image of the target object is finally obtained. This application has high measurement accuracy based on the TOF depth value. The TOF depth value is used to correct the error caused by the matching calculation of the structured light depth map to obtain a high-precision depth map, combined with the high resolution advantages of structured light measurement, to achieve A depth measuring device with high precision, high resolution, low power consumption and miniaturization is developed.

Description of the drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

Fig. 1 is a schematic diagram of a fusion depth measuring device according to an embodiment of the present application.

Fig. 2 is a schematic diagram of the principle of a fusion depth measuring device according to an embodiment of the present application.

Fig. 3 is a schematic diagram of the control and processing circuit architecture of a fused depth measurement device according to an embodiment of the present application.

Fig. 4 is a flowchart of a fusion depth measurement method according to an embodiment of the present application.

Fig. 5 is a schematic diagram of an electronic device integrated with the depth measurement device fused in Fig. 1 according to an embodiment of the present application.

Detailed ways

In order to make the technical problems, technical solutions, and beneficial effects to be solved by the embodiments of the present application clearer, the following further describes the present application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application, and are not used to limit the present application.

It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, the connection can be used for fixing or circuit connection.

It should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top" The orientation or positional relationship indicated by "bottom", "inner", "outer", etc. are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the embodiments of the application and simplifying the description, rather than indicating or implying what is meant. The device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present application.

In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first" and "second" may explicitly or implicitly include one or more of these features. In the description of the embodiments of the present application, "plurality" means two or more, unless otherwise specifically defined.

Referring to FIG. 1, FIG. 1 is a schematic diagram of a fusion depth measurement device according to an embodiment of the application. The fused depth measurement device 10 includes a transmitting module 11, a receiving module 12, and a control and processing circuit 13 respectively connected to the transmitting module 11 and the receiving module 12; wherein, the transmitting module 11 is used to transmit a light beam 30 to a target object 20, and transmit The light beam 30 is a spot pattern light beam whose amplitude is modulated in time sequence. The spot pattern light beam is emitted into the target space to illuminate the target object 20 in the space. At least part of the emitted light beam 30 is reflected by the target object 20 to form a reflected light beam 40. The reflected light beam At least part of the light beam in 40 is received by the receiving module 12; the control and processing circuit 13 is respectively connected with the transmitting module 11 and the receiving module 12 to control the transmission and reception of the light beam, and at the same time receive the information generated by the receiving module 12 receiving the reflected light beam, And calculate the information to obtain the depth information of the target object.

The emission module 11 includes a light source array 111, an optical element 112, a light source driver (not shown in the figure), and the like. The light source array 111 may be a light source array composed of multiple light sources such as light emitting diodes (LED), edge emitting lasers (EEL), and vertical cavity surface emitting lasers (VCSEL). The light beams emitted by the light sources may be visible light, infrared light, or ultraviolet light. Wait. Preferably, the light source array 111 is an irregularly arranged VCSEL _S array for emitting irregular spot pattern beams. The light source array 111 is controlled by the light source driver (which may be further controlled by the control and processing circuit 13) to emit light beams after being modulated with a certain timing amplitude. For example, in one embodiment, the light source array 111 is under the control of the light source driver. Transmit pulse modulated beams, square wave modulated beams, sine wave modulated beams and other beams at a certain frequency. In an embodiment of the present application, the amplitude of the beam corresponding to each spot in the irregular spot pattern beam is modulated in a continuous wave, square wave or pulse manner in time sequence. It is understandable that a part of the control and processing circuit 13 or a sub-circuit independent of the control and processing circuit 13 can be used to control the light source array 111 to emit related light beams, such as a pulse signal generator.

The optical element 112 receives the light beam from the light source array 111 and emits the spot pattern light beam outward. In some embodiments, the optical element 112 is also used to expand the received light beam to expand the angle of view of the measuring device. It can be understood that the amplitude of the light beam modulated by the optical element 112 is still modulated in a certain time sequence, that is, the incident sine wave modulated light beam is still emitted by the sine wave modulated light beam. The optical element 112 may be one or a combination of a lens, a diffractive optical element (DOE), a microlens array, and a liquid crystal.

The receiving module 12 includes a TOF image sensor 121, a filter unit 122, and a lens unit 123. The lens unit 123 receives and images at least part of the spot pattern beam reflected by the target object on at least part of the TOF image sensor 121; the filter unit 122 is configured to interact with the light source Narrowband filters with matched wavelengths are used to suppress background light noise in other wavelength bands. The TOF image sensor 121 can be an image sensor array composed of charge coupled devices (CCD), complementary metal oxide semiconductors (CMOS), avalanche diodes (AD), single photon avalanche diodes (SPAD), etc. The size of the array represents the depth of the camera. Resolution, such as 320×240, etc. Generally, connected to the image sensor 121 also includes a readout circuit composed of one or more of a signal amplifier, a time-to-digital converter (TDC), an analog-to-digital converter (ADC) and other devices (not shown in the figure). ).

Generally, the TOF image sensor 121 includes at least one pixel. Compared with a traditional image sensor that is only used for taking pictures, each pixel in the TOF image sensor 121 here includes two or more taps (tap, used in the corresponding electrode). Store and read or discharge the charge signal generated by incident photons under control), for example, including 2 taps, and switch the taps in a certain order within a single frame period (or single exposure time) to collect the corresponding photons. Receive optical signals and convert them into electrical signals.

The control and processing circuit 13 can be an independent dedicated circuit, such as a dedicated SOC chip, FPGA chip, ASIC chip, etc. composed of CPU, memory, bus, etc., or a general processing circuit, such as when the depth measuring device is integrated into When in a smart terminal such as a mobile phone, a TV, a computer, etc., the processing circuit in the smart terminal can be used as at least a part of the control and processing circuit 13.

The control and processing circuit 13 is used to provide a modulation signal (transmission signal) required when the light source array 111 emits laser light, and the light source emits a light beam to a target object under the control of the modulation signal. For example, in one embodiment, the modulation signal is a square wave signal or a pulse signal, and the light source is modulated in time sequence in amplitude under the modulation of the modulation signal to generate a square wave signal or a pulse signal for external emission.

The control and processing circuit 13 also provides a demodulated signal (collection signal) of each tap in each pixel of the TOF image sensor 121, and the tap collects the electrical signal generated by the reflected light beam containing the target object under the control of the demodulated signal. On the one hand, the control and processing circuit 13 processes the electrical signal and calculates the intensity information reflecting the intensity of the reflected beam to form a structured light pattern. Finally, based on the structured light pattern, it uses matching calculations, triangulation calculations, etc. to perform calculations to obtain the test Structured light depth image of the target object. At the same time, the control and processing circuit 13 processes the electrical signal and calculates the phase difference reflecting the light beam from emission to reception, calculates the flight time of the light beam based on the phase difference, and further obtains the TOF depth image of the target object. Further, the control and processing circuit 13 can also correct the structured light depth image according to the TOF depth image. For example, the depth value in the TOF depth map can be used as a reliable point and assigned to the corresponding pixel position in the structured light depth map, and The structured light depth map is corrected using the reliable point, and a more specific correction method will be described later.

In some embodiments, the depth measurement device 10 may also include a drive circuit, a power supply, a color camera, an infrared camera, an IMU, and other devices, which are not shown in the figure. The combination with these devices can achieve richer functions, such as 3D. Texture modeling, infrared face recognition, SLAM and other functions. The depth measuring device 10 may be embedded in electronic products such as mobile phones, tablet computers, and computers.

Fig. 2 is a schematic diagram of the principle of a depth measuring device according to an embodiment of the present application. The control and processing circuit 13 controls the light source array 111 to emit a spot pattern beam 301 whose amplitude is modulated by a square wave or pulse toward the target object, and the amplitude of each spot 302 is modulated by a square wave or pulse in time sequence. It is understandable that the light sources in the light source array 111 are modulated in the same manner. In some other embodiments, a sine wave may also be used to modulate the amplitude of the emitted light beam.

In one embodiment, each pixel of the TOF image sensor in the receiving module 12 includes 4 taps, which are respectively used to collect light signals 4 times in a single frame period and convert them into electrical signals C ₁ , C ₂ , C ₃ Same as C ₄ , the time and interval of 4 acquisitions are the same.

The control and processing circuit 13 receives the electrical signals C ₁ , C ₂ , C ₃ and C _{4 to} calculate the intensity information of the spot pattern beam. In one embodiment, the intensity information is calculated according to the following formula:

After obtaining the intensity information of all pixels, the structured light pattern can be formed, and finally the structured light pattern is used for matching calculation to obtain the parallax and calculate the structured light depth image according to the parallax.

When there is an ambient light signal, this beam intensity calculation method is the same as the traditional method, which is difficult to eliminate, resulting in a low signal-to-noise ratio of the final grayscale pattern. Therefore, in one embodiment, the intensity information will be calculated according to the following formula:

The structured light pattern is generated according to the calculated intensity information of the spot pattern beam, and further matching calculation is performed according to the structured light pattern to obtain the parallax and the structured light depth image is calculated according to the parallax.

The above-mentioned structured light acquisition scheme based on the 4-tap TOF image sensor and the square wave or pulse modulated light emission signal is also applicable to other tapped TOF image sensors and other types of depth measurement devices with modulated light emission signals. It is understandable that, compared with the traditional structured light depth measurement, the present application uses the transmitting end to emit a time-modulated speckle projection beam, and adopts the receiving end multi-tap pixel collection method, so that this method can have more functions than the traditional solution. For example, it is possible to implement depth measurement methods that are difficult to resist environmental interference in traditional solutions.

At the same time, the control and processing circuit 13 also receives the TOF image sensor output. Under the control of the demodulation signal, each tap collects the electrical signal generated by the reflected beam reflected by the target object to calculate the reflected beam phase difference and calculate the reflected beam according to the phase difference. The flight time from the transmitting end to the receiving end is further calculated based on the flight time to calculate the TOF depth image of the target object.

Fig. 3 is a schematic diagram of a control and processing circuit architecture according to an embodiment of the present application. The control and processing circuit 13 includes a phase calculation module 131 and an intensity calculation module 133; wherein the output of the phase calculation module 131 is connected to the calibration module 132; the output of the intensity calculation module 133 is connected to the preprocessing module 134, the preprocessing module 134 The output of is connected to the matching module 135. Among them, the input of the calibration module 132 and the matching module 135 is also connected to the memory 137, and the output is connected to the correction module 136.

The control and processing circuit 13 receives the electrical signal from the TOF image sensor. In the embodiment of the present application, the transmitting module 11 emits an irregular spot pattern beam with modulated amplitude toward the target area, and the receiving module 12 receives the spot pattern reflected by the target object. beam. For each pixel in the TOF image sensor that receives the reflected light beam, the electrical signal generated by it is transmitted to the phase calculation module 131 and the intensity calculation module 133 at the same time, and the phase information and intensity information corresponding to the pixel are obtained through processing and calculation. The information and intensity information further obtain the TOF depth map and the structured light depth map corresponding to the pixel, which have a corresponding positional relationship. Further, the depth value in the TOF depth map is used as a reliable point, assigned to the corresponding pixel position in the structured light depth map, and the structured light depth map is corrected by using the reliable point. Specifically, including:

(1) Calculate TOF depth map

After the electrical signal generated by the TOF image sensor 121 is transmitted to the phase calculation module 131, the phase calculation module 131 processes and calculates the electrical signal to obtain the phase difference. Based on the phase difference, the flight time of the beam from emission to reception can be calculated to further obtain the target object. In the TOF depth image of, there is a linear relationship between the phase difference and the depth value. Therefore, in some embodiments, the TOF depth image can be directly calculated by the phase calculation module 131. Then the depth map is sent to the calibration module 132 for calibration. Because TOF measurement is often interfered by noise, there is a certain error between the measured value and the actual value. Therefore, a calibration step will be adopted before actual use, such as in a certain measurement interval. The calibration board is set at intervals in the internal, and the actual depth value of the calibration board is known, and then the actual measurement of the calibration board at different distances is performed successively to obtain the measured value corresponding to each distance, and the relationship between the measured value and the actual value can be As the pre-calibration parameters are stored in the memory 137, the calibration module will call the pre-calibration parameters from the memory 137 to calibrate the current measurement value during calibration. The pre-calibration parameter here can be a comparison table (index) between the actual value and the measured value. At this time, the calibration process of the calibration module 132 is actually a table look-up process; it can also be a certain mathematical method to model the error and pass Multiple measurements are performed in advance to calculate the unknown parameters in the model. The calibration process of the calibration module 132 is actually a process of calculating actual values based on the model and the measured values. After calibration, an accurate TOF depth map can be obtained.

(2) Calculate structured light depth map

After the electrical signal generated by the TOF image sensor 121 is transmitted to the intensity calculation module 133, the intensity calculation module 133 performs phase calculation of the electrical signal to obtain intensity information reflecting the intensity of the light beam to form a structured light image. Subsequently, the structured light image is sent to the preprocessing module 134 for processing such as denoising and contrast enhancement. Pre-processing tasks such as image distortion correction can also be performed on the structured light image. The pre-processed image then enters the matching module 135 for matching calculation. During the matching calculation, the matching module 135 will call the pre-stored reference image from the memory 137. In one embodiment, the matching module 135 uses zero mean normalization The least square distance function is used to estimate the pixel deviation between the structured light image and the reference image. According to the structured light triangulation method, there is a certain relationship between the pixel deviation value and the depth value of the target. Therefore, the matching module 135 can directly calculate the depth value to obtain the structured light depth image of the target to be measured. In some embodiments, it is also possible to perform post-processing on the depth image, such as image enhancement, interpolation calculation, etc., to optimize it, such as hole filling, edge optimization, and so on.

(3) Correct structured light depth map

Since structured light depth calculation requires matching calculation, matching calculation consumes a lot of resources in the entire depth calculation link and has a greater impact on accuracy, resulting in errors in the final structured light depth map; while the TOF depth map is directly obtained by phase difference calculation. The measurement accuracy is relatively accurate, but in the TOF calculation mode, only the depth value of the part of the pixel that the reflected beam enters can be obtained, but the depth value of all the pixels cannot be obtained. Therefore, in the embodiment of the present application, the TOF depth map is extracted The depth value is used as a reliable point to correct the structured light depth map.

Specifically, the TOF depth map obtained by the calibration module 132 and the structured light depth map obtained by the matching module 135 are input to the correction module 135 to establish a mapping between the TOF depth map and the structured light depth map to ensure the TOF calculated from the electrical signal generated by each pixel The depth value corresponds to the calculated structured light depth value. The TOF depth value is assigned to the corresponding pixel coordinates in the structured light depth map. These assigned points are called reliable points, and the unassigned points are called unreliable points. Based on the reliable points, the unreliable points in the structured light depth map are corrected to improve the accuracy of the structured light depth map. The correction method may include the region growing method, the confidence weighting method, and so on.

In one embodiment, the area growth method is used to correct the structured light depth map. Region growth is calculated based on the assumption that the object has continuity and the depth values between two close points on the object are approximately equal. In the region to be predicted, multiple reliable points are selected as seed points, and by setting appropriate growth criteria, Use four connectivity (up, down, left, right) to connect unreliable points and reliable points that satisfy the correlation to the same area to form a new area.

In the embodiment of the present application, the reliable point assigned by the TOF depth value is called the seed point, an appropriate growth criterion is set, and a first-in first-out area q is set. The depth value of all pixels in the area is equal to The depth value of the seed point. For example, if the pixel coordinates of a seed point p are selected as (x ₀ , y ₀ ), the depth value of the seed point p is d, and the four pixel coordinates (x ₀ , y ₀ -1) adjacent to the point p are predicted, (x 0, y 0 -1) ₀ , y ₀ +1), (x ₀ -1, y ₀ ), (x ₀ +1, y ₀ ) the correlation between the depth value and the seed point p, assuming that the growth criterion sets two adjacent pixels The absolute value of the difference between the point depth values is less than the custom threshold T. If the correlation between more than one unreliable point and the seed point p meets the set growth criterion, the unreliable point that meets the condition is merged with the seed point When the region q where the seed point is located becomes a reliable point, the newly obtained reliable point is used as the seed point to continue the prediction; if the correlation between the point p and the depth value of the adjacent four pixel coordinate points does not meet the set growth criterion , The growth of the region is terminated. After the correction processing, the measurement accuracy of the depth map is effectively improved.

It is understandable that the above-mentioned embodiment is only a specific embodiment, and does not specifically limit the correction method of the present application, and any correction method that belongs to the design concept of the present application belongs to the disclosure scope of the present application.

Based on the fused depth measurement device of the foregoing embodiment, this application also proposes a fused depth measurement method.

Referring to Fig. 4, Fig. 4 is a flowchart of a fusion depth measurement method according to another embodiment of the application, which includes the following steps:

S1. Control the light source array to emit a light beam whose amplitude is modulated in time sequence, and the optical element emits a spot pattern light beam to the target object after receiving the light beam;

S2. Receive the spot pattern light beam reflected by the target object through the pixel array in the TOF image sensor and form an electrical signal;

S3. Receive the electrical signal and calculate the phase difference, calculate the TOF depth map of the target object based on the phase difference; and, receive the electrical signal to form a structured light pattern, and use the structured light pattern to calculate the structured light depth map of the target object; and The depth value in the TOF depth map is used as a reliable point, assigned to the corresponding pixel position in the structured light depth map, and the structured light depth map is corrected by the reliable point, and finally the depth image of the target object is obtained.

Specifically, the light source array emits the spot pattern beam toward the target area, and the control circuit controls the amplitude of the beam corresponding to each spot in the spot pattern beam to be modulated by at least one of continuous wave, square wave, or pulse in time sequence.

Specifically, the TOF image sensor includes at least one pixel, and each pixel includes two or more taps. Preferably, each pixel includes 4 taps, which can be determined within a single frame period (or single exposure time). The taps are sequentially switched in order to collect the corresponding photons to receive the optical signal and convert it into an electrical signal.

Specifically, the control circuit receives the electrical signal input from the pixel array to perform phase calculation and intensity calculation respectively, wherein the TOF depth map of the target area to be measured is obtained through phase calculation; the structured light depth map of the target area to be measured is obtained through intensity calculation. Assign the TOF depth value to the corresponding pixel in the structured light depth map, thereby distinguishing the points on the structured light depth map into reliable points (TOF depth values) and unreliable points, and use the reliable points to correct unreliable points in combination with the correction algorithm point.

In one embodiment, reliable points are used as seed points, and unreliable points are optimized in combination with a region growing algorithm, so as to correct the structured light depth map to obtain a depth image with higher accuracy.

This application proposes a fusion depth measurement device and measurement method composed of a structured light emitting end and a TOF image sensor based on amplitude timing modulation. By projecting a spot patterned beam toward the target area, TOF depth value calculation and structured light depth are respectively performed Value calculation, based on the TOF depth value, has high measurement accuracy. Based on the TOF depth value, the error caused by the matching calculation of the structured light depth map is corrected to obtain a high-precision depth map, combined with the high resolution advantages of structured light measurement, A high-precision, high-resolution, low-power, and miniaturized depth measurement solution is realized.

As another embodiment of the present application, an electronic device is also provided. The electronic device may be a mobile phone, a tablet, a computer, a TV, a smart helmet, smart glasses, a robot, and so on. 5, taking a mobile phone as an example for description, the electronic device 500 includes a housing 51, a screen 52, and the fused depth measurement device described in the foregoing embodiment; the screen 52 is used for information display; and the The housing 51 can provide protection functions such as dustproof, waterproof, and drop-proof for the electronic device.

Specifically, the transmitting module 11 and the receiving module 12 of the fused depth measuring device are arranged on the first plane of the electronic device 500, and are used to transmit to the target object the spot pattern beam whose amplitude is modulated in time sequence and to receive the light beam reflected by the target object. Spot pattern light beam; the screen 52 is installed on the second plane of the electronic device to display information such as images or text; the first plane and the second plane are the same plane or the first plane and the second plane are opposite planes. In some embodiments, the control and processing circuit of the fused depth measurement device can be shared with the electronic device; and in some embodiments, when the electronic device itself is provided with a TOF image sensor, the receiving module of the fused depth measurement device can also be used. The TOF image sensor is shared.

By integrating the fused depth measurement device into electronic devices, such as mobile phones, tablets, computers, TVs, smart helmets, smart glasses, and robots, the functions of electronic devices are continuously expanded and their applications are more and more extensive. For example, Realize the depth measurement of the target object to obtain a higher-precision depth image containing the target, and further based on the high-precision depth image can realize three-dimensional reconstruction, face recognition, human-computer interaction and other functions.

It is understandable that the above content is a further detailed description of the application in combination with specific/preferred implementations, and it cannot be considered that the specific implementation of the application is limited to these descriptions. For those of ordinary skill in the technical field to which this application belongs, without departing from the concept of this application, they can also make several substitutions or modifications to the described implementations, and these substitutions or modifications should be regarded as It belongs to the protection scope of this application. In the description of this specification, reference to the description of the terms "one embodiment", "some embodiments", "preferred embodiment", "examples", "specific examples", or "some examples" etc. means to incorporate the implementation The specific features, structures, materials or characteristics described by the examples or examples are included in at least one embodiment or example of the present application.

In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the described specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples and the features of the different embodiments or examples described in this specification without contradicting each other. Although the embodiments of the present application and its advantages have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the scope defined by the appended claims.

In addition, the scope of the present application is not intended to be limited to the specific embodiments of the processes, machines, manufacturing, material composition, means, methods, and steps described in the specification. A person of ordinary skill in the art will easily understand that the above-mentioned disclosures, processes, machines, processes, machines, processes that currently exist or will be developed later that perform substantially the same functions as the corresponding embodiments described herein or obtain substantially the same results as the embodiments described herein can be used. Manufacturing, material composition, means, method, or step. Therefore, the appended claims intend to include these processes, machines, manufacturing, material compositions, means, methods, or steps within their scope.

Claims

A fusion depth measurement device, which is characterized by comprising a transmitting module, a receiving module, and a control and processing circuit respectively connected with the transmitting module and the receiving module; wherein,

The transmitting module includes a light source array and an optical element, the light source array is used to emit a light beam whose amplitude is modulated in time sequence, and the optical element emits a spot pattern light beam to a target object after receiving the light beam;

A receiving module, including a TOF image sensor, the TOF image sensor including a pixel array, the pixel array receives the spot pattern light beam reflected by the target object and forms an electrical signal;

The control and processing circuit receives the electrical signal and calculates the phase difference, uses the phase difference to calculate the TOF depth map of the target object; and, receives the electrical signal and calculates the structured light pattern, and uses the structured light Pattern calculation of the structured light depth map of the target object; and, taking the depth value in the TOF depth map as a reliable point, assigning values to the corresponding pixel position in the structured light depth map, and using the reliable point to correct the Structured light depth map, and finally get the depth image of the target object.
The fusion depth measurement device according to claim 1, wherein the TOF image sensor includes at least one pixel; wherein each pixel includes two or more taps.
The fusion depth measurement device of claim 2, wherein the control and processing circuit provides demodulated signals of each tap in each pixel of the TOF image sensor, and the taps are collected by the demodulated signal under the control of the demodulated signal. Contains the electrical signal generated by the reflected beam of the target object.
The fusion depth measurement device of claim 1, wherein the control and processing circuit includes a phase calculation module and an intensity calculation module, and the electrical signal generated by the TOF image sensor is simultaneously transmitted to the phase calculation module and The intensity calculation module obtains the phase information and intensity information corresponding to the pixel through processing and calculation, and further obtains the TOF depth map and the structured light depth map corresponding to the pixel according to the phase information and the intensity information.
The fusion depth measurement device according to claim 4, wherein the control and processing circuit further includes a calibration module, a matching module, and a correction module; wherein the TOF depth map obtained by the calibration module and the matching module The structured light depth map obtained by the module is input to the correction module to establish a mapping between the TOF depth map and the structured light depth map to ensure that the TOF depth value calculated from the electrical signal generated by each pixel corresponds to the calculated structured light depth value. The TOF depth value is assigned to the corresponding pixel coordinates in the structured light depth map, and the assigned points are used as reliable points, and the unassigned points in the structured light depth map are corrected.
A fusion depth measurement method is characterized in that it comprises the following steps:

S1. Control the light source array to emit a light beam whose amplitude is modulated in time sequence, and the optical element emits a spot pattern light beam to a target object after receiving the light beam;

S2. Receive the spot pattern light beam reflected by the target object through the pixel array in the TOF image sensor and form an electric signal;

S3. Receive the electrical signal and calculate the phase difference, calculate the TOF depth map of the target object based on the phase difference; and, receive the electrical signal to form a structured light pattern, and use the structured light pattern to calculate the The structured light depth map of the target object; and, the depth value in the TOF depth map is used as a reliable point, assigned to the corresponding pixel position in the structured light depth map, and the structured light depth is corrected by using the reliable point Figure, finally get the depth image of the target object.
The fusion depth measurement method according to claim 6, characterized in that: in step S1, the light source array emits the spot pattern beam toward the target area, and the control and processing circuit controls the amplitude of the beam corresponding to each spot in the spot pattern beam in time sequence. It is modulated by at least one of continuous wave, square wave, or pulse mode.
The fusion depth measurement method of claim 6, wherein: in step S2, the TOF image sensor includes at least one pixel, and each pixel includes two or more taps; within a single frame period, in a certain order Switch the taps in turn to collect the corresponding photons to receive the optical signal and convert it into an electrical signal.
The fusion depth measurement method according to claim 6, characterized in that: in step S3, the electrical signal input by the pixel array is received through the control and processing circuit to perform phase calculation and intensity calculation respectively to obtain the TOF depth map and structure of the target object Light depth map, assigning TOF depth values to the corresponding pixels in the structured light depth map, thereby distinguishing points on the structured light depth map into reliable points and unreliable points, and using the reliable point in combination with a correction algorithm Point correction of the unreliable point to obtain a depth image of the target object.
An electronic device comprising: a housing, a screen, and a fused depth measuring device; the fused depth measuring device includes a transmitting module, a receiving module, and a control and processing circuit respectively connected to the transmitting module and the receiving module; The transmitting module includes a light source array and an optical element, the light source array is used to transmit a light beam whose amplitude is modulated in time sequence, and the optical element emits a spot pattern light beam to a target object after receiving the light beam; the receiving module includes a TOF image Sensor, the TOF image sensor includes a pixel array, the pixel array receives the spot pattern beam reflected by the target object and forms an electrical signal; the control and processing circuit receives the electrical signal and calculates the phase difference, using all The phase difference calculates the TOF depth map of the target object; and, receiving the electrical signal and calculating the structured light pattern, using the structured light pattern to calculate the structured light depth map of the target object; and, converting the TOF The depth value in the depth map is used as a reliable point, assigned to the corresponding pixel position in the structured light depth map, and the structured light depth map is corrected by using the reliable point to finally obtain a depth image of the target object; wherein, the fusion The transmitting module and the receiving module of the depth measuring device are arranged on the first plane of the electronic device, and are used to transmit a spot pattern light beam whose amplitude is modulated in time sequence to a target object and receive a spot pattern light beam reflected by the target object; The screen is installed on the second plane of the electronic device to display information such as images or text; the first plane and the second plane are the same plane or the first plane and the second plane For opposite planes.