WO2022088913A1

WO2022088913A1 - Imaging distance measurement sensor, method, and system, and storage medium

Info

Publication number: WO2022088913A1
Application number: PCT/CN2021/115371
Authority: WO
Inventors: 张学勇
Original assignee: Oppo广东移动通信有限公司
Priority date: 2020-10-28
Filing date: 2021-08-30
Publication date: 2022-05-05
Also published as: CN112363180A

Abstract

The present application is applicable to the technical field of time of flight, and provides an imaging distance measurement sensor, method, and system, and a storage medium. The imaging distance measurement sensor comprises a plurality of pixels arranged in an array form; each pixel comprises at least four sub pixels arranged in an array form; each sub pixel comprises a single-photon photosensitive unit and an optical filter covering the single-photon photosensitive unit; and the at least four sub pixels comprise an infrared sub pixel, a red sub pixel, a green sub pixel, and a blue sub pixel, so that one imaging distance measurement sensor can have the function of simultaneously obtaining depth information and RGB information. A depth image obtained on the basis of the depth information and an RGB image obtained on the basis of the RGB information have no viewing angle difference, and alignment is not needed when image fusion is performed on the depth image and the RGB image, and thus, an operation resource and time can be effectively reduced.

Description

An imaging ranging sensor, method, system and storage medium

This application claims the priority of the Chinese patent application with the application number 202011173365.6 and the application name "An imaging ranging sensor, method, system and storage medium" filed with the China Patent Office on October 28, 2020, the entire content of which is approved by Reference is incorporated into this application.

technical field

The present application belongs to the technical field of Time Of Flight (TOF), and in particular, relates to an imaging ranging sensor, method, system and storage medium.

Background technique

The time-of-flight ranging method obtains the distance of the target by continuously sending optical pulse signals to the target, then receiving the optical signal reflected by the target, and detecting the flight (round-trip) time of the optical pulse signal. Single-photon ranging systems based on time-of-flight technology have been widely used in consumer electronics, unmanned aerial vehicles, virtual reality, augmented reality and other fields.

SUMMARY OF THE INVENTION

The embodiments of the present application provide an imaging ranging sensor, a method, a system and a storage medium, which can solve the problems in the prior art that the depth image and the RGB image need to be aligned through an algorithm, the calculation is complex, and computing resources and time are consumed .

A first aspect of the embodiments of the present application provides an imaging ranging sensor, which includes a plurality of pixels arranged in an array, each of the pixels includes at least four sub-pixels arranged in an array, and each of the sub-pixels includes one A single-photon photosensitive unit and a filter covering the single-photon photosensitive unit, each of the pixels includes an infrared sub-pixel, a red sub-pixel, a green sub-pixel and a blue sub-pixel.

A second aspect of the embodiments of the present application provides an imaging ranging method, which is implemented based on the imaging ranging sensor described in the first aspect of the embodiments of the present application, and the method includes:

acquiring a first light sensing signal output by each pixel and used to indicate the time when infrared light reflected by the object is received;

acquiring a second light sensing signal output by each pixel and used to indicate the time when the visible light reflected by the object is received;

Acquiring depth information of the object according to the first light sensing signal;

The RGB information of the object is acquired according to the second light sensing signal.

A third aspect of the embodiments of the present application provides an imaging ranging system, including:

Light emitters for emitting infrared and visible light to objects;

The imaging ranging sensor according to the first aspect of the embodiments of the present application;

A controller, connected to the light transmitter and the imaging ranging sensor, respectively, is configured to execute the imaging ranging method according to the second aspect of the embodiments of the present application.

A fourth aspect of the embodiments of the present application provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by the controller, the implementation of the second aspect of the embodiments of the present application is implemented The steps of the imaging ranging method described above.

Description of drawings

In order to illustrate the technical solutions in the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only for the present application. In some embodiments, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without any creative effort.

FIG. 1 is a first structural schematic diagram of a pixel provided by an embodiment of the present application;

2 is a schematic diagram of a second structure of a pixel provided by an embodiment of the present application;

3 is a schematic diagram of a third structure of a pixel provided by an embodiment of the present application;

4 is a schematic flowchart of an imaging ranging method provided by an embodiment of the present application;

5 is a correspondence table provided by an embodiment of the present application;

FIG. 6 is a schematic structural diagram of an imaging ranging system provided by an embodiment of the present application.

Detailed ways

In the following description, for the purpose of illustration rather than limitation, specific details such as a specific system structure and technology are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be practiced in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, systems, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

It is to be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described feature, integer, step, operation, element and/or component, but does not exclude one or more other The presence or addition of features, integers, steps, operations, elements, components and/or sets thereof.

It will also be understood that, as used in this specification and the appended claims, the term "and/or" refers to and including any and all possible combinations of one or more of the associated listed items.

As used in the specification of this application and the appended claims, the term "if" may be contextually interpreted as "when" or "once" or "in response to determining" or "in response to detecting ". Similarly, the phrases "if it is determined" or "if the [described condition or event] is detected" may be interpreted, depending on the context, to mean "once it is determined" or "in response to the determination" or "once the [described condition or event] is detected. ]" or "in response to detection of the [described condition or event]".

In addition, in the description of the specification of the present application and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the description, and should not be construed as indicating or implying relative importance.

References in this specification to "one embodiment" or "some embodiments" and the like mean that a particular feature, structure or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in other embodiments," etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean "one or more but not all embodiments" unless specifically emphasized otherwise. The terms "including", "including", "having" and their variants mean "including but not limited to" unless specifically emphasized otherwise.

At present, most TOF devices are specially used to measure distance. , B)) The image sensor obtains RGB information, and performs image fusion on the depth image obtained based on the depth information and the RGB image obtained based on the RGB information, and then the depth image and the RGB image need to be aligned by an algorithm, which is complicated and requires computational complexity. resources and time.

Based on this, the embodiments of the present application provide an imaging ranging sensor, method, system, and storage medium, which can solve the problem that in the prior art, the depth image and the RGB image need to be aligned through an algorithm, which is complicated in calculation and requires consumption of computing resources and question of time.

An embodiment of the present application provides an imaging ranging sensor, which includes several pixels arranged in an array, each pixel includes at least four sub-pixels arranged in an array, and each sub-pixel includes a single-photon photosensitive unit and a covering single-photon photosensitive unit Each pixel includes an infrared sub-pixel, a red sub-pixel, a green sub-pixel and a blue sub-pixel.

In applications, the number of pixels included in the imaging ranging sensor can be set according to actual needs, and the more the number of pixels, the higher the resolution of the imaging ranging sensor. Several pixels in an imaging ranging sensor can be arranged in an array of any regular shape, eg, a rectangular array, a circular array, or any other array of regular polygons. The number of sub-pixels included in each pixel can be set according to actual needs, as long as each pixel includes infrared sub-pixels, red sub-pixels, green sub-pixels and blue sub-pixels, so that the depth of the object can be obtained through the imaging ranging sensor information and RGB information, the depth information and RGB information of each reflective point corresponding to each pixel on the object can be obtained. The depth information of each reflective point on the object includes the distance between the reflective point and the imaging ranging sensor, and the RGB information of each reflective point includes the R value, G value and B value of the reflective point, R value, G value The and B values may specifically be CIE spectral tristimulus values.

In application, each sub-pixel includes a single-photon photosensitive unit and a filter covering the single-photon photosensitive unit. The filters of the infrared sub-pixel, red sub-pixel, green sub-pixel and blue sub-pixel are only passable Infrared filters for infrared light, red filters for only red light, green filters for only green light, and blue filters for only blue light. It needs to be set to a reflective filter or an absorptive filter.

In applications, the single-photon photosensitive unit can be a single-photon avalanche diode (Single Photon Avalanche Diode, SPAD), or a photomultiplier tube. The single-photon photosensitive unit can respond to an incident single photon and output it to indicate that the photon reaches the single photon. The light sensing signal of the time at the photon photosensitive unit, based on the light sensing signal output by the single photon photosensitive unit, can use the time-correlated single photon counting method (Time-Correlated Single Photon Counting, TCSPC) to realize the collection of weak light signals and the time of flight calculation.

The imaging ranging sensor provided by the embodiments of the present application includes several pixels arranged in an array, each of the pixels includes at least four sub-pixels arranged in an array, and each of the sub-pixels includes a single-photon photosensitive unit and A filter covering the single-photon photosensitive unit, the at least four sub-pixels include infrared sub-pixels, red sub-pixels, green sub-pixels and blue sub-pixels, by making each pixel include infrared sub-pixels, red sub-pixels , green sub-pixels and blue sub-pixels, and each sub-pixel is composed of a single-photon photosensitive unit and a filter covering the single-photon photosensitive unit, so that an imaging ranging sensor can obtain the function of depth information and RGB information at the same time , the depth image obtained based on the depth information and the RGB image obtained based on the RGB information have no difference in perspective, and there is no need to align the depth image and the RGB image when performing image fusion, which can effectively save computing resources and time.

As shown in FIG. 1 , a first structural schematic diagram of an imaging ranging sensor is exemplarily shown;

Wherein, each pixel 101 includes four infrared sub-pixels IR, three red sub-pixels R, six green sub-pixels G and three blue sub-pixels B;

Four infrared sub-pixels IR are arranged in a 2×2 array to form an infrared sub-pixel unit 10, and a red sub-pixel R, two green sub-pixels G and a blue sub-pixel B are arranged in a Bayer array to form a color sub-pixel unit 20;

Each pixel 101 includes one infrared sub-pixel unit 10 and three color sub-pixel units 20 .

In applications, the arrangement of one infrared sub-pixel unit and three color sub-pixel units included in each pixel can be set according to actual needs, for example, in a 1×4, 4×1 or 2×2 array.

FIG. 1 exemplarily shows a schematic structural diagram of an imaging ranging sensor when one infrared sub-pixel unit 10 and three color sub-pixel units 20 are arranged in a 2×2 array.

In the structure of the imaging ranging sensor shown in FIG. 1, by arranging four infrared sub-pixels in a 2×2 array into one infrared sub-pixel unit, four infrared sub-pixels can be binning into one super-pixel, for example, Connect four infrared sub-pixels to an "OR gate" or an "AND gate" to form an infrared superpixel, which can effectively improve the signal-to-noise ratio of the infrared sensing signal and improve the ranging accuracy; by using the Bayer array form (ie RGGB ) arranges one red sub-pixel, two green sub-pixels and one blue sub-pixel into a color sub-pixel unit, and can use a mature Bayer array-based algorithm to restore color information, with high imaging accuracy.

As shown in FIG. 2 , a schematic diagram of the second structure of the imaging ranging sensor is exemplarily shown, wherein each pixel 102 includes an infrared sub-pixel IR, a red sub-pixel R, One green sub-pixel G and one blue sub-pixel B.

The structure of the imaging ranging sensor shown in Figure 2, by arranging four sub-pixels in a 2×2 array to form one pixel, a single pixel occupies a small area, and the imaging ranging sensor can accommodate more pixels under the same area , which can effectively improve the resolution of the imaging ranging sensor and make the fusion result of depth information and RGB information more accurate.

As shown in FIG. 3 , the third structural schematic diagram of the imaging ranging sensor is exemplarily shown;

Wherein, each pixel 103 includes M×M infrared sub-pixels IR, M×M red sub-pixels R, M×M green sub-pixels G and M×M blue sub-pixels B;

M×M infrared sub-pixels IR are arranged in an array to form an infrared sub-pixel unit 1, M×M red sub-pixels R are arranged in an array to form a red sub-pixel unit 2, and M×M green sub-pixels G are arranged in an array. A green sub-pixel unit 3 is arranged in a form, and M×M blue sub-pixels B are arranged in an array to constitute a blue sub-pixel unit 4;

Each pixel 103 includes an infrared sub-pixel unit 1 , a red sub-pixel unit 2 , a green sub-pixel unit 3 and a blue sub-pixel unit 4 .

In application, M is an integer greater than or equal to 2, and the specific value of M can be set according to actual needs. The arrangement of one infrared sub-pixel unit, one red sub-pixel unit, one green sub-pixel unit and one blue sub-pixel unit included in each pixel can be set according to actual needs, for example, 1×4, 4×1 Or arranged in a 2x2 array.

FIG. 3 exemplarily shows that when M=2 and one infrared sub-pixel unit, one red sub-pixel unit, one green sub-pixel unit and one blue sub-pixel unit are arranged in a 2×2 array, the imaging distance measuring sensor Schematic.

In the structure of the imaging ranging sensor shown in FIG. 3, by arranging M×M infrared sub-pixels in an M×M array to form an infrared sub-pixel unit, M×M infrared sub-pixels can be combined into an infrared super-pixel, For example, connecting four infrared sub-pixels to an "OR gate" or an "AND gate" to form an infrared superpixel can effectively improve the signal-to-noise ratio of the infrared sensing signal and improve the ranging accuracy;

By arranging M×M red sub-pixels in an M×M array to form a red sub-pixel unit, M×M red sub-pixels can be combined into a red super-pixel, for example, four red sub-pixels are connected to a " "OR gate" or an "AND gate" to form a red superpixel, which can effectively improve the signal-to-noise ratio of the red light sensing signal and improve the accuracy of the R value;

By arranging M×M green sub-pixels in an M×M array to form a green sub-pixel unit, M×M green sub-pixels can be combined into a green super-pixel, for example, four green sub-pixels are connected to a " "OR gate" or an "AND gate" to form a green super pixel, which can effectively improve the signal-to-noise ratio of the green light sensing signal and improve the accuracy of the G value;

By arranging M×M blue sub-pixels in an M×M array to form a blue sub-pixel unit, M×M blue sub-pixels can be combined into a blue superpixel, for example, four blue sub-pixels can be combined The pixel is connected to an "OR gate" or an "AND gate" to form a blue superpixel, which can effectively improve the signal-to-noise ratio of the blue light sensing signal and improve the accuracy of the B value;

The structure of the red sub-pixel unit, the green sub-pixel unit and the blue sub-pixel unit in each pixel can improve the accuracy of the RGB information as a whole.

In the imaging ranging sensor provided by the embodiments of the present application, each pixel includes an infrared sub-pixel, a red sub-pixel, a green sub-pixel, and a blue sub-pixel, and each sub-pixel is composed of a single-photon photosensitive unit and a single-photon coverage unit. The filter composition of the photon photosensitive unit enables an imaging ranging sensor to obtain the function of depth information and RGB information at the same time. There is no difference in perspective between the depth image obtained based on the depth information and the RGB image obtained based on the RGB information. Image fusion does not require alignment, which can effectively save computing resources and time.

As shown in FIG. 4 , based on the structure of the imaging ranging sensor in any of the foregoing embodiments, an embodiment of the present application further provides an imaging ranging method, including:

Step S201, obtaining a first light sensing signal output by each pixel and used to indicate the time when the infrared light reflected by the object is received;

Step S202, acquiring a second light sensing signal output by each pixel and used to indicate the time when the visible light reflected by the object is received;

Step S203, acquiring depth information of the object according to the first light sensing signal;

Step S204: Acquire RGB information of the object according to the second light sensing signal.

The imaging ranging method provided by the embodiment of the present application is implemented based on the imaging ranging sensor provided by the embodiment of the present application, by acquiring a first light sensing signal output by each pixel and used to indicate the time when infrared light reflected by an object is received; Obtain a second light sensing signal output by each pixel and used to indicate the time when the visible light reflected by the object is received; obtain depth information of the object according to the first light sensing signal; obtain according to the second light sensing signal The RGB information of the object can directly obtain the depth signal and RGB information of the object through the first light sensing signal and the second light sensing signal output by the imaging ranging sensor, so that the depth image obtained based on the depth information and the RGB information based on the depth image. There is no difference in viewing angle to obtain RGB images, and no alignment is required when performing image fusion between depth images and RGB images, which can effectively save computing resources and time.

In application, the infrared sub-pixel in each pixel is used to receive the infrared light reflected by the reflective point at the corresponding position on the object, and output a first light sensing signal when receiving the infrared light, based on the reception of the first light The time of the light sensing signal and the time of emitting infrared light to the object, the time-of-flight method can be used to calculate the distance between each reflective point on the object and the imaging ranging sensor. The depth information of the object includes all the reflective points on the object and the imaging distance. The distance between the ranging sensors, and the depth image of the object can be obtained based on the depth information of the object. When the imaging ranging sensor is only used to obtain depth information, the depth information of the object can also be obtained according to the second light-sensing signal, that is, based on the time when the second light-sensing signal is received and the time when visible light is emitted to the object. The time-of-flight method is used to calculate the distance between each reflective point on the object and the imaging ranging sensor.

It should be understood that since the depth information of the object includes the distance between each reflective point on the object and the imaging ranging sensor, the imaging ranging sensor can also be used for ranging in addition to obtaining depth information and RGB information.

In application, the colored sub-pixels in each pixel are used to receive visible light reflected by the reflective point at the corresponding position on the object, and output a second light-sensing signal when receiving the visible light, based on the second light-sensing signal The RGB value (including R value, G value and B value) of each reflective point on the object can be obtained, the RGB information of the object includes the RGB value of all reflective points on the object, and the RGB image of the object is obtained based on the RGB information of the object. Similarly, the infrared image information of the object can also be obtained based on the first light sensing signal, and the infrared image of the object can be obtained based on the infrared image information.

In one embodiment, an imaging ranging method includes:

obtaining the light intensity of the infrared light reflected by the object according to the first light sensing signal;

obtaining infrared image information of the object according to the light intensity of the infrared light reflected by the object;

Step S204 includes:

obtaining the light intensity of the visible light reflected by the object according to the second light sensing signal;

Acquire RGB information of the object according to the light intensity of the visible light reflected by the object;

In the application, the infrared image information of the object is obtained based on the light intensity of the infrared light reflected by each reflection point on the object. Similarly, the RGB information of the object is obtained based on the light intensity of the visible light reflected by each reflection point on the object. According to the light intensity of the infrared light reflected by each point on the object, the brightness value corresponding to each reflective point in the infrared image information of the object can be obtained, and the brightness value can be reflected as a gray value in the infrared image. The obtained infrared image is a grayscale image that contains only the grayscale values corresponding to each reflective point. According to the light intensity of the visible light reflected by each reflective point on the object, the brightness value corresponding to each reflective point in the RGB information of the object can be obtained. According to the brightness value of each reflective point and the color of each color pixel in its corresponding pixel, Based on the calculation formula of the tristimulus value, the RGB value of each reflective point can be obtained, so as to obtain RGB information including the RGB value of each reflective point of the object, and then the RGB image of the object can be obtained.

In one embodiment, the obtaining, according to the second light sensing signal, the light intensity of the visible light reflected by the object includes:

obtaining the total number of photons or the total number of peak signals received by the imaging ranging sensor according to the second light sensing signal;

obtaining the light intensity of the visible light reflected by the object according to the preset first correspondence between the visible light intensity and the total number of photons or the total number of peak signals;

Similarly, obtaining the light intensity of the infrared light reflected by the object according to the first light sensing signal includes:

obtaining the total number of photons or the total number of peak signals received by the imaging ranging sensor according to the first light sensing signal;

According to the second correspondence between the preset infrared light intensity and the total number of photons or the total peak signal times, the light intensity of the infrared light reflected by the object is acquired.

In application, since each single-photon photosensitive unit is used to respond to an incident single photon and output a light-sensing signal indicating the time when the photon arrives at the single-photon photosensitive unit, therefore, according to the light-sensing signal output by all color sub-pixels The signal, that is, the second light sensing signal, can obtain the total number of photons of visible light received by the imaging ranging sensor. Through the Time to Digital Converter (TDC) connected to the color sub-pixels in each pixel, the time of flight is calculated according to the light sensing signal output by each color sub-pixel and converted into time code, and then the time-to-digital The histogram circuit connected to the converter saves the time code output by the time-to-digital converter and generates a histogram according to the time code saved in at least one emission period of visible light, and obtains the number of peak signals in the histogram. The total number of peak signals in the histogram generated by the light-sensing signals output by the color sub-pixels in each pixel is the total number of peak signals. In the same way, the total number of photons of infrared light received by the imaging ranging sensor and the total number of peak signals in the histogram generated according to the photosensitive signals output by the infrared sub-pixels in each pixel can also be obtained to obtain the total peak signal. frequency. The first corresponding relationship and the second corresponding relationship may be obtained through a large number of experiments by using the above-mentioned acquisition method of the total number of photons and the total number of peak signal times in advance.

In one embodiment, before acquiring the total number of photons or the total number of peak signals received by the imaging ranging sensor according to the second light sensing signal, the method includes:

Obtaining the total number of photons or the total number of peak signals received by the imaging ranging sensor under different light intensities of the visible light reflected by the object respectively;

According to the total number of photons or the total peak signal times obtained under each light intensity of visible light reflected by the object, obtain a first correspondence between different visible light intensities and the total photon number or total peak signal times;

Similarly, before acquiring the total number of photons or the total number of peak signals received by the imaging ranging sensor according to the first light sensing signal, the method includes:

Obtain the total number of photons or total peak signal times received by the imaging ranging sensor under different light intensities of the infrared rays reflected by the object respectively;

A second correspondence between different infrared light intensities and total photons or total peak signal times is obtained according to the total number of photons or total peak signal times obtained under each light intensity of the infrared light reflected by the object.

In the application, infrared rays of different light intensities and visible rays of different light intensities can be emitted to the object separately or simultaneously, and then based on the above method, the first correspondence between the different visible light intensities and the total number of photons or the total number of peak signals is obtained. relationship, and a second correspondence between the different infrared light intensities and the total number of photons or the total number of peak signals. Therefore, when the RGB information needs to be obtained subsequently, the light intensity of the visible light can be obtained according to the first correspondence and the total number of photons or the total peak signal times obtained based on the second light sensing signal; when the infrared image information needs to be obtained subsequently, it can be obtained. The light intensity of the infrared light is obtained according to the second correspondence and the total number of photons or the total number of peak signals obtained based on the first light sensing signal.

In an application, the first correspondence and the second correspondence may exist in the form of a correspondence table, for example, a lookup table is displayed.

As shown in FIG. 5 , a correspondence table between the light intensity, the total number of photons and the total peak signal times is exemplarily shown.

The imaging ranging method provided by the embodiment of the present application can directly obtain the depth signal and RGB information of the object through the first light sensing signal and the second light sensing signal output by the imaging ranging sensor, so that the depth image obtained based on the depth information can be obtained. Compared with obtaining RGB images based on RGB information, there is no difference in viewing angle, and there is no need to align the depth image and RGB image when image fusion is performed, which can effectively save computing resources and time.

It should be understood that the size of the sequence number of each step in the above-mentioned embodiment does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiment of the present application.

As shown in FIG. 6, an embodiment of the present application provides an imaging ranging system 1000, including:

the light emitter 200, for emitting infrared light and visible light to the object 2000;

Imaging ranging sensor 100;

The controller 300 is connected to the light transmitter 200 and the imaging ranging sensor 100 respectively, and is used for executing the imaging ranging method in the above-mentioned embodiment.

In application, the imaging ranging system at least includes a controller, a light emitter and an imaging ranging sensor, and may also include collimating optical elements and diffractive optical elements (DOE) covering the light emitter, covering imaging ranging The sensor's focusing lens or microlens array, etc. The collimating optical element is used for collimating the light emitted by the light emitter, and the diffractive optical element is used for diffracting the light. A lens or microlens array is used to focus the light reflected by the object on the photosensitive surface of the imaging ranging sensor. The controller is used to control the light emitter and the imaging ranging sensor to turn on or off, and can also adjust the light intensity of the light emitted by the light emitter. An object can be any object in free space that reflects light.

In applications, the light transmitter can be set as a laser, light emitting diode (LED), laser diode (LD), edge-emitting laser (EEL), etc. according to actual needs. The laser may specifically be a Vertical-Cavity Surface-Emitting Laser (VCSEL). The optical transmitter may be a tunable device.

In application, the controller may include a signal amplifier, a time-to-digital converter (TDC), an analog-to-digital converter (Analog-to- Digital Converter, ADC), etc. The time-to-speed converter connected to each pixel is connected to a histogram circuit.

In an application, the controller may also include a Central Processing Unit (CPU), other general-purpose controllers, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a ready-made Field-Programmable Gate Array (FPGA), other programmable logic devices, discrete gates, transistor logic devices, or discrete hardware components, etc. The general purpose controller may be a microcontroller or any conventional controller or the like.

The imaging ranging sensor, method, and system provided by the embodiments of the present application can be applied to mobile phones, tablet computers, wearable devices, in-vehicle devices, augmented reality (AR) devices, virtual reality (VR) devices, For terminal devices such as notebook computers, netbooks, and personal digital assistants (personal digital assistants, PDAs), the embodiments of the present application do not impose any restrictions on the specific types of terminal devices.

Embodiments of the present application further provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by the controller, implements the steps in each of the foregoing imaging ranging method embodiments.

Embodiments of the present application further provide a computer program product, which, when the computer program product runs on a terminal device, enables the terminal device to implement the steps in each of the foregoing imaging ranging method embodiments.

In the foregoing embodiments, the description of each embodiment has its own emphasis. For parts that are not described or described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments. In the embodiments provided in this application, it should be understood that the disclosed system/terminal device and method may be implemented in other manners. For example, the system/terminal device embodiments described above are merely illustrative.

The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The recorded technical solutions are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application, and should be included in the application. within the scope of protection.

Claims

An imaging ranging sensor, characterized in that it includes a plurality of pixels arranged in an array, each of the pixels includes at least four sub-pixels arranged in an array, and each of the sub-pixels includes a single-photon photosensitive unit and a cover In the filter of the single-photon photosensitive unit, each of the pixels includes an infrared sub-pixel, a red sub-pixel, a green sub-pixel and a blue sub-pixel.
The imaging ranging sensor of claim 1, wherein each of the pixels comprises four of the infrared sub-pixels, three of the red sub-pixels, six of the green sub-pixels, and three of the blue subpixel;

Four of the infrared sub-pixels are arranged in a 2×2 array to form an infrared sub-pixel unit, and one of the red sub-pixels, two of the green sub-pixels, and one of the blue sub-pixels are arranged in a Bayer array to form a unit. Color sub-pixel unit;

Each of the pixels includes one of the infrared sub-pixel units and three of the color sub-pixel units.
The imaging ranging sensor according to claim 2, wherein the one infrared sub-pixel unit and the three color sub-pixel units are arranged in a 1×4, 4×1 or 2×2 array any of the forms.
The imaging ranging sensor according to claim 1, wherein each of the pixels comprises one of the infrared sub-pixels, one of the red sub-pixels, and one of the green sub-pixels arranged in a 2×2 array. and one of the blue subpixels.
The imaging ranging sensor according to claim 1, wherein each of the pixels comprises M×M infrared sub-pixels, M×M red sub-pixels, and M×M green sub-pixels pixel and M×M said blue sub-pixels;

The M×M infrared sub-pixels are arranged in an array to form an infrared sub-pixel unit, the M×M red sub-pixels are arranged in an array to form a red sub-pixel unit, and the M×M green sub-pixels are A green sub-pixel unit is arranged in an array form, and M×M blue sub-pixels are arranged in an array form to form a blue sub-pixel unit;

Each of the pixels includes one of the infrared sub-pixel units, one of the red sub-pixel units, one of the green sub-pixel units, and one of the blue sub-pixel units, and M is an integer greater than or equal to 2.
The imaging distance measuring sensor according to any one of claims 1-5, wherein the filter is a reflective filter or an absorption filter.
An imaging ranging method, characterized in that, implemented based on the imaging ranging sensor according to any one of claims 1 to 6, the method comprising:

acquiring a first light sensing signal output by each pixel and used to indicate the time when infrared light reflected by the object is received;

acquiring a second light sensing signal output by each pixel and used to indicate the time when the visible light reflected by the object is received;

Acquiring depth information of the object according to the first light sensing signal;

The RGB information of the object is acquired according to the second light sensing signal.
The imaging ranging method according to claim 7, wherein the imaging ranging method further comprises:

obtaining the light intensity of the infrared light reflected by the object according to the first light sensing signal;

The infrared image information of the object is acquired according to the light intensity of the infrared light reflected by the object.
The imaging ranging method according to claim 8, wherein the infrared image information includes a brightness value; and the acquiring the infrared image information of the object according to the light intensity of the infrared light reflected by the object comprises:

According to the light intensity of the infrared rays reflected by each reflective point on the object, the brightness value corresponding to each reflective point in the infrared image information of the object is acquired.
The imaging ranging method according to claim 8, wherein the obtaining the light intensity of the infrared light reflected by the object according to the first light sensing signal comprises:

obtaining the total number of photons or the total number of peak signals received by the imaging ranging sensor according to the first light sensing signal;

According to the second correspondence between the preset infrared light intensity and the total number of photons or the total peak signal times, the light intensity of the infrared light reflected by the object is acquired.
The imaging ranging method according to claim 10, wherein in the second correspondence between the preset infrared light intensity and the total number of photons or the total peak signal times, the infrared light reflected by the object is obtained. Before the light intensity of the light, including:

Obtaining the total number of photons or the total number of peak signals received by the imaging ranging sensor under different light intensities of the infrared rays reflected by the object;

According to the total number of photons or total peak signal times obtained under each light intensity of infrared light reflected by the object, a second correspondence between different infrared light intensities and total photons or total peak signal times is obtained.
The imaging ranging method according to claim 7, wherein the acquiring the RGB information of the object according to the second light sensing signal comprises:

obtaining the light intensity of the visible light reflected by the object according to the second light sensing signal;

The RGB information of the object is acquired according to the light intensity of the visible light reflected by the object.
The imaging ranging method according to claim 12, wherein the acquiring the RGB information of the object according to the light intensity of the visible light reflected by the object comprises:

According to the light intensity of the visible light reflected by each reflective point on the object, obtain the brightness value corresponding to each reflective point in the RGB information of the object;

According to the brightness value of each reflective point in the RGB information of the object and the color of each color pixel in its corresponding pixel, based on the calculation formula of tristimulus value, the RGB value of each reflective point is calculated, and the RGB value of each reflective point including the object is obtained. RGB information for RGB values.
The imaging ranging method according to claim 12 or 13, wherein the obtaining the light intensity of the visible light reflected by the object according to the second light sensing signal comprises:

obtaining the total number of photons or the total number of peak signals received by the imaging ranging sensor according to the second light sensing signal;

According to the preset first correspondence between the visible light intensity and the total number of photons or the total peak signal times, the light intensity of the visible light reflected by the object is acquired.
The imaging ranging method according to claim 14, wherein the acquiring, according to the second light sensing signal, the total number of peak signals received by the imaging ranging sensor comprises:

Through the time-to-digital converter connected to the color sub-pixels in each pixel, the time of flight is calculated according to the second light sensing signal output by each color sub-pixel and converted into a time code; the color sub-pixels include red sub-pixels, green subpixels and blue subpixels;

The time code output by the time-to-digital converter is saved by a histogram circuit connected with the time-to-digital converter, and a histogram is generated according to the time code saved in at least one emission period of the visible light;

Obtain the number of peak signals in the histogram, obtain the total number of peak signals in the histogram generated according to the second photosensitive signal output by the color sub-pixels in each pixel, and obtain the total number of peak signals.
The imaging ranging method according to claim 14, wherein in the first correspondence between the preset visible light intensity and the total number of photons or the total number of peak signals, the visible light reflected by the object is acquired the light intensity before including:

Obtaining the total number of photons or the total number of peak signals received by the imaging ranging sensor under different light intensities of the visible light reflected by the object respectively;

According to the total number of photons or total peak signal times obtained under each light intensity of visible light reflected by the object, a first correspondence between different visible light intensities and total photons or total peak signal times is obtained.
The imaging ranging method according to claim 7, wherein the imaging ranging method further comprises:

Based on the time of receiving the second light sensing signal and the time of emitting visible light to the object, the distance between each reflective point on the object and the imaging ranging sensor is calculated by using the time-of-flight method.
An imaging ranging system, characterized in that it includes:

Light emitters for emitting infrared and visible light to objects;

The imaging ranging sensor according to any one of claims 1 to 4;

A controller, connected to the light transmitter and the imaging ranging sensor, respectively, is configured to execute the imaging ranging method according to any one of claims 5 to 8.
The imaging ranging system according to claim 18, wherein the controller comprises: signal amplifiers, time-to-digital converters, and analog-to-digital converters equal to the number of pixels or single-photon photosensitive units in the imaging ranging sensor The converter, and the speed converter corresponding to each pixel are connected to a histogram circuit.
A computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a controller, the imaging ranging method according to any one of claims 7 to 17 is implemented A step of.