WO2021170114A1 - Depth image obtaining method and device, and display device - Google Patents

Abstract

The present disclosure provides a depth image obtaining method and device, and a display device, for use in an image depth sensing assembly. A filtering layer is provided on the outer part of the image depth sensing assembly. The method comprises: transmitting reference light having a preset phase to the filtering layer; obtaining at least four output images by receiving reflected light reflected by the reference light by means of an object; obtaining at least four grayscale images according to filtering parameters and the at least four output images; and obtaining a depth image according to the at least four grayscale images.

Description

Method, device and display device for acquiring depth image

This application claims the priority of the Chinese patent application with the application number 202010124155.1 filed on February 27, 2020, and the application title is "Method for Obtaining Depth Image and Display Device", the entire content of which is incorporated into this application by reference.

Technical field

The present disclosure relates to the field of display technology, and in particular to a method, device and display device for acquiring a depth image.

Background technique

Compared with the traditional ordinary camera, the depth camera can also obtain the depth information of the scene in the image (that is, the distance between the scene and the camera lens) while shooting the image. Because of this advantage, the depth camera can be used in gestures. It is widely used in technical fields such as interaction, stereo display, machine vision, and satellite remote sensing.

In related technologies, there are mainly two solutions for depth extraction using ordinary cameras: binocular vision and structured light.

Summary of the invention

The present disclosure provides a method, device and display device for acquiring a depth image.

In one aspect, a method for acquiring a depth image is provided for use in an image depth sensing component, wherein a filter layer is provided outside the image depth sensing component, and the method includes:

Emitting reference light with a preset phase to the filter layer;

By receiving reflected light, at least four output images are acquired, and the received phases of the at least four output images and the preset phases have different phase differences, and the reflected light is the reference light after the filtering. The light reflected by the object behind the layer;

Obtaining at least four grayscale images according to the filter parameters of the filter layer and the at least four output images;

Obtain a depth image according to the at least four grayscale images.

Optionally, before obtaining at least four grayscale images according to the filtering parameters of the filter layer and the at least four output images, the method further includes:

Obtain the filter parameters of the filter layer.

Optionally, the obtaining the filter parameters of the filter layer includes:

Obtain the undetermined filter parameter formula, the undetermined filter parameter formula includes:

Where H is the transfer function matrix of the filter layer, H ^* is the adjoint matrix of the transfer function matrix H, H _w is the undetermined filter parameter, and ε is the undetermined coefficient;

The at least four output images are respectively convolved with the pending filter parameters to obtain at least four grayscale images, and it is determined that the definition of the at least four grayscale images is the highest corresponding to at least four first pending coefficient;

Determining an average value of the at least four first undetermined coefficients as the target coefficient;

The target coefficient is taken as the undetermined coefficient into the undetermined filter parameter formula to obtain the filter parameter.

Optionally, the obtaining at least four grayscale images according to the filtering parameters and the at least four output images includes:

The at least four output images are respectively convolved with the filter parameter to obtain the at least four grayscale images.

Optionally, the different phase differences between the received phases of the at least four output images and the preset phases are an arithmetic sequence with a difference of x, where x×n=360°, and the n Is the total number of output images.

Optionally, the number of the at least four output images is four, and the four output images include a first output image, a second output image, a third output image, and a fourth output image, wherein:

The phase difference between the reception phase of the first output image and the preset phase is 0°,

There is a 90° phase difference between the reception phase of the second output image and the preset phase,

There is a phase difference of 180° between the reception phase of the third output image and the preset phase,

There is a phase difference of 270° between the reception phase of the fourth output image and the preset phase.

Optionally, the at least four grayscale images are a first grayscale image corresponding to the first output image, a second grayscale image corresponding to the second output image, and a first grayscale image corresponding to the third output image. The three grayscale images and the fourth grayscale image corresponding to the fourth output image, the obtaining a depth image according to the at least four grayscale images includes:

Acquiring the gray value of each pixel in each gray image in the first gray image, the second gray image, the third gray image, and the fourth gray image;

The depth image is obtained according to the gray value of each pixel and the depth image formula, and the depth image formula includes:

Wherein, A ₁ , A ₂ , A ₃ and A ₄ are respectively the same position in the first gray scale image, the second gray scale image, the third gray scale image, and the fourth gray scale image , L is the depth of field parameter corresponding to each pixel in the depth image, φ is the phase difference corresponding to each pixel in the depth image, C is the speed of light, π is the circumference of the circle, and f is the The frequency of the reference light.

Optionally, before obtaining at least four grayscale images according to the filtering parameters of the filter layer and the at least four output images, the method further includes:

Obtain the undetermined filter parameter formula, the undetermined filter parameter formula includes:

Where H is the transfer function matrix of the filter layer, H ^* is the adjoint matrix of the transfer function matrix H, H _w is the undetermined filter parameter, and ε is the undetermined coefficient;

The at least four output images are respectively convolved with the pending filter parameters to obtain at least four grayscale images, and it is determined that the definition of the at least four grayscale images is the highest corresponding to at least four first pending coefficient;

Determining an average value of the at least four first undetermined coefficients as the target coefficient;

Bringing the target coefficient as the undetermined coefficient into the undetermined filter parameter formula to obtain the filter parameter;

The obtaining at least four grayscale images according to the filtering parameters and the at least four output images includes:

Convolve the at least four output images with the filter parameters to obtain the at least four grayscale images;

The number of the at least four output images is four, and the four output images include a first output image, a second output image, a third output image, and a fourth output image, wherein:

The phase difference between the reception phase of the first output image and the preset phase is 0°,

There is a 90° phase difference between the reception phase of the second output image and the preset phase,

There is a phase difference of 180° between the reception phase of the third output image and the preset phase,

A phase difference of 270° between the reception phase of the fourth output image and the preset phase;

The at least four grayscale images are a first grayscale image corresponding to the first output image, a second grayscale image corresponding to the second output image, and a third grayscale image corresponding to the third output image And a fourth grayscale image corresponding to the fourth output image, the obtaining a depth image according to the at least four grayscale images includes:

Acquiring the gray value of each pixel in each gray image in the first gray image, the second gray image, the third gray image, and the fourth gray image;

The depth image is obtained according to the gray value of each pixel and the depth image formula, and the depth image formula includes:

Wherein, A ₁ , A ₂ , A ₃ and A ₄ are respectively the same position in the first gray scale image, the second gray scale image, the third gray scale image, and the fourth gray scale image , L is the depth of field parameter corresponding to each pixel in the depth image, φ is the phase difference corresponding to each pixel in the depth image, C is the speed of light, π is the circumference of the circle, and f is the The frequency of the reference light.

In another aspect, a display device is provided. The display device includes a display panel and an image depth sensing component, the display panel includes a filter layer, and the image depth sensing component is used to perform the above-mentioned depth image acquisition method.

Optionally, the image depth sensing component is located outside the display panel.

Optionally, the image depth sensing component is located inside the display panel.

Optionally, the display panel includes a laminated transparent substrate, the filter layer, and a transparent packaging film layer, and the image depth sensing component is located on a side of the transparent substrate away from the transparent packaging film layer.

Optionally, the filter layer includes a stacked light-emitting unit layer and a mask layer, and the mask layer is located on a side of the filter layer close to the image depth sensing component.

Optionally, the material of the mask layer includes a molybdenum metal material.

Optionally, the display panel further includes an anode conductive layer, and the anode conductive layer and the mask layer are an integral structure.

In another aspect, a depth image acquisition device is provided, which includes an image depth sensing component and a filter layer located outside the image depth sensing component;

The image depth sensing component is configured to emit reference light with a preset phase to the filter layer;

The image depth sensing component is configured to obtain at least four output images by receiving reflected light, and the received phases of the at least four output images and the preset phases have different phase differences, and the reflection Light is the light reflected by the object after the reference light passes through the filter layer;

The image depth sensing component is configured to obtain at least four grayscale images according to the filter parameters of the filter layer and the at least four output images;

The image depth sensing component is used to obtain a depth image based on the at least four gray-scale images.

Optionally, the image depth sensing component is used to obtain a undetermined filter parameter formula, and the undetermined filter parameter formula includes:

Where H is the transfer function matrix of the filter layer, H ^* is the adjoint matrix of the transfer function matrix H, H _w is the undetermined filter parameter, and ε is the undetermined coefficient;

The image depth sensing component is configured to convolve the at least four output images with the undetermined filter parameter to obtain at least four grayscale images, and determine that the definition of the at least four grayscale images is the highest At least four first undetermined coefficients corresponding to time;

The image depth sensing component is used to determine the average value of the at least four first undetermined coefficients as the target coefficient;

The image depth sensing component is used to bring the target coefficient as the undetermined coefficient into the undetermined filter parameter formula to obtain the filter parameter.

Optionally, the number of the at least four output images is four, and the four output images include a first output image, a second output image, a third output image, and a fourth output image, wherein:

The phase difference between the reception phase of the first output image and the preset phase is 0°,

There is a 90° phase difference between the reception phase of the second output image and the preset phase,

There is a phase difference of 180° between the reception phase of the third output image and the preset phase,

There is a phase difference of 270° between the reception phase of the fourth output image and the preset phase.

Optionally, the at least four grayscale images are a first grayscale image corresponding to the first output image, a second grayscale image corresponding to the second output image, and a first grayscale image corresponding to the third output image. A three grayscale image and a fourth grayscale image corresponding to the fourth output image,

The image depth sensing component is used to obtain each of the first grayscale image, the second grayscale image, the third grayscale image, and the fourth grayscale image. The gray value of each pixel;

The image depth sensing component is configured to obtain the depth image according to the gray value of each pixel and a depth image formula, and the depth image formula includes:

Wherein, A ₁ , A ₂ , A ₃ and A ₄ are respectively the same position in the first gray scale image, the second gray scale image, the third gray scale image, and the fourth gray scale image , L is the depth of field parameter corresponding to each pixel in the depth image, φ is the phase difference corresponding to each pixel in the depth image, C is the speed of light, π is the circumference of the circle, and f is the The frequency of the reference light.

Optionally, the image depth sensing component is configured to convolve the at least four output images with the filter parameters to obtain the at least four grayscale images.

Description of the drawings

FIG. 1 is a flowchart of a method for acquiring a depth image provided by an embodiment of the disclosure.

Fig. 2 is another flow chart of a method for acquiring a depth image provided by an embodiment of the disclosure.

FIG. 3 is a diagram of the main circuit composition of an image depth sensing component provided by an embodiment of the disclosure.

FIG. 4 is a waveform diagram of receiving pixels of an image depth sensing component provided by an embodiment of the disclosure.

Fig. 5 is a flow chart of obtaining filter parameters of a filter layer in the embodiment shown in Fig. 2.

FIG. 6 is a schematic diagram of a display device provided by an embodiment of the disclosure.

FIG. 7 is a top view of a pattern of a mask layer in a display device provided by an embodiment of the disclosure.

Detailed ways

To make the advantages and advantages of the present disclosure clearer, the following describes the embodiments of the present disclosure in detail with reference to the accompanying drawings.

As shown in FIG. 1, it is a flowchart of a method for acquiring a depth image provided by an embodiment of the present disclosure, which is used in an image depth sensing component (the image depth sensing component may also be referred to as an image depth sensing device, or Image depth sensing structure, the embodiment of the present disclosure does not limit this) In the image depth sensing component, a filter layer is provided outside the image depth sensing component, and the method includes:

In step 101, reference light of a preset phase is emitted to the filter layer.

Wherein, the reference light is used to pass through the filter layer and be directed toward the object, and the object may be an object located on the side of the filter layer away from the image depth sensing component.

In step 102, at least four output images are acquired by receiving the reflected light.

Wherein, the reflected light is the light reflected by the object after the reference light passes through the filter layer, and the received phases and the preset phases of the at least four output images have different phase differences respectively.

In step 103, at least four grayscale images are obtained according to the filter parameters of the filter layer and at least four output images.

In step 104, a depth image is obtained based on at least four grayscale images.

In summary, the depth image acquisition method provided by the embodiments of the present disclosure can emit reference light with a preset phase through the image depth sensor component, and receive the reflected light reflected by the reference light by the object, and obtain at least four received phase sums. The output images with different preset phases of the reference light are restored, and at least four output images are restored according to the filter parameters of the filter layer to obtain at least four gray-scale images, and the depth image is obtained from the at least four gray-scale images. Since this method replaces the optical lens in the traditional depth camera with a filter layer, a complicated optical lens set is not needed, so that the method can obtain a depth image with a relatively simple structure.

On this basis, the depth image acquisition method can be conveniently applied to the display device.

As shown in FIG. 2, which is a flowchart of a method for acquiring a depth image provided by an embodiment of the present disclosure, which is used in an image depth sensing component, and a filter layer is provided outside the image depth sensing component, and the method includes:

In step 201, the image depth sensing component emits reference light of a preset phase to the filter layer.

Wherein, the reference light is used to pass through the filter layer and be directed toward the object, and the object may be an object located on the side of the filter layer away from the image depth sensing component.

It can be understood that the reference light is emitted by the image depth sensing component, which is a function of the image depth sensing component.

In an optional manner, the preset phase can be determined in advance, and after the preset phase is determined, the reference light is phase modulated in advance to make it have the preset phase, which can facilitate the subsequent phase difference calculate.

The method provided by the embodiments of the present disclosure can be applied to a depth image acquisition device, which includes the above-mentioned image depth sensing component and a filter layer.

The object within the range that the reference light can irradiate can be the target object that is expected to obtain depth information. The depth of a certain point of the object refers to the distance between the point and the depth image acquisition device, and the different depths of multiple points are analyzed , The posture of the object can be obtained, and the trend of the depth of multiple points over time can be analyzed, and the movement trend of the object can be obtained, so as to analyze the change of the overall state of the object.

In step 202, the image depth sensing component acquires at least four output images by receiving the reflected light.

Wherein, the reflected light is the light reflected by the object after the reference light passes through the filter layer, and the received phases of at least four output images and the preset phases of the reference light have different phase differences.

It can be understood that the preset phase refers to the specific phase obtained by modulating the reference light when the image depth sensing component emits the reference light, and is the phase of the reference light.

The reception phase refers to the specific phase of the light received by the image depth sensor component, and the image depth sensor component can be set to only receive light whose phase is the reception phase.

The reference light is reflected by the object and returned to the depth image acquisition device. After being filtered by the filter layer, the image depth sensor component receives and forms an output image. Theoretically, due to the difference in the receiving phase, the received output image can be countless Therefore, there is a different phase difference between the received phase of these output images and the preset phase of the reference light. When the number of acquired output images is less than four, only approximate calculations can be used to obtain grayscale images based on the output images and depth images based on the degree images, and the accuracy of the final acquired depth images is not high. In order to improve accuracy, the method provided by the embodiment of the present disclosure selects to acquire at least four output images, and the received phases of the at least four output images and the preset phases of the reference light have different phase differences.

The different phase differences between the received phases of the at least four output images and the preset phases of the reference light need to meet a certain distribution law, so that a grayscale image can be obtained from the output image, and a depth image can be obtained from the grayscale image.

In an optional manner, the different phase differences between the received phases of the at least four output images and the preset phases of the reference light are equally divided by 360°. That is, the different phase differences between the received phase and the preset phase of the at least four output images are an arithmetic sequence with a difference of x, where x×n=360°, and n is the total number of output images.

In an optional manner, the number of at least four output images is four, and the four output images include the first output image, the second output image, the third output image, and the fourth output image, where:

There is no phase difference between the received phase of the first output image and the preset phase of the reference light, that is, the phase difference is 0°.

There is a 90° phase difference between the reception phase of the second output image and the preset phase of the reference light.

There is a phase difference of 180° between the reception phase of the third output image and the preset phase of the reference light.

There is a phase difference of 270° between the reception phase of the fourth output image and the preset phase of the reference light.

It is understandable that the four output images are all grayscale images.

Optionally, the step of acquiring the output image in step 202 may be completed by the photosensitive circuit in the image depth sensing component. Each pixel of the image depth sensing component is equipped with a photosensitive circuit, and the photosensitive circuit is equipped with a capacitor. The change of the capacitance value can reflect the change of the amplitude value of the light signal of each pixel, which is reflected in the grayscale image. The gray value of each pixel is calculated.

The following is a detailed introduction to the principle of the image depth sensing component:

The structure of the photosensitive circuit of the image depth sensing component may include the structure shown in FIG. 3, DM1 and DM2 are respectively two photoelectric switches arranged on one pixel in the image depth sensing component, which can sense light. Corresponding to switches DM1 and DM2 are capacitors Ca and Cb. When the gate voltage of DM1 is at the first level and the gate voltage of DM2 is at the second level (the first level is high relative to the second level), the switch of DM1 is turned on and DM2 is turned off, that is, the current direction Ca storage, and vice versa, storage to Cb. The voltage change generated on the capacitor Ca is a, and the voltage change generated on the capacitor Cb is b. In addition to the above, Vrst in Figure 3 is the reset voltage, and Mr1 and Mr2 are transistors used for voltage reset. In addition, Fig. 3 also includes a constant current source CCS and a ground terminal GND.

The receiving pixel waveform diagram of the image depth sensor component is shown in Figure 4. The measurement of the transmission and reception timing is divided into four types, namely: the transmission and reception are completely synchronized (that is, the phase difference is 0 degrees), and the difference between the transmission and reception is 90 degrees. Transmitting and receiving are 180 degrees apart, and transmitting and receiving are 270 degrees apart.

Collect the output images under these four conditions respectively. There are four images, namely the first output image, the second output image, the third output image and the fourth output image. In Figure 4, clk is the clock signal, and U1M is the emission waveform. , The high level is the stage of transmitting the ultrasonic wave, and DM1 and DM2 are the turn-off timings of the two switches respectively. It can be seen from Figure 4 that in the upper left picture in Fig. 4, the transmit and receive phases are synchronized, that is, 0 degree reception, and then the lower left picture shows the phase difference of 90 degrees, and the upper right shows the phase difference of 180. In the case of degrees, the lower right is the case where the phase difference is 270 degrees. From the circuit shown in Figure 3 and the pixel waveform diagram shown in Figure 4, the following formula can be obtained:

a0-b0=(VAa+VOa+VR-Ga*2cos(Trt/T))-(VAb+VOb+VR+Gb*2cos(Trt/T)).

a180-b180=(VAa+VOa+VR+Ga2cos(Trt/T))-(VAb+VOb+VR-Gb*2cos(Trt/T)).

a90-b90=(VAa+VOa+VR-Ga*2cos(Trt/T))-(VAb+VOb+VR+Gb*2cos(Trt/T)).

a270-b270=(VAa+VOa+VR+Ga2cos(Trt/T))-(VAb+VOb+VR-Gb*2cos(Trt/T)).

Among them, a0 is the voltage change generated on the capacitor Ca when the transmission and reception are 0 degrees apart, a90 is the voltage change generated on the capacitor Ca when the transmission and reception are 90 degrees apart, and a180 is the voltage change generated on the capacitor Ca when the transmission and reception are 180 degrees apart The voltage change, a270 is the voltage change generated on the capacitor Ca when the transmit and receive are 270 degrees different.

b ₀ is the voltage change generated on the capacitor Cb when the transmitting and receiving are 0 degrees, b ₉₀ is the voltage change generated on the capacitor Cb when the transmitting and receiving are 90 degrees, and b ₁₈₀ is the voltage change on the capacitor Cb when the transmitting and receiving are 180 degrees different. The voltage change of b ₂₇₀ is the voltage change generated on the capacitor Cb when the difference between transmitting and receiving is 270 degrees.

V _Aa and ambient light variation V _Ab, V _0a and V _0b initial bias, V _R is the initial potential (equal to the value Vrst when the reset Vrst), Ga and Gb are signal gain, T _rt after the light switch is turned on The time stored on the capacitor, T is the clk clock cycle.

Simplifying and combining the above four formulas, the following formula can be obtained:

a ₀ -b ₀ ＝(V _Aa +V _Oa +V _R )-(V _Ab +V _Ob +V _R )-(Ga+Gb)*2cos(T _rt /T)

a ₁₈₀ -b ₁₈₀ =(V _Aa +V _Oa +V _R )-(V _Ab +V _Ob +V _R )+(Ga+Gb)*2cos(T _rt /T)

a ₉₀ -b ₉₀ =(V _Aa +V _Oa +V _R )-(V _Ab +V _Ob +V _R )-(Ga+Gb)*2sin(T _rt /T)

a ₂₇₀ -b ₂₇₀ = (V _Aa +V _Oa +V _R )-(V _Ab +V _Ob +V _R )+(Ga+Gb)*2sin(T _rt /T)

Among them, a ₀ -b ₀ is the gray value of a certain pixel on the first output image when the difference between transmitting and receiving is 0 degrees, that is, A ₁ .

a _90- b ₉₀ is the gray value of a certain pixel on the second output image when the transmit and receive are 90 degrees apart, that is, A ₂ .

a ₁₈₀ -b ₁₈₀ is the gray value of a certain pixel on the third output image when the transmission and reception differ by 180 degrees, that is, A ₃ .

a _270- b ₂₇₀ is the gray value of a certain pixel on the fourth output image when the difference between the emission and the reception is 270 degrees, that is, A ₄ .

In this way, the gray value of each pixel in each output image can be obtained through the photosensitive circuit in each pixel.

In step 203, the image depth sensing component obtains the filter parameters of the filter layer.

In an optional manner, as shown in FIG. 5, obtaining the filter parameters of the image of the object by the filter layer in step 203 includes:

Step 2031: The image depth sensing component obtains the undetermined filter parameter formula.

Among them, the undetermined filter parameter formula includes:

Among them, H is the transfer function matrix of the filter layer, H ^* is the adjoint matrix of the transfer function matrix H, H _w is the undetermined filter parameter, and ε is the undetermined coefficient.

It can be understood that the transfer function matrix H in the undetermined filter parameter formula is obtained by actually measuring the filter layer after the filter layer in the depth image acquisition device is set. When the filter layer is determined, the undetermined filter parameter formula The transfer function matrix H in is a known quantity.

Accordingly, the transfer function matrix H after the matrix H ^* is also accompanied by the transfer function matrix H can be calculated accordingly, and thus determined filter parameters in the transfer function matrix equation adjoint matrix H ^* H is a known quantity. In the same way, |H| ² in the undetermined filter parameter formula is also a known quantity. Therefore, the unknown quantity is the undetermined coefficient ε and the undetermined filter parameter H _w , and the undetermined filter parameter is expressed by the undetermined coefficient ε and multiple known quantities using the above-mentioned undetermined filter parameter formula.

In step 2032, the image depth sensing component convolves at least four output images with undetermined filter parameters to obtain at least four grayscale images, and determines that the resolution of the at least four grayscale images is at least the fourth corresponding to the highest resolution. A coefficient to be determined.

In the process of convolving at least four output images with undetermined filter parameters to obtain at least four grayscale images, since the undetermined filter parameters are expressed by the undetermined coefficient ε and multiple known quantities through the above-mentioned undetermined filter parameter formula, There is a undetermined unknown quantity in the undetermined filter parameters, namely the undetermined coefficient ε.

In an optional way, the undetermined coefficient ε can be assigned by an iterative method first, and the corresponding coefficient ε after the assignment is used to represent the undetermined filter parameter, and it is convolved with at least four output images to obtain at least four Create a grayscale image and calculate the sharpness of the grayscale image corresponding to each assignment.

Optionally, the sharpness can be calculated using a gradient algorithm. The value of the coefficient ε is different, and the definition of the corresponding gray image obtained after convolution is also different. After at least four output images are respectively convolved, there must be at least four gray images with the highest definition respectively, which can make at least When the definitions of the four grayscale images are all the highest, the corresponding at least four coefficients are the first undetermined coefficients. It can be understood that the number of the first undetermined coefficients is the same as the number of output images.

Step 2033: The image depth sensing component determines the average value of the at least four first undetermined coefficients as the target coefficient.

In an optional manner, the average value may be an arithmetic average or geometric average of at least four first undetermined coefficients.

In step 2034, the image depth sensing component brings the target coefficient into the undetermined filter parameter formula to obtain the filter parameter.

It is understandable that when the clearest grayscale image is obtained through the first undetermined coefficient, it is equivalent to proper sharpening of the output image, for example, proper sharpening of the edge of the output image. Therefore, the most The clear grayscale image may have some distortion compared with the real image. The clearest grayscale image may not be the most suitable for analysis and obtain the depth image. Therefore, at least four first undetermined coefficients can be averaged Value processing is used to obtain the final target coefficient, and the target coefficient is used as the coefficient in the undetermined filter parameter formula, which can ensure the accuracy of the depth image while ensuring the definition of the gray image.

In step 204, the image depth sensing component obtains at least four grayscale images according to the filter parameters and at least four output images.

In an optional manner, obtaining at least four grayscale images according to the filter parameters and at least four output images includes:

At least four output images are convolved with the filter parameters to obtain at least four grayscale images.

The specific convolution formula is as follows:

S=D*H _w

Among them, D represents the output image, _Hw represents the filter parameter, and S represents the grayscale image.

Since the traditional depth camera uses an optical lens group for optical transformation, the light reflected from the actual scene passes through the optical lens group to obtain the resulting image. However, the embodiment of the present disclosure uses a filter layer to replace the traditional optical lens set. Therefore, the light reflected from the actual scene passes through the filter layer to obtain the resulting image. The above formula indicates that the filter parameter H _{w of the} filter layer is used to In the process of restoring the resulting output image D and obtaining the grayscale image S, it can be understood that the filter parameter H _w of the filter layer is actually equivalent to the inverse transfer function of the filter layer.

Optionally, the image restoration process in step 204 can be completed by using a processor integrated in the image depth sensing component without using other optical elements for optical transformation. After the restored grayscale image is obtained through step 204, the grayscale value of each pixel in each grayscale image is also obtained at the same time, so that these grayscale values can be used to calculate the depth value to obtain the depth image.

In step 205, the image depth sensing component obtains a depth image based on at least four grayscale images.

In an optional manner, the at least four grayscale images are a first grayscale image corresponding to the first output image, a second grayscale image corresponding to the second output image, and a third grayscale image corresponding to the third output image. The image and the fourth grayscale image corresponding to the fourth output image, obtaining a depth image based on at least four grayscale images includes:

Obtain the grayscale value of each pixel in each grayscale image in the first grayscale image, the second grayscale image, the third grayscale image, and the fourth grayscale image.

The depth image is obtained according to the gray value of each pixel and the depth image formula. The depth image formula includes:

Among them, φ is the intermediate quantity, and A ₁ , A ₂ , A ₃ and A ₄ are the values of pixels at the same position in the first, second, third, and fourth grayscale images, respectively. Gray value, L is the depth of field parameter corresponding to each pixel in the depth image, φ is the phase difference corresponding to each pixel in the depth image, C is the speed of light, π is the circumference of the circle, and f is the frequency of the reference light.

It is understandable that step 205 is also a calculation step implemented by the processor in the image depth sensing component, and the photosensitive circuit in the image depth sensing component does not need to be used. The photosensitive circuit in the image depth sensing component can be Play a role in step 202.

When the depth image acquisition method is applied to a display device (the display device includes a display panel):

In an optional manner, the depth sensing component and the display panel can be powered by two separate driving power sources.

In an optional manner, the depth sensing component and the display panel may also share the same driving power supply, and the depth sensing component and the display panel may respectively correspond to two different power supply modules in the same driving power supply.

In summary, the depth image acquisition method provided by the embodiments of the present disclosure can emit reference light with a preset phase through the image depth sensor component, and receive the reflected light reflected by the reference light by the object, and obtain at least four received phase sums. The output images with different preset phases of the reference light are restored, and at least four output images are restored according to the filter parameters of the filter layer to obtain at least four gray-scale images, and the depth image is obtained from the at least four gray-scale images. Since this method replaces the optical lens in the traditional depth camera with a filter layer, a complicated optical lens set is not needed, so that the method can obtain a depth image with a relatively simple structure.

On this basis, the depth image acquisition method can be conveniently applied to the display device.

As shown in FIG. 6, it is a structural diagram of a display device provided by an embodiment of the present disclosure. The display device includes a display panel 31 and an image depth sensing component 32. The display panel 31 includes a filter layer 311 and an image depth sensing component. 32 is used to perform the depth image acquisition method.

In an optional manner, the image depth sensing component 32 is located outside the display panel 31.

In an optional manner, if the image depth sensing component 32 is made by using a silicon-based process, the image depth sensing component 32 can be directly arranged inside the display panel 31 and integrated with the display panel 31.

In an optional manner, the display panel 31 includes a transparent substrate 312, a filter layer 311 and a transparent packaging film layer 313 which are stacked, and the image depth sensor component 32 is located on the side of the transparent substrate 312 away from the transparent packaging film layer 313.

In an optional manner, the depth sensing component 32 and the display panel 31 may be powered by two separate driving power sources.

In an optional manner, the depth sensing component 32 may also share the same driving power source with the display panel 31, and the depth sensing component 32 and the display panel 31 respectively correspond to two different power modules in the same driving power source.

In an optional manner, the filter layer 311 includes a stacked light-emitting unit layer 3111 and a mask layer 3112, and the mask layer 3112 is located on the side of the filter layer 311 close to the image depth sensing component 32. Disposing the mask layer 3112 on the side of the filter layer 311 close to the image depth sensor component 32 can increase the light output rate of the display device and prevent the mask layer 3112 from blocking the light emitted from the light emitting unit layer 3111 to the light emitting side of the display panel 31.

Since the transparent substrate 312 and the transparent encapsulation film layer 313 in the display panel 31 are made of transparent materials, the light transmittance is improved, so that the light reflected from the object can be irradiated to the image depth sensing component 32 more effectively.

In an optional manner, the material of the mask layer 3112 includes a molybdenum metal material.

In an optional manner, the specific pattern of the mask layer 3112 is shown in FIG. 7, and FIG. 7 is a top view of the mask layer 3112. The black area in FIG. 7 is a blocking area, which can block light, and the white area is a hollow area, which can transmit light. Since the mask layer 3112 has a pattern as shown in FIG. 7, the mask layer 3112 has a light filtering effect. Encode the transmitted light. The specific pattern can be set according to actual filtering and coding requirements, so that the mask layer 3112 has a specific coding function. The light-emitting unit layer 3111 may include at least one organic light-emitting diode (OLED) 31111, and the light-emitting unit layer 3111 is used to emit red light, green light, or blue light.

The number of image depth sensors included in the image depth sensing assembly 32 is the same as the number of organic light emitting diodes 31111.

It can be understood that since the organic light emitting diode 31111 is not completely transparent, the light-emitting unit layer 3111 is not completely transparent, and will play a certain passive filtering role. The light-emitting unit layer 3111 and the mask layer 3112 are stacked to form the filter layer 311. Therefore, the filter layer 311 is an incompletely transparent hierarchical structure in the display panel and has a filtering effect. After the light reflected from the object enters the display panel, it first passes through the light emitting unit layer 3111, the light is encoded once, and then passes through the mask layer 3112, the light is encoded a second time, and all the light reflected from the object is After encoding twice, the output image can be obtained. Suitable light-emitting unit layer 3111 and mask layer 3112 can be selected according to actual imaging requirements, so that the combination of light-emitting unit layer 3111 and mask layer 3112 has filter coding characteristics suitable for imaging and has sufficient coding capability.

Since the shape of the organic light-emitting diode 31111 is relatively fixed, the single light-emitting unit layer 3111 only has a certain passive encoding capability. Based on this, a mask layer 3112 is also provided in the embodiment of the present disclosure to cooperate with the light-emitting unit layer 3111 , To effectively and controllably encode the light.

The light reflected by the object is generally divergent and chaotic, and it is not possible to directly use photosensitive equipment for effective collection. Therefore, a set of optical lenses is set in a traditional camera to converge and converge the divergent and chaotic light reflected by the object. Encoding, and then restore the image obtained after passing through the optical lens to obtain the desired image.

By adopting the above-mentioned setting method provided by the embodiment of the present disclosure, after the chaotic light reflected by the object is encoded by the filter layer 311, the gray-scale image is also obtained by the restoration algorithm in the subsequent steps. That is to say, the filter layer 311 can also encode chaotic light, which is a good substitute for the optical lens in a traditional camera, so that the display device can have a higher degree of integration and a smaller volume.

The encoding characteristics of the filter layer 311 are expressed by a transfer function matrix when performing image restoration. Different filter layers 311 can have different transfer function matrices. In an optional manner, you can select according to actual imaging requirements. A filter layer 311 with a corresponding transfer function matrix.

In an optional manner, the display panel further includes an anode conductive layer, and the anode conductive layer and the mask layer 3112 are an integrated structure, that is, the anode conductive layer and the mask layer 3112 may have the same structure, or the anode conductive layer may be an anode conductive layer. The layer and the mask layer 3112 may be two connected structures.

Using the anode conductive layer directly as the mask layer 3112 can reduce the number of layers of the display device, simplify the structure, and reduce the cost.

In conjunction with FIG. 6, the working process of the display device provided by the present disclosure may include:

The image depth sensor component 32 emits reference light, passes through the display panel 31, illuminates the object and returns. The returned light passes through the display panel 31 and is collected by the image depth sensor component 32. Obtain four output images with specific phases from the multiple output images, and perform image restoration on these four output images to obtain four grayscale images, and then obtain a depth image based on the restored four grayscale images.

With the above-mentioned setting method, after the chaotic light reflected by the object is coded by the filter layer 311, the final gray-scale image is also subsequently obtained by the restoration algorithm. The filter layer 311 replaces the optical lens in the traditional camera, so that the display device can have a higher integration level and a smaller volume.

It is understandable that the photosensitive circuit of the image depth sensing component 32 can only obtain unreduced output images, but the image depth sensing component 32 also has the function of restoring the image, and the image depth sensing component 32 further performs the output The image is restored to obtain a grayscale image, and the depth information is determined according to the grayscale image. That is, although the image depth sensing component 32 has collected the output image, the final depth information is determined based on the restored grayscale image. The two steps of restoring the image and obtaining the grayscale image are both completed by the processor integrated in the image depth sensing component 32.

The beneficial effects brought by the technical solutions provided by the present disclosure include at least:

The display device provided by the present disclosure includes a display panel 31 and an image depth sensing component 32, the display panel 31 includes a filter layer 311, and the image depth sensing component 32 is used to implement a depth image acquisition method. Since the filter layer 311 is used to replace the optical lens in the traditional depth camera, there is no need to set up a complicated optical lens group, so that the filter layer and the image depth sensing component can be well integrated in the display device, so that the display device can be used to obtain the depth image .

In addition, embodiments of the present disclosure also provide a depth image acquisition device, including an image depth sensing component and a filter layer located outside the image depth sensing component; the depth image acquisition device can be used to implement the depth image acquisition device provided in the above embodiments The method of obtaining the depth image.

The image depth sensing component is used to emit reference light with a preset phase to the filter layer;

The image depth sensing component is used to obtain at least four output images by receiving the reflected light. The received phase and the preset phase of the at least four output images have different phase differences, respectively, and the reflected light is the reference light after passing through the filter layer. Light reflected by an object;

The image depth sensing component is used to obtain at least four grayscale images according to the filter parameters of the filter layer and at least four output images;

The image depth sensing component is used to obtain a depth image based on at least four gray-scale images.

Optionally, the image depth sensing component is used to obtain the undetermined filter parameter formula, and the undetermined filter parameter formula includes:

Among them, H is the transfer function matrix of the filter layer, H ^* is the adjoint matrix of the transfer function matrix H, H _w is the undetermined filter parameter, and ε is the undetermined coefficient;

The image depth sensing component is used to convolve at least four output images with undetermined filter parameters to obtain at least four grayscale images, and determine that the resolution of the at least four grayscale images is at least the fourth corresponding to the highest resolution A undetermined coefficient;

The image depth sensing component is used to determine the average value of at least four first undetermined coefficients as the target coefficient;

The image depth sensing component is used to bring the target coefficient as the undetermined coefficient into the undetermined filter parameter formula to obtain the filter parameter.

Optionally, the number of at least four output images is four, and the four output images include the first output image, the second output image, the third output image, and the fourth output image, where:

The phase difference between the received phase of the first output image and the preset phase is 0°,

There is a 90° phase difference between the reception phase of the second output image and the preset phase,

There is a phase difference of 180° between the reception phase of the third output image and the preset phase,

There is a phase difference of 270° between the reception phase of the fourth output image and the preset phase.

Optionally, the at least four grayscale images are a first grayscale image corresponding to the first output image, a second grayscale image corresponding to the second output image, a third grayscale image corresponding to the third output image, and a fourth output image. The fourth grayscale image corresponding to the image,

The image depth sensing component is used to obtain the gray value of each pixel in each gray image in the first gray image, the second gray image, the third gray image, and the fourth gray image;

The image depth sensing component is used to obtain the depth image according to the gray value of each pixel and the depth image formula. The depth image formula includes:

Among them, A ₁ , A ₂ , A ₃ and A ₄ are respectively the gray scales of the pixels at the same position in the first gray scale image, the second gray scale image, the third gray scale image, and the fourth gray scale image, L Is the depth of field parameter corresponding to each pixel in the depth image, φ is the phase difference corresponding to each pixel in the depth image, C is the speed of light, π is the circumference of the circle, and f is the frequency of the reference light.

Optionally, the image depth sensing component is used to convolve at least four output images with filter parameters to obtain at least four grayscale images.

In addition, an embodiment of the present disclosure also provides a computer storage medium, the computer storage medium stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set consists of The processor loads and executes some steps of the depth image acquisition method provided in the above-mentioned embodiment, for example, step 103 to step 104 in the embodiment shown in FIG. 1 and step 203 to step in the embodiment shown in FIG. 2 205.

The above are only optional embodiments of the present disclosure and are not intended to limit the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the specification of the present disclosure. Within the scope of protection defined by the appended claims.

Claims (20)

1. A method for acquiring a depth image is used in an image depth sensing component, and a filter layer is provided outside the image depth sensing component, and the method includes:

   Emitting reference light with a preset phase to the filter layer;

   By receiving reflected light, at least four output images are acquired, and the received phases of the at least four output images and the preset phases have different phase differences, and the reflected light is the reference light after the filtering. The light reflected by the object behind the layer;

   Obtaining at least four grayscale images according to the filter parameters of the filter layer and the at least four output images;

   Obtain a depth image according to the at least four grayscale images.
2. The method according to claim 1, before obtaining at least four grayscale images according to the filtering parameters of the filter layer and the at least four output images, the method further comprises:

   Obtain the filter parameters of the filter layer.
3. The method according to claim 2, wherein said obtaining the filter parameters of the filter layer comprises:

   Obtain the undetermined filter parameter formula, the undetermined filter parameter formula includes:

   

   Where H is the transfer function matrix of the filter layer, H ^* is the adjoint matrix of the transfer function matrix H, H _w is the undetermined filter parameter, and ε is the undetermined coefficient;

   The at least four output images are respectively convolved with the pending filter parameters to obtain at least four grayscale images, and it is determined that the definition of the at least four grayscale images is the highest corresponding to at least four first pending coefficient;

   Determining an average value of the at least four first undetermined coefficients as the target coefficient;

   The target coefficient is taken as the undetermined coefficient into the undetermined filter parameter formula to obtain the filter parameter.
4. The method according to any one of claims 1 to 3, wherein the obtaining at least four grayscale images according to the filtering parameters and the at least four output images includes:

   The at least four output images are respectively convolved with the filter parameter to obtain the at least four grayscale images.
5. The method according to any one of claims 1 to 4, wherein the different phase differences between the received phases of the at least four output images and the preset phases are an arithmetic sequence with a difference of x, where x ×n=360°, the n is the total number of output images.
6. The method according to claim 5, wherein the number of the at least four output images is four, and the four output images include a first output image, a second output image, a third output image, and a fourth output image, wherein:

   The phase difference between the reception phase of the first output image and the preset phase is 0°,

   There is a 90° phase difference between the reception phase of the second output image and the preset phase,

   There is a phase difference of 180° between the reception phase of the third output image and the preset phase,

   There is a phase difference of 270° between the reception phase of the fourth output image and the preset phase.
7. The method according to claim 6, wherein the at least four grayscale images are a first grayscale image corresponding to the first output image, a second grayscale image corresponding to the second output image, and the third grayscale image. The third grayscale image corresponding to the output image and the fourth grayscale image corresponding to the fourth output image, the obtaining a depth image according to the at least four grayscale images includes:

   Acquiring the gray value of each pixel in each gray image in the first gray image, the second gray image, the third gray image, and the fourth gray image;

   The depth image is obtained according to the gray value of each pixel and the depth image formula, and the depth image formula includes:

   

   

   Wherein, A ₁ , A ₂ , A ₃ and A ₄ are respectively the same position in the first gray scale image, the second gray scale image, the third gray scale image, and the fourth gray scale image , L is the depth of field parameter corresponding to each pixel in the depth image, φ is the phase difference corresponding to each pixel in the depth image, C is the speed of light, π is the circumference of the circle, and f is the The frequency of the reference light.
8. The method according to claim 1, before obtaining at least four grayscale images according to the filtering parameters of the filter layer and the at least four output images, the method further comprises:

   Obtain the undetermined filter parameter formula, the undetermined filter parameter formula includes:

   

   Where H is the transfer function matrix of the filter layer, H ^* is the adjoint matrix of the transfer function matrix H, H _w is the undetermined filter parameter, and ε is the undetermined coefficient;

   The at least four output images are respectively convolved with the pending filter parameters to obtain at least four grayscale images, and it is determined that the definition of the at least four grayscale images is the highest corresponding to at least four first pending coefficient;

   Determining an average value of the at least four first undetermined coefficients as the target coefficient;

   Bringing the target coefficient as the undetermined coefficient into the undetermined filter parameter formula to obtain the filter parameter;

   The obtaining at least four grayscale images according to the filtering parameters and the at least four output images includes:

   Convolve the at least four output images with the filter parameters to obtain the at least four grayscale images;

   The number of the at least four output images is four, and the four output images include a first output image, a second output image, a third output image, and a fourth output image, wherein:

   The phase difference between the reception phase of the first output image and the preset phase is 0°,

   There is a 90° phase difference between the reception phase of the second output image and the preset phase,

   There is a phase difference of 180° between the reception phase of the third output image and the preset phase,

   A phase difference of 270° between the reception phase of the fourth output image and the preset phase;

   The at least four grayscale images are a first grayscale image corresponding to the first output image, a second grayscale image corresponding to the second output image, and a third grayscale image corresponding to the third output image And a fourth grayscale image corresponding to the fourth output image, the obtaining a depth image according to the at least four grayscale images includes:

   Acquiring the gray value of each pixel in each gray image in the first gray image, the second gray image, the third gray image, and the fourth gray image;

   The depth image is obtained according to the gray value of each pixel and the depth image formula, and the depth image formula includes:

   

   

   Wherein, A ₁ , A ₂ , A ₃ and A ₄ are respectively the same position in the first gray scale image, the second gray scale image, the third gray scale image, and the fourth gray scale image , L is the depth of field parameter corresponding to each pixel in the depth image, φ is the phase difference corresponding to each pixel in the depth image, C is the speed of light, π is the circumference of the circle, and f is the The frequency of the reference light.
9. A display device, the display device comprising a display panel and an image depth sensing component, the display panel comprising a filter layer, and the image depth sensing component is used to perform the depth image processing of any one of claims 1-8 Obtaining method.
10. 9. The display device of claim 9, wherein the image depth sensing component is located outside the display panel.
11. 9. The display device of claim 9, wherein the image depth sensing component is located inside the display panel.
12. The display device according to claim 10, wherein the display panel comprises a laminated transparent substrate, the filter layer and a transparent packaging film layer, and the image depth sensing component is located on the transparent substrate far away from the transparent packaging film layer. side.
13. 12. The display device of claim 12, the filter layer comprises a stacked light-emitting unit layer and a mask layer, and the mask layer is located on a side of the filter layer close to the image depth sensing component.
14. The display device according to claim 12, wherein the material of the mask layer includes a molybdenum metal material.
15. 13. The display device according to claim 13, wherein the display panel further comprises an anode conductive layer, and the anode conductive layer and the mask layer are an integral structure.
16. A depth image acquisition device, comprising an image depth sensing component and a filter layer located outside the image depth sensing component;

    The image depth sensing component is configured to emit reference light with a preset phase to the filter layer;

    The image depth sensing component is configured to obtain at least four output images by receiving reflected light, and the received phases of the at least four output images and the preset phases have different phase differences. Light is the light reflected by the object after the reference light passes through the filter layer;

    The image depth sensing component is configured to obtain at least four grayscale images according to the filter parameters of the filter layer and the at least four output images;

    The image depth sensing component is used to obtain a depth image based on the at least four gray-scale images.
17. The depth image acquiring device according to claim 16, wherein the image depth sensing component is used to acquire a undetermined filter parameter formula, the undetermined filter parameter formula comprising:

    

    Where H is the transfer function matrix of the filter layer, H ^* is the adjoint matrix of the transfer function matrix H, H _w is the undetermined filter parameter, and ε is the undetermined coefficient;

    The image depth sensing component is configured to convolve the at least four output images with the undetermined filter parameter to obtain at least four grayscale images, and determine that the definition of the at least four grayscale images is the highest At least four first undetermined coefficients corresponding to time;

    The image depth sensing component is used to determine the average value of the at least four first undetermined coefficients as the target coefficient;

    The image depth sensing component is used to bring the target coefficient as the undetermined coefficient into the undetermined filter parameter formula to obtain the filter parameter.
18. The depth image acquisition device according to claim 17, wherein the number of the at least four output images is four, and the four output images include a first output image, a second output image, a third output image, and a fourth output Image, where:

    The phase difference between the reception phase of the first output image and the preset phase is 0°,

    There is a 90° phase difference between the reception phase of the second output image and the preset phase,

    There is a phase difference of 180° between the reception phase of the third output image and the preset phase,

    There is a phase difference of 270° between the reception phase of the fourth output image and the preset phase.
19. The depth image acquisition device according to claim 18, wherein the at least four grayscale images are a first grayscale image corresponding to the first output image, a second grayscale image corresponding to the second output image, A third grayscale image corresponding to the third output image and a fourth grayscale image corresponding to the fourth output image,

    The image depth sensing component is used to obtain each of the first grayscale image, the second grayscale image, the third grayscale image, and the fourth grayscale image. The gray value of each pixel;

    The image depth sensing component is configured to obtain the depth image according to the gray value of each pixel and a depth image formula, and the depth image formula includes:

    

    

    Wherein, A ₁ , A ₂ , A ₃ and A ₄ are respectively the same position in the first gray scale image, the second gray scale image, the third gray scale image, and the fourth gray scale image , L is the depth of field parameter corresponding to each pixel in the depth image, φ is the phase difference corresponding to each pixel in the depth image, C is the speed of light, π is the circumference of the circle, and f is the The frequency of the reference light.
20. The depth image acquisition device according to any one of claims 16-19, wherein the image depth sensing component is configured to convolve the at least four output images with the filter parameters to obtain the at least four gray images. Degree image.

Images