CN112866675B

CN112866675B - Depth map generation method and device, electronic equipment and computer-readable storage medium

Info

Publication number: CN112866675B
Application number: CN201911102721.2A
Authority: CN
Inventors: 贾玉虎
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2019-11-12
Filing date: 2019-11-12
Publication date: 2022-09-09
Anticipated expiration: 2039-11-12
Also published as: CN112866675A

Abstract

The application relates to a depth map generation method and device, electronic equipment and a computer readable storage medium, and the method comprises the following steps: when the object distance in the original image is smaller than a preset object distance threshold value, calculating a phase difference in a first direction and a phase difference in a second direction of the original image, wherein a preset included angle is formed between the first direction and the second direction, and the preset object distance threshold value is the minimum object distance within the range of the overlapped field angle of a camera in the electronic equipment. And determining a target phase difference from the phase difference in the first direction and the phase difference in the second direction, calculating depth information of the original image according to the target phase difference, and generating a depth map of the original image according to the depth information. And determining a target phase difference from the phase difference in the first direction and the phase difference in the second direction, and calculating depth information according to the target phase difference to generate a depth map, so that the accuracy of the finally generated depth map can be greatly improved.

Description

Depth map generation method and device, electronic equipment and computer-readable storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a depth map generation method and apparatus, an electronic device, and a computer-readable storage medium.

Background

With the development of electronic device technology, more and more users shoot images through electronic devices. The depth map of the image can be calculated by shooting the image, and the depth map is widely focused more and more because the depth map has a wide application range and many functions. The depth map generated by the conventional method has low accuracy, and therefore, it is necessary to provide a method for generating a depth map to improve the accuracy of the generated depth map.

Disclosure of Invention

The embodiment of the application provides a depth map generation method and device, electronic equipment and a computer readable storage medium, which can improve the accuracy of a generated depth map.

A depth map generation method is applied to electronic equipment and comprises the following steps:

when the object distance in the original image is smaller than a preset object distance threshold value, calculating a phase difference in a first direction and a phase difference in a second direction of the original image, wherein a preset included angle is formed between the first direction and the second direction, and the preset object distance threshold value is the minimum object distance in the range of the overlapped field angle of a camera in the electronic equipment; determining a target phase difference from the phase difference in the first direction and the phase difference in the second direction;

calculating the depth information of the original image according to the target phase difference;

generating a depth map of the original image from the depth information.

A depth map generating apparatus comprising:

the target phase difference calculation module is used for calculating a phase difference in a first direction and a phase difference in a second direction for an original image when an object distance in the original image is smaller than a preset object distance threshold value, a preset included angle is formed between the first direction and the second direction, the preset object distance threshold value is a minimum object distance within an overlapped field angle range of a camera in the electronic equipment, and a target phase difference is determined from the phase difference in the first direction and the phase difference in the second direction;

the depth information calculation module is used for calculating the depth information of the original image according to the target phase difference;

and the depth map generating module is used for generating a depth map of the original image according to the depth information.

An electronic device comprising a memory and a processor, the memory having stored therein a computer program, which, when executed by the processor, causes the processor to carry out the steps of the above method.

A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above method.

According to the depth map generation method, the depth map generation device, the electronic equipment and the computer readable storage medium, when the object distance in the original image is smaller than a preset object distance threshold value, the phase difference in the first direction and the phase difference in the second direction are calculated for the original image, a preset included angle is formed between the first direction and the second direction, and the preset object distance threshold value is the minimum object distance within the range of the overlapped field angle of the camera in the electronic equipment. And determining a target phase difference from the phase difference in the first direction and the phase difference in the second direction, calculating depth information of the original image according to the target phase difference, and generating a depth map of the original image according to the depth information.

When the depth of field is calculated through the main camera and the auxiliary camera, the depth information can be calculated only when an object falls into the overlapped field angle of the main camera and the auxiliary camera. Therefore, when the object distance in the original image is smaller than the preset object distance threshold value, that is, the object in the original image does not fall within the overlapped field angle of the main and sub cameras, the depth information cannot be calculated by the main and sub cameras. The method for calculating the depth information through the phase difference does not have any requirement on the object distance, so that the depth information can be calculated by adopting the method for calculating the phase difference of the shot original images. Specifically, the phase difference in the first direction and the second direction is calculated for the original image. Compared with the traditional method that only the phase difference in one direction can be calculated, the phase difference in two directions can obviously reflect more phase difference information. And determining a target phase difference from the phase difference in the first direction and the phase difference in the second direction, and calculating depth information according to the target phase difference to generate a depth map, so that the accuracy of the finally generated depth map can be greatly improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.

FIG. 1 is a schematic diagram of phase detection autofocus;

fig. 2 is a schematic diagram of arranging phase detection pixels in pairs among pixels included in an image sensor;

FIG. 3 is a schematic diagram showing a partial structure of an image sensor according to an embodiment;

FIG. 4 is a schematic diagram of an embodiment of a pixel structure;

FIG. 5 is a schematic diagram showing an internal structure of an image sensor according to an embodiment;

FIG. 6 is a diagram illustrating an embodiment of a filter disposed on a pixel group;

FIG. 7 is a flow diagram of a method for depth map generation in one embodiment;

FIG. 8 is a schematic view of the field angles of two cameras;

FIG. 9 is a schematic diagram of binocular camera imaging;

FIG. 10 is a flowchart of the method of FIG. 7 for calculating a phase difference in a first direction and a phase difference in a second direction for an original image;

FIG. 11 is a flowchart of a method for calculating a phase difference in a first direction and a phase difference in a second direction according to the target luminance graph in FIG. 10;

FIG. 12 is a diagram illustrating a group of pixels in one embodiment;

FIG. 13 is a diagram of a sub-luminance graph in one embodiment;

FIG. 14 is a flow diagram of a 3D modeling method in one embodiment;

fig. 15 is a schematic configuration diagram of a depth map generating apparatus in one embodiment;

fig. 16 is a schematic diagram of the internal structure of the electronic device in one embodiment.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

It will be understood that, as used herein, the terms "first," "second," and the like may be used herein to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, a first camera may be referred to as a second camera, and similarly, a second camera may be referred to as a first camera, without departing from the scope of the present application. The first camera and the second camera are both cameras, but they are not the same camera.

Fig. 1 is a schematic diagram of a Phase Detection Auto Focus (PDAF) principle. As shown in fig. 1, M1 is the position of the image sensor when the imaging device is in the in-focus state, where the in-focus state refers to a successfully focused state. When the image sensor is located at the position M1, the imaging light rays g reflected by the object W in different directions toward the Lens converge on the image sensor, that is, the imaging light rays g reflected by the object W in different directions toward the Lens are imaged at the same position on the image sensor, and at this time, the image sensor is imaged clearly.

M2 and M3 indicate positions where the image sensor may be located when the imaging device is not in focus, and as shown in fig. 1, when the image sensor is located at the M2 position or the M3 position, the imaging light rays g reflected by the object W in different directions toward the Lens will be imaged at different positions. Referring to fig. 1, when the image sensor is located at position M2, the imaging light rays g reflected by the object W in different directions toward the Lens are imaged at position a and position B, respectively, and when the image sensor is located at position M3, the imaging light rays g reflected by the object W in different directions toward the Lens are imaged at position C and position D, respectively, and at this time, the image sensor is not clear.

In the PDAF technique, the difference in the position of the image formed by the imaging light rays entering the lens from different directions in the image sensor can be obtained, for example, as shown in fig. 1, the difference between the position a and the position B, or the difference between the position C and the position D can be obtained; after acquiring the difference of the positions of the images formed by the imaging light rays entering the lens from different directions in the image sensor, the out-of-focus distance can be obtained according to the difference and the geometric relationship between the lens and the image sensor in the camera, wherein the out-of-focus distance refers to the distance between the current position of the image sensor and the position where the image sensor should be located in the in-focus state; the imaging device can focus according to the obtained defocus distance.

From this, it is understood that the calculated PD value is 0 at the time of focusing, whereas the larger the calculated value is, the farther the position of the clutch focus is, and the smaller the value is, the closer the clutch focus is. When PDAF focusing is adopted, the PD value is calculated, the corresponding relation between the PD value and the defocusing distance is obtained according to calibration, the defocusing distance can be obtained, and then the lens is controlled to move to reach the focusing point according to the defocusing distance, so that focusing is realized.

In the related art, some of the phase detection pixel points may be provided in pairs among the pixel points included in the image sensor, and as shown in fig. 2, a phase detection pixel point pair (hereinafter, referred to as a pixel point pair) a, a pixel point pair B, and a pixel point pair C may be provided in the image sensor. In each pixel point pair, one phase detection pixel point performs Left shielding (English), and the other phase detection pixel point performs Right shielding (English).

For the phase detection pixel point which is shielded on the left side, only the light beam on the right side in the imaging light beam which is emitted to the phase detection pixel point can image on the photosensitive part (namely, the part which is not shielded) of the phase detection pixel point, and for the phase detection pixel point which is shielded on the right side, only the light beam on the left side in the imaging light beam which is emitted to the phase detection pixel point can image on the photosensitive part (namely, the part which is not shielded) of the phase detection pixel point. Therefore, the imaging light beam can be divided into a left part and a right part, and the phase difference can be obtained by comparing images formed by the left part and the right part of the imaging light beam.

However, since the phase detection pixel points arranged in the image sensor are generally sparse, only a horizontal phase difference can be obtained through the phase detection pixel points, and a scene with horizontal textures cannot be calculated, and the calculated PD values are mixed up to obtain an incorrect result, for example, a scene is photographed as a horizontal line, two left and right images are obtained according to PD characteristics, but the PD values cannot be calculated.

In order to solve the problem that the phase detection autofocus cannot calculate a PD value for some horizontal texture scenes to achieve focusing, an embodiment of the present application provides an imaging component, which may be configured to detect and output a phase difference value in a first direction and a phase difference value in a second direction, and may implement focusing by using the phase difference value in the second direction for horizontal texture scenes.

In one embodiment, the present application provides an imaging assembly. The imaging assembly includes an image sensor. The image sensor may be a Metal Oxide Semiconductor (CMOS) image sensor, a Charge-coupled Device (CCD), a quantum thin film sensor, an organic sensor, or the like.

Fig. 3 is a schematic structural diagram of a part of an image sensor in one embodiment. The image sensor 300 includes a plurality of pixel groups Z arranged in an array, each pixel group Z includes a plurality of pixels D arranged in an array, and each pixel D corresponds to one photosensitive unit. Each pixel point D comprises a plurality of sub-pixel points D arranged in an array. That is, each photosensitive unit may be composed of a plurality of photosensitive elements arranged in an array. The photosensitive element is an element capable of converting an optical signal into an electrical signal. In one embodiment, the light sensing element may be a photodiode. In the embodiment of the present application, each pixel group Z includes 4 pixel points D arranged in 2 × 2 arrays, and each pixel point may include 4 sub pixel points D arranged in 2 × 2 arrays. Each pixel group forms 2 x 2PD, can directly receive optical signals, performs photoelectric conversion, and can simultaneously output left and right and up and down signals. Each color channel may consist of 4 sub-pixel points.

As shown in fig. 4, taking each pixel point including a sub-pixel point 1, a sub-pixel point 2, a sub-pixel point 3, and a sub-pixel point 4 as an example, the sub-pixel point 1 and the sub-pixel point 2 may be synthesized, the sub-pixel point 3 and the sub-pixel point 4 are synthesized to form a PD pixel pair in the up-down direction, and a horizontal edge is detected to obtain a phase difference value in the second direction, that is, a PD value in the vertical direction; the sub-pixel point 1 and the sub-pixel point 3 are synthesized, and the sub-pixel point 2 and the sub-pixel point 4 are synthesized to form a PD pixel pair in the left and right directions, so that the vertical edge can be detected, and the phase difference value in the first direction, namely the PD value in the horizontal direction, is obtained.

Fig. 5 is a schematic internal configuration diagram of an image forming apparatus in one embodiment. As shown in fig. 5, the imaging device includes a lens 50, a filter 52, and an image sensor 54. The lens 50, the filter 52 and the image sensor 54 are sequentially located on the incident light path, i.e., the lens 50 is disposed on the filter 52 and the filter 52 is disposed on the image sensor 54.

The filter 52 may include three types of red, green and blue, which can only transmit the light with the wavelength corresponding to the red, green and blue. A filter 52 is disposed on one pixel.

The image sensor 54 may be the image sensor of fig. 3.

The lens 50 is used to receive incident light and transmit the incident light to the filter 52. The filter 52 smoothes incident light, and then makes the smoothed light incident on the light receiving unit of the image sensor 54 on a pixel basis.

The light-sensing unit in the image sensor 54 converts light incident from the optical filter 52 into a charge signal by the photoelectric effect, and generates a pixel signal in accordance with the charge signal. The charge signal corresponds to the received light intensity.

Fig. 6 is a schematic diagram illustrating a filter disposed on a pixel group according to an embodiment. The pixel point group Z comprises 4 pixel points D arranged in an array arrangement manner of two rows and two columns, wherein color channels of the pixel points in the first row and the first column are green, that is, the optical filters arranged on the pixel points in the first row and the first column are green optical filters; the color channel of the pixel points in the first row and the second column is red, that is, the optical filter arranged on the pixel points in the first row and the second column is a red optical filter; the color channel of the pixel points in the second row and the first column is blue, that is, the optical filter arranged on the pixel points in the second row and the first column is a blue optical filter; the color channel of the pixel points in the second row and the second column is green, that is, the optical filter arranged on the pixel points in the second row and the second column is a green optical filter.

FIG. 7 is a flow diagram of a method for depth map generation in one embodiment. The depth map generation method in the present embodiment is described by taking the example of the method performed on the image sensor in fig. 5. As shown in fig. 7, the depth map generating method includes steps 720 to 760.

Step 720, when the object distance in the original image is smaller than a preset object distance threshold, calculating a phase difference in a first direction and a phase difference in a second direction for the original image, wherein a preset included angle is formed between the first direction and the second direction, and the preset object distance threshold is the minimum object distance within the range of the overlapped field angle of the camera in the electronic equipment; a target phase difference is determined from the phase difference in the first direction and the phase difference in the second direction.

The original image refers to an RGB image obtained by shooting a shooting scene by a camera module of the electronic device, and the display range of the original image is consistent with the range of image information that can be captured by the camera module. And when the object distance in the original image is smaller than a preset object distance threshold value, calculating the phase difference of the first direction and the phase difference of the second direction for the original image. The preset object distance threshold is the minimum object distance within the range of the overlapped field angle of the cameras in the electronic equipment. As shown in fig. 8, the view angle of two cameras is schematically shown. There are two cameras (cameras) indicated as cam1 and cam2, respectively, and assuming cam1 as the primary camera, cam2 is the secondary camera. The main camera is used for shooting images, and the auxiliary camera is mainly used for calculating depth information, namely depth of field. The sub-camera here may be a general camera similar to the main camera, or may be a TOF camera. TOF is an abbreviation of Time of flight, interpreted as the meaning of Time of flight. The TOF camera is a time of flight camera. TOF cameras obtain target object distance by continuously sending light pulses to the target and then receiving light returning from the object with a sensor by detecting the time of flight (round trip) of the light pulses. The technology is different from a 3D laser sensor in principle, the 3D laser sensor scans point by point, and a TOF camera obtains depth information of the whole image at the same time.

The premise of the main camera and the auxiliary camera for shooting images and calculating depth information is that a shot object falls within the range of the overlapped field angles of the main camera and the auxiliary camera. That is, the object distance in the original image is required to be greater than or equal to the preset object distance threshold value, so that the object in the original image can fall within the range of the overlapping field angles of the main camera and the auxiliary camera. When the object distance in the original image is smaller than a preset object distance threshold value, the depth information cannot be calculated through a double-shot camera or a TOF camera. The method for calculating the depth information through the phase difference does not have any requirement on the object distance, so that the depth information can be calculated by adopting the method for calculating the phase difference of the shot original images.

The phase difference may be calculated by calculating a phase difference in a first direction and a phase difference in a second direction for the original image, and determining a target phase difference from the phase difference in the first direction and the phase difference in the second direction. Wherein, a preset included angle is formed between the first direction and the second direction. For example, the first direction and the second direction are perpendicular to each other. Of course, in other embodiments, only two different orientations are required, and the two different orientations need not be arranged in a perpendicular relationship. Specifically, the first direction and the second direction may be determined according to texture features in the original image. When the determined first direction is a horizontal direction in the original image, then the second direction may be a vertical direction in the original image. The phase difference in the horizontal direction can reflect the horizontal texture features in the original image, and the phase difference in the vertical direction can reflect the vertical texture features in the original image, so that the main body of the horizontal texture and the main body of the vertical texture in the original image can be considered.

Specifically, a luminance map is generated from an original image (RGB map). And then, calculating the phase difference in the first direction and the phase difference in the second direction according to the brightness graph, wherein a preset included angle is formed between the first direction and the second direction. Generally, a frequency domain algorithm and a space domain algorithm can be used for calculating the phase difference value, but other methods can be used for calculating the phase difference value. The frequency domain algorithm is to calculate by using the characteristic of Fourier displacement, convert the acquired target brightness image from the space domain to the frequency domain by using Fourier transformation, then calculate phase compensation, when the compensation calculates the maximum value (peak), it shows that there is the maximum displacement, at this time, it can know how much the maximum displacement is in the space domain by doing inverse Fourier. The spatial domain algorithm is to find out feature points, such as edge features, dog (difference of gaussian), Harris corner points, and calculate displacement by using the feature points.

When the target phase difference is determined from the phase difference in the first direction and the phase difference in the second direction, a first confidence degree of the phase difference value in the first direction and a second confidence degree of the phase difference value in the second direction can be obtained; and selecting the larger phase difference value of the first confidence coefficient and the second confidence coefficient as a target phase difference value. Specifically, the first confidence level and the second confidence level may be adjusted according to the movement information of the electronic device when the original image is captured, so as to obtain the adjusted first confidence level and the adjusted second confidence level. And screening out the phase difference corresponding to the larger confidence coefficient from the adjusted first confidence coefficient and the second confidence coefficient to serve as the target phase difference. Specifically, movement information of the electronic device when the electronic device captures an original image may be collected through a gyroscope in the electronic device, and the movement information includes a movement speed and a movement direction. Of course, the movement information of the electronic device when the electronic device shoots the original image may be acquired by a sensor such as a direction sensor or an acceleration sensor, which is not limited in this application. A target phase difference is determined from the phase difference in the first direction and the phase difference in the second direction based on movement information when the original image is captured by the electronic device. Because the movement information of the electronic device may affect the phase difference between the first direction and the second direction of the captured original image during the process of capturing the original image, for example, when the movement direction of the electronic device is the same as the first direction, a horizontal smear in the first direction may be generated on the captured original image, and the horizontal smear in the first direction is similar to a texture in the first direction. Therefore, the accuracy of the phase difference in the first direction calculated at this time is low, or the phase difference value in the first direction cannot be obtained even from the PD pixel pair in the first direction. Therefore, the target phase difference is determined from the phase difference of the first direction and the phase difference of the second direction according to the moving speed and the moving direction when the electronic equipment shoots the original image, and the influence of the moving speed and the moving direction of the electronic equipment on the accuracy of the determined target phase difference is avoided.

And step 740, calculating the depth information of the original image according to the target phase difference.

After the target phase difference (referred to as PD or shift _ cur) is obtained in the above calculation, the defocus distance is calculated from the target phase difference. Wherein, the calculation formula of the defocus distance is as follows:

Defocus＝PD*slope(DCC)， (1-1)

where, Defocus refers to a Defocus distance, PD refers to a target phase difference, slope (dcc) refers to a Defocus conversion coefficient. slope (dcc) is the defocus transform coefficient calculated during camera calibration, and is fixed for the same camera.

Then, depth information is calculated from the defocus distance. Specifically, a phase difference at the time of focusing is calculated from the Defocus distance (the focal length is subtracted from the image distance of the lens at the time of focusing the image in the area is calculated from the Defocus distance) (denoted by shift), and the phase difference at the time of focusing is shift _ cur (the focal length is subtracted from the image distance of the current position of the lens) + Defocus. The formula for calculating the object distance is as follows:

u＝f*f/shift+f (1-2)

where u denotes an object distance, f denotes a focal length, and shift denotes a phase difference in focusing.

Thus, the object distance is calculated for all the pixels in the original image, and the depth information of the original image is obtained.

Step 760, generating a depth map of the original image from the depth information.

After the depth information of the original image is obtained, a depth map is constructed from the depth information of the original image.

In the embodiment of the application, when the depth of field is calculated by the main camera and the auxiliary camera, the depth information can be calculated only when the object falls into the overlapped field angle of the main camera and the auxiliary camera. Therefore, when the object distance in the original image is smaller than the preset object distance threshold value, that is, the object in the original image does not fall within the overlapped field angle of the main and sub cameras, the depth information cannot be calculated by the main and sub cameras. The method for calculating the depth information through the phase difference does not have any requirement on the object distance, so that the depth information can be calculated by adopting the method for calculating the phase difference of the shot original images. Specifically, the phase difference in the first direction and the second direction is calculated for the original image. Compared with the traditional method that only the phase difference in one direction can be calculated, the phase difference in two directions can obviously reflect more phase difference information. And determining a target phase difference from the phase difference in the first direction and the phase difference in the second direction, and calculating depth information according to the target phase difference to generate a depth map, so that the accuracy of the finally generated depth map can be greatly improved.

In the embodiment, the electronic device includes at least two ordinary cameras and at least one time of flight TOF camera, and the method for generating the depth map further includes:

when the object distance in the original image is larger than or equal to a preset object distance threshold value, judging whether the texture direction of the original image is consistent with the arrangement direction of two common cameras used for calculating depth information in the electronic equipment;

and if the texture direction of the original image is judged to be consistent with the arrangement direction of the two common cameras in the electronic equipment, calculating the depth information of the original image through the TOF camera in the electronic equipment.

Specifically, since the main camera and the sub-camera realize image capturing and depth information calculation, it is a prerequisite that the object to be captured falls within the overlapping field angle range of the main and sub-cameras. That is, the object distance in the original image is required to be greater than or equal to the preset object distance threshold value, so that the object in the original image can fall within the range of the overlapping field angles of the main camera and the auxiliary camera. When the object distance in the original image is greater than or equal to the preset object distance threshold, it is further necessary to determine whether the texture direction of the original image is consistent with the arrangement direction of two common cameras used for calculating depth information in the electronic device. Because the process of calculating depth information with two common cameras (binocular cameras) is: and calculating a depth map according to the displacement of the images of the two cameras, the installation positions of the cameras and the optical characteristics of the lens. As shown in fig. 9, a schematic diagram of binocular camera imaging is shown. The formula for calculating the depth of field in this figure is:

Z＝B*f/d， (1-3)

wherein, B refers to the distance between the two cameras, f refers to the focal length, d is the phase difference in the left and right images shot by the binocular camera, and Z is the depth of field of the region.

When the texture direction of the original image is consistent with the arrangement direction of the binocular cameras in the electronic equipment, the phase difference between the left and right images shot by the binocular cameras cannot be calculated. Therefore, the depth of field cannot be calculated. At this time, the depth information of the original image can be calculated through a TOF camera in the electronic equipment, so that a depth map is constructed. TOF cameras obtain target object distance by continuously sending light pulses to the target and then receiving light returning from the object with a sensor by detecting the time of flight (round trip) of the light pulses. Because, the TOF camera is not affected by the texture features in the original image in the process of calculating the depth of field.

In the embodiment of the application, when the object distance in the original image is greater than or equal to the preset object distance threshold, that is, when the object in the original image falls within the range of the overlapped field angles of the main camera and the auxiliary camera, the depth information can be calculated by using the binocular camera originally, and the depth information can also be calculated by using the TOF camera. However, when the texture direction of the original image coincides with the arrangement direction of the binocular cameras in the electronic apparatus, the depth of field cannot be calculated, so that it is necessary to further determine whether the texture direction of the original image coincides with the arrangement direction of two general cameras in the electronic apparatus for calculating depth information. If the depth information is consistent with the preset depth information, only the TOF camera can be adopted to calculate the depth information, the accuracy of the calculated depth information can be ensured, and the accuracy of the generated depth map is further improved.

In the above embodiment, a depth map generating method is provided, which further includes: when the accuracy of the depth information of the preset area in the original image calculated by a TOF camera in the electronic equipment is lower than a preset accuracy threshold, calculating first depth information of the preset area in the original image according to the target phase difference;

acquiring depth information of other regions except the preset region from the depth information of the original image calculated by a TOF camera in the electronic equipment;

and synthesizing the first depth information of the preset area in the original image with the depth information of other areas except the preset area to obtain the depth information of the original image.

Specifically, when the object distance in the original image is greater than or equal to a preset object distance threshold, and the texture direction of the original image is consistent with the arrangement direction of two common cameras used for calculating the depth information in the electronic device, the TOF camera is used for calculating the depth information. TOF cameras, on the one hand, emit infrared radiation at a limited range and, on the other hand, for transparent objects, or objects with too low a reflectivity, absorb infrared radiation, which results in a decrease in the accuracy of the depth information obtained by the TOF camera. In addition, the method is not suitable for a TOF camera to calculate depth information for a scene with strong infrared interference. Due to the defects of the TOF camera, the accuracy of the depth information of the partial region in the original image calculated by the TOF camera is low. When the accuracy of the depth information of the partial region in the original image calculated by the TOF camera is judged to be lower than a preset accuracy threshold, a target phase difference can be calculated in a phase difference calculation mode, and then first depth information of the preset region in the original image is calculated. The accuracy of the depth information of the original image calculated by the camera can be calculated through a deep learning algorithm. Then, the accuracy of the calculated depth information of the original image is compared with a preset accuracy threshold. The preset accuracy threshold may be a self-defined threshold, for example, 80%, and this is not limited in this application.

Then, from the depth information of the original image calculated by the TOF camera in the electronic apparatus, the depth information of the other region excluding the preset region is acquired. And synthesizing the first depth information of the preset area in the original image and the depth information of other areas except the preset area to obtain the depth information of the whole area of the original image.

In the embodiment of the application, when the accuracy of the depth information of the preset region in the original image calculated by the TOF camera in the electronic device is lower than a preset accuracy threshold, the first depth information of the preset region in the original image is calculated by calculating the target phase difference. Then, the depth information of the other region except the preset region calculated by the TOF camera in the electronic device is synthesized with the first depth information of the preset region calculated by calculating the target phase difference, so as to obtain the depth information of the whole region of the original image. The accuracy of the depth information of the whole area of the finally obtained original image is ensured.

In one embodiment, after determining whether the texture direction of the original image is consistent with the arrangement direction of two common cameras used for calculating the depth information in the electronic device, the method includes:

and if the texture direction of the original image is judged to be inconsistent with the arrangement direction of the two common cameras used for calculating the depth information in the electronic equipment, calculating the depth information of the original image through the two common cameras in the electronic equipment.

In the embodiment of the application, because the TOF camera is limited by the range problem of the transmitted light pulses, and the like, the depth information calculated by the binocular camera has wider applicability and higher accuracy. Therefore, when the object distance in the original image is greater than or equal to the preset object distance threshold value, and the texture direction of the original image is not consistent with the arrangement direction of the two ordinary cameras used for calculating the depth information in the electronic equipment, the depth information of the original image is calculated through the two ordinary cameras in the electronic equipment.

In another embodiment, a depth map generating method is provided, which further includes: when the accuracy of the depth information of the preset area in the original image calculated by two common cameras in the electronic equipment is lower than a preset accuracy threshold, calculating first depth information of the preset area in the original image according to a target phase difference, or calculating second depth information of the preset area in the original image according to a TOF camera in the electronic equipment;

screening out depth information with higher accuracy from first depth information and second depth information of a preset area in an original image to serve as target depth information of the preset area in the original image;

acquiring depth information of other areas except a preset area from the depth information of the original image calculated by two common cameras in the electronic equipment;

and synthesizing the target depth information of the preset area in the original image and the depth information of other areas except the preset area to obtain the depth information of the original image.

In the embodiment of the application, when the accuracy of the depth information of the preset area in the original image calculated by two common cameras in the electronic device is lower than a preset accuracy threshold, the depth information of the preset area is calculated by a phase difference calculating mode and a TOF camera respectively. And then, screening out the depth information with higher accuracy from the depth information calculated by the two modes to serve as the target depth information of the preset area in the original image. And finally, synthesizing the target depth information of the preset area in the original image with the depth information of other areas except the preset area to obtain the depth information of the whole original image. The accuracy of the depth information of the whole area of the finally obtained original image is ensured.

In a specific embodiment, a depth map generation method is provided, including:

when the object distance in the original image is smaller than a preset object distance threshold value, namely the depth of field is shallow, calculating a phase difference in a first direction and a phase difference in a second direction for the original image, wherein a preset included angle is formed between the first direction and the second direction, and the preset object distance threshold value is the minimum object distance in the range of the overlapped field angle of a camera in the electronic equipment. Determining a target phase difference from the phase difference in the first direction and the phase difference in the second direction, calculating depth information of an original image according to the target phase difference, and generating a depth map of the original image according to the depth information. Because the mode of calculating the depth information by using the phase difference is adopted, the accuracy of the obtained depth information is lower for the condition that the depth of field is deeper. Therefore, for the case of a deep depth of field, other methods are needed for calculation.

When the object distance in the original image is greater than or equal to the preset object distance threshold value and the texture in the original image is abundant enough (including texture features in multiple directions), that is, it can be determined that the texture direction of the original image is inconsistent with the arrangement direction of the two ordinary cameras in the electronic device, the depth information of the original image is calculated through the two ordinary cameras (binocular cameras) in the electronic device, and the depth map of the original image is generated through the depth information.

And when the object distance in the original image is greater than or equal to a preset object distance threshold value, and the texture in the original image is not rich enough, namely if the texture direction of the original image is determined to be consistent with the arrangement directions of the two common cameras in the electronic equipment, calculating the depth information of the original image through a TOF camera in the electronic equipment, and generating the depth map of the original image according to the depth information.

When the depth information of the original image cannot be calculated by a binocular camera or a TOF camera in the electronic equipment for a partial area in the original image, calculating a phase difference in a first direction and a phase difference in a second direction for the partial area in the original image, determining a target phase difference from the phase difference in the first direction and the phase difference in the second direction, and calculating the depth information of the partial area in the original image according to the target phase difference. And combining the depth information of other areas in the original image, and generating a depth map of the original image according to the depth information.

In the embodiment of the application, for various scenes in the process of generating the depth map by calculating the depth information, the modes of calculating the depth information are correspondingly set respectively, so that the accuracy of the calculated depth information is ensured, and the accuracy of the obtained depth map is finally improved.

In one embodiment, an electronic device includes an image sensor including a plurality of pixel groups arranged in an array, each pixel group including M × N pixels arranged in an array; each pixel point corresponds to one photosensitive unit; each pixel point comprises a plurality of sub pixel points which are arranged in an array, wherein M and N are both natural numbers which are more than or equal to 2; as shown in fig. 10, step 720 of calculating the phase difference in the first direction and the phase difference in the second direction for the original image includes:

step 722, for each pixel point group in the original image, obtaining a sub-brightness map corresponding to the pixel point group according to the brightness value of the sub-pixel point at the same position of each pixel point in the pixel point group.

In general, the luminance value of a pixel of an image sensor may be represented by the luminance value of a sub-pixel included in the pixel. The imaging device can obtain the sub-brightness graph corresponding to the pixel point group according to the brightness value of the sub-pixel point at the same position of each pixel point in the pixel point group. The brightness value of the sub-pixel point refers to the brightness value of the optical signal received by the photosensitive element corresponding to the sub-pixel point.

As described above, the sub-pixel included in the image sensor is a photosensitive element capable of converting an optical signal into an electrical signal, so that the intensity of the optical signal received by the sub-pixel can be obtained according to the electrical signal output by the sub-pixel, and the brightness value of the sub-pixel can be obtained according to the intensity of the optical signal received by the sub-pixel.

And 724, generating a target brightness map according to the sub-brightness map corresponding to each pixel point group.

The imaging device can splice the sub-luminance graphs corresponding to the pixel groups according to the array arrangement mode of the pixel groups in the image sensor to obtain a target luminance graph.

Step 726, calculating the phase difference of the first direction and the phase difference of the second direction according to the target brightness map.

And performing segmentation processing on the target brightness image to obtain a first segmentation brightness image and a second segmentation brightness image, and determining the phase difference value of the matched pixels according to the position difference of the matched pixels in the first segmentation brightness image and the second segmentation brightness image. And determining the phase difference value of the first direction and the phase difference value of the second direction according to the phase difference values of the matched pixels.

In this embodiment of the application, because the image sensor in the electronic device includes a plurality of pixel groups that the array was arranged, every pixel group includes a plurality of pixels that the array was arranged. Each pixel point corresponds to one photosensitive unit and comprises a plurality of sub-pixel points arranged in an array, and each sub-pixel point corresponds to one photodiode. Therefore, a brightness value, namely the brightness value of the sub-pixel point, can be obtained through each photodiode. And acquiring a sub-brightness image corresponding to the pixel group according to the brightness value of the sub-pixel, and splicing the sub-brightness images corresponding to the pixel groups according to the array arrangement mode of the pixel groups in the image sensor to obtain a target brightness image. Finally, the phase difference in the first direction and the phase difference in the second direction can be calculated according to the target brightness map.

And then, the target phase difference is determined from the phase difference in the first direction and the phase difference in the second direction, so that the accuracy of the target phase difference is higher, and the accuracy in the focusing process can be greatly improved by focusing the original image according to the target phase difference.

In one embodiment, as shown in FIG. 11, step 726 of calculating the phase difference in the first direction and the phase difference in the second direction from the target luminance map comprises: step 726a and step 726 b.

Step 726a, the target luminance graph is segmented to obtain a first segmented luminance graph and a second segmented luminance graph, and the phase difference values of the pixels matched with each other are determined according to the position difference of the pixels matched with each other in the first segmented luminance graph and the second segmented luminance graph.

In the method for performing the segmentation processing on the target brightness map, an imaging device may perform the segmentation processing on the target brightness map along a column direction (a y-axis direction in an image coordinate system), and in the process of performing the segmentation processing on the target brightness map along the column direction, each segmentation line of the segmentation processing is perpendicular to the column direction.

In another way of performing the segmentation processing on the target luminance map, the imaging device may perform the segmentation processing on the target luminance map along the row direction (the x-axis direction in the image coordinate system), and each segmentation line of the segmentation processing is perpendicular to the row direction during the segmentation processing on the target luminance map along the row direction.

The first and second sliced luminance graphs obtained by slicing the target luminance graph in the column direction may be referred to as upper and lower graphs, respectively. The first and second sliced luminance maps obtained by slicing the target luminance map in the row direction may be referred to as left and right maps, respectively.

Here, "pixels matched with each other" means that pixel matrices composed of the pixels themselves and their surrounding pixels are similar to each other. For example, the pixel a and its surrounding pixels in the first tangential luminance map form a 3-row and 3-column pixel matrix, and the pixel values of the pixel matrix are:

2 15 70

1 35 60

0 100 1

the pixel b and its surrounding pixels in the second sliced luminance graph also form a pixel matrix with 3 rows and 3 columns, and the pixel values of the pixel matrix are:

1 15 70

1 36 60

0 100 2

as can be seen from the above, the two matrices are similar, and pixel a and pixel b can be considered to match each other. The pixel matrixes are judged to be similar in many ways, usually, the pixel values of each corresponding pixel in two pixel matrixes are subtracted, the absolute values of the obtained difference values are added, and the result of the addition is used for judging whether the pixel matrixes are similar, that is, if the result of the addition is smaller than a preset threshold, the pixel matrixes are considered to be similar, otherwise, the pixel matrixes are considered to be dissimilar.

For example, for the two pixel matrices of 3 rows and 3 columns, 1 and 2 are subtracted, 15 and 15 are subtracted, 70 and 70 are subtracted, … … are added, and the absolute values of the obtained differences are added to obtain an addition result of 3, and if the addition result of 3 is smaller than a preset threshold, the two pixel matrices of 3 rows and 3 columns are considered to be similar.

Another way to judge whether the pixel matrixes are similar is to extract the edge features of the pixel matrixes by using a sobel convolution kernel calculation way or a high laplacian calculation way, and the like, and judge whether the pixel matrixes are similar through the edge features.

Here, the "positional difference of mutually matched pixels" refers to a difference in the position of a pixel located in the first sliced luminance graph and the position of a pixel located in the second sliced luminance graph among mutually matched pixels. As exemplified above, the positional difference of the pixel a and the pixel b that match each other refers to the difference in the position of the pixel a in the first sliced luminance graph and the position of the pixel b in the second sliced luminance graph.

The pixels matched with each other respectively correspond to different images formed in the image sensor by imaging light rays entering the lens from different directions. For example, a pixel a in the first sliced luminance graph and a pixel B in the second sliced luminance graph match each other, where the pixel a may correspond to the image formed at the a position in fig. 1 and the pixel B may correspond to the image formed at the B position in fig. 1.

Since the matched pixels respectively correspond to different images formed by imaging light rays entering the lens from different directions in the image sensor, the phase difference of the matched pixels can be determined according to the position difference of the matched pixels.

In step 726b, a phase difference value in the first direction or a phase difference value in the second direction is determined according to the phase difference values of the matched pixels.

When the first split luminance map includes even-numbered lines of pixels, the second split luminance map includes odd-numbered lines of pixels, and the pixel a in the first split luminance map and the pixel b in the second split luminance map are matched with each other, the phase difference value in the first direction can be determined according to the phase difference between the pixel a and the pixel b which are matched with each other.

When the first split luminance map includes pixels in even columns and the second split luminance map includes pixels in odd columns, and the pixel a in the first split luminance map and the pixel b in the second split luminance map are matched with each other, the phase difference value in the second direction can be determined according to the phase difference between the pixel a and the pixel b which are matched with each other.

In the embodiment of the application, the sub-luminance graphs corresponding to the pixel point groups are spliced to obtain the target luminance graph, the target luminance graph is divided into two segmentation luminance graphs, the phase difference values of the pixels matched with each other can be rapidly determined through pixel matching, meanwhile, rich phase difference values are contained, the accuracy of the phase difference values can be improved, and the focusing accuracy and stability are improved.

Fig. 12 is a schematic diagram of a pixel point group in an embodiment, as shown in fig. 12, the pixel point group includes 4 pixel points arranged in an array arrangement manner of two rows and two columns, where the 4 pixel points are a D1 pixel point, a D2 pixel point, a D3 pixel point, and a D4 pixel point, where each pixel point includes 4 sub pixel points arranged in an array arrangement manner of two rows and two columns, where the sub pixel points are D11, D12, D13, D14, D21, D22, D23, D24, D31, D32, D33, D34, D41, D42, D43, and D44, respectively.

As shown in fig. 13, the arrangement positions of the sub-pixel points d11, d21, d31 and d41 in each pixel point are the same and are all first rows and first columns, the arrangement positions of the sub-pixel points d12, d22, d32 and d42 in each pixel point are the same and are all first rows and second columns, the arrangement positions of the sub-pixel points d13, d23, d33 and d43 in each pixel point are the same and are all second rows and first columns, and the arrangement positions of the sub-pixel points d14, d24, d34 and d44 in each pixel point are the same and are all second rows and second columns.

In one embodiment, the step 722 of obtaining the sub-luminance graph corresponding to the pixel point group according to the luminance value of the sub-pixel point at the same position of each pixel point in the pixel point group may include steps a1 to A3.

Step a1, the imaging device determines sub-pixel points at the same position from each pixel point, and obtains a plurality of sub-pixel point sets. And the positions of the sub-pixel points included in each sub-pixel point set in the pixel points are the same.

The imaging equipment respectively determines sub-pixel points at the same position from the D1 pixel point, the D2 pixel point, the D3 pixel point and the D4 pixel point to obtain 4 sub-pixel point sets J1, J2, J3 and J4, wherein the sub-pixel set J1 comprises sub-pixels d11, d21, d31 and d41, the positions of the sub-pixel points included in the pixel points are the same, the sub-pixel point set J2 comprises sub-pixel points d12, d22, d32 and d42, the positions of the sub-pixel points included in the pixel points are the same, the sub-pixel point set J3 comprises sub-pixel points d13, d23, d33 and d43, the positions of the included sub-pixel points in the pixel points are the same and are in a second row and a first column, the sub-pixel point set J4 comprises sub-pixel points d14, d24, d34 and d44, and the positions of the included sub-pixel points in the pixel points are the same and are in a second row and a second column.

Step A2, for each sub-pixel point set, the imaging device obtains the brightness value corresponding to the sub-pixel point set according to the brightness value of each sub-pixel point in the sub-pixel point set.

Optionally, in step a2, the imaging device may determine a color coefficient corresponding to each sub-pixel point in the sub-pixel point set, where the color coefficient is determined according to a color channel corresponding to the sub-pixel point.

For example, the sub-pixel D11 belongs to the D1 pixel, the color filter included in the D1 pixel may be a green color filter, that is, the color channel of the D1 pixel is green, the color channel of the sub-pixel D11 included in the D1 pixel is also green, and the imaging device may determine the color coefficient corresponding to the sub-pixel D11 according to the color channel (green) of the sub-pixel D11.

After determining the color coefficient corresponding to each sub-pixel point in the sub-pixel point set, the imaging device may multiply the color coefficient corresponding to each sub-pixel point in the sub-pixel point set with the luminance value to obtain a weighted luminance value of each sub-pixel point in the sub-pixel point set.

For example, the imaging device may multiply the luminance value of the sub-pixel d11 with the color coefficient corresponding to the sub-pixel d11 to obtain a weighted luminance value of the sub-pixel d 11.

After the weighted brightness value of each sub-pixel in the sub-pixel set is obtained, the imaging device may add the weighted brightness values of each sub-pixel in the sub-pixel set to obtain a brightness value corresponding to the sub-pixel set.

For example, for the sub-pixel point set J1, the brightness value corresponding to the sub-pixel point set J1 can be calculated based on the following first formula.

Y_TL＝Y_21*C_R+(Y_11+Y_41)*C_G/2+Y_31*C_B。

Y _ TL is a luminance value corresponding to the sub-pixel set J1, Y _21 is a luminance value of the sub-pixel d21, Y _11 is a luminance value of the sub-pixel d11, Y _41 is a luminance value of the sub-pixel d41, Y _31 is a luminance value of the sub-pixel d31, C _ R is a color coefficient corresponding to the sub-pixel d21, C _ G/2 is a color coefficient corresponding to the sub-pixels d11 and d41, C _ B is a color coefficient corresponding to the sub-pixel d31, Y _21 × C _ R is a weighted luminance value of the sub-pixel d21, Y _11 × C _ G/2 is a weighted luminance value of the sub-pixel d11, Y _41 × C _ G/2 is a weighted luminance value of the sub-pixel d41, and Y _31 × C _ B is a weighted luminance value of the sub-pixel d 31.

For the sub-pixel point set J2, the brightness value corresponding to the sub-pixel point set J2 can be calculated based on the following second formula.

Y_TR＝Y_22*C_R+(Y_12+Y_42)*C_G/2+Y_32*C_B。

Y _ TR is a brightness value corresponding to the sub-pixel set J2, Y _22 is a brightness value of the sub-pixel d22, Y _12 is a brightness value of the sub-pixel d12, Y _42 is a brightness value of the sub-pixel d42, Y _32 is a brightness value of the sub-pixel d32, C _ R is a color coefficient corresponding to the sub-pixel d22, C _ G/2 is a color coefficient corresponding to the sub-pixels d12 and d42, C _ B is a color coefficient corresponding to the sub-pixel d32, Y _22 × C _ R is a weighted brightness value of the sub-pixel d22, Y _12 × C _ G/2 is a weighted brightness value of the sub-pixel d12, Y _42 × C _ G/2 is a weighted brightness value of the sub-pixel d42, and Y _32 × C _ B is a weighted brightness value of the sub-pixel d 32.

For the sub-pixel point set J3, the brightness value corresponding to the sub-pixel point set J3 can be calculated based on the following third formula.

Y_BL＝Y_23*C_R+(Y_13+Y_43)*C_G/2+Y_33*C_B。

Y _ BL is a brightness value corresponding to the sub-pixel set J3, Y _23 is a brightness value of the sub-pixel d23, Y _13 is a brightness value of the sub-pixel d13, Y _43 is a brightness value of the sub-pixel d43, Y _33 is a brightness value of the sub-pixel d33, C _ R is a color coefficient corresponding to the sub-pixel d23, C _ G/2 is a color coefficient corresponding to the sub-pixels d13 and d43, C _ B is a color coefficient corresponding to the sub-pixel d33, Y _23 × C _ R is a weighted brightness value of the sub-pixel d23, Y _13 × C _ G/2 is a weighted brightness value of the sub-pixel d13, Y _43 × C _ G/2 is a weighted brightness value of the sub-pixel d43, and Y _33 × C _ B is a weighted brightness value of the sub-pixel d 33.

For the sub-pixel point set J4, the brightness value corresponding to the sub-pixel point set J4 can be calculated based on the following fourth formula.

Y_BR＝Y_24*C_R+(Y_14+Y_44)*C_G/2+Y_34*C_B。

Y _ BR is a brightness value corresponding to the sub-pixel set J4, Y _24 is a brightness value of the sub-pixel d24, Y _14 is a brightness value of the sub-pixel d14, Y _44 is a brightness value of the sub-pixel d44, Y _34 is a brightness value of the sub-pixel d34, C _ R is a color coefficient corresponding to the sub-pixel d24, C _ G/2 is a color coefficient corresponding to the sub-pixels d14 and d44, C _ B is a color coefficient corresponding to the sub-pixel d34, Y _24 × C _ R is a weighted brightness value of the sub-pixel d24, Y _14 × C _ G/2 is a weighted brightness value of the sub-pixel d14, Y _44 × C _ G/2 is a weighted brightness value of the sub-pixel d44, and Y _34 × C _ B is a weighted brightness value of the sub-pixel d 34.

Step a3, the imaging device generates a sub-luminance map according to the luminance value corresponding to each sub-pixel set.

The sub-luminance map comprises a plurality of pixels, each pixel in the sub-luminance map corresponds to one sub-pixel set, and the pixel value of each pixel is equal to the luminance value corresponding to the corresponding sub-pixel set.

FIG. 13 is a diagram of a sub-luminance graph in one embodiment. As shown in fig. 13, the sub-luminance map includes 4 pixels, wherein the pixels in the first row and the first column correspond to the sub-pixel set J1 and have the pixel value Y _ TL, the pixels in the first row and the second column correspond to the sub-pixel set J2 and have the pixel value Y _ TR, the pixels in the second row and the first column correspond to the sub-pixel set J3 and have the pixel value Y _ BL, and the pixels in the second row and the second column correspond to the sub-pixel set J4 and have the pixel value Y _ BR.

It should be understood that, although the steps in the above-described flowcharts are shown in sequence as indicated by the arrows, the steps are not necessarily performed in sequence as indicated by the arrows. The steps are not limited to being performed in the exact order illustrated and, unless explicitly stated herein, may be performed in other orders. Moreover, at least a portion of the steps in the above-described flowcharts may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the sub-steps or the stages is not necessarily sequential, but may be performed alternately or alternatingly with other steps or at least a portion of the sub-steps or stages of other steps.

In one embodiment, as shown in fig. 14, there is provided a 3D modeling method including:

step 1420, when the object distance in the original image is smaller than a preset object distance threshold, calculating a phase difference in a first direction and a phase difference in a second direction for the original image, wherein a preset included angle is formed between the first direction and the second direction, and the preset object distance threshold is a minimum object distance within an overlapped view angle range of a camera in the electronic device; a target phase difference is determined from the phase difference in the first direction and the phase difference in the second direction.

Step 1440, calculating depth information of the original image according to the target phase difference.

At step 1460, a depth map of the original image is generated from the depth information.

Step 1480, 3D modeling is performed according to the depth map of the original image.

In the embodiment of the application, when the object distance in an original image is smaller than a preset object distance threshold, calculating a phase difference in a first direction and a phase difference in a second direction for the original image, wherein a preset included angle is formed between the first direction and the second direction, and the preset object distance threshold is a minimum object distance in an overlapped view angle range of a camera in electronic equipment; a target phase difference is determined from the phase difference in the first direction and the phase difference in the second direction. And calculating the depth information of the original image according to the target phase difference, and generating a depth map of the original image according to the depth information. 3D modeling is performed according to the depth map of the original image.

Of course, after the depth map of the original image is obtained, the original image may be subjected to background blurring, object segmentation, 3D scanning or AR enhancement, motion recognition, and the like.

In one embodiment, as shown in fig. 15, there is provided a depth map generating apparatus 1500 including: a target phase difference calculation module 1520, a depth information calculation module 1540, and a depth map generation module 1560. Wherein the content of the first and second substances,

the target phase difference calculation module 1520, configured to calculate, for the original image, a phase difference in a first direction and a phase difference in a second direction when an object distance in the original image is smaller than a preset object distance threshold, where the first direction and the second direction form a preset included angle, the preset object distance threshold is a minimum object distance within an overlapping field angle range of a camera in the electronic device, and the target phase difference is determined from the phase difference in the first direction and the phase difference in the second direction;

a depth information calculating module 1540, configured to calculate depth information of the original image according to the target phase difference;

and a depth map generating module 1560 for generating a depth map of the original image from the depth information.

In one embodiment, the electronic device includes at least two general cameras, at least one time of flight TOF camera, and the depth map generating apparatus 1500 is further configured to: when the object distance in the original image is larger than or equal to a preset object distance threshold value, judging whether the texture direction of the original image is consistent with the arrangement direction of two common cameras used for calculating depth information in the electronic equipment; and if the texture direction of the original image is judged to be consistent with the arrangement direction of the two common cameras in the electronic equipment, calculating the depth information of the original image through the TOF camera in the electronic equipment.

In one embodiment, the depth map generating apparatus 1500 is further configured to: when the accuracy of the depth information of the preset area in the original image calculated by a TOF camera in the electronic equipment is lower than a preset accuracy threshold, calculating first depth information of the preset area in the original image according to the target phase difference; acquiring depth information of other regions except the preset region from the depth information of the original image calculated by a TOF camera in the electronic equipment; and synthesizing the first depth information of the preset area in the original image with the depth information of other areas except the preset area to obtain the depth information of the original image.

In one embodiment, the depth map generating apparatus 1500 is further configured to: and if the texture direction of the original image is judged to be inconsistent with the arrangement direction of the two ordinary cameras used for calculating the depth information in the electronic equipment, calculating the depth information of the original image through the two ordinary cameras in the electronic equipment.

In one embodiment, the depth map generating apparatus 1500 is further configured to: when the accuracy of the depth information of the preset area in the original image calculated by two common cameras in the electronic equipment is lower than a preset accuracy threshold, calculating first depth information of the preset area in the original image according to a target phase difference, or calculating second depth information of the preset area in the original image according to a TOF camera in the electronic equipment; screening out depth information with higher accuracy from first depth information and second depth information of a preset region in an original image, and taking the depth information as target depth information of the preset region in the original image; acquiring depth information of other areas except a preset area from the depth information of an original image calculated by two ordinary cameras in the electronic equipment; and synthesizing the target depth information of the preset area in the original image with the depth information of other areas except the preset area to obtain the depth information of the original image.

In one embodiment, the target phase difference calculation module 1520 includes a sub-luminance map obtaining unit, a target luminance map obtaining unit, and a first direction and second direction phase difference unit, wherein,

the sub-brightness graph acquisition unit is used for acquiring a sub-brightness graph corresponding to each pixel point group in the original image according to the brightness value of the sub-pixel point at the same position of each pixel point in the pixel point group;

the target brightness graph acquisition unit is used for generating a target brightness graph according to the sub-brightness graph corresponding to each pixel point group;

and the first direction and second direction phase difference unit is used for calculating the phase difference of the first direction and the phase difference of the second direction according to the target brightness map.

In one embodiment, the phase difference unit in the first direction and the second direction is further configured to segment the target luminance map to obtain a first segmented luminance map and a second segmented luminance map, and determine a phase difference value of pixels matched with each other according to a position difference of the pixels matched with each other in the first segmented luminance map and the second segmented luminance map; and determining the phase difference value of the first direction and the phase difference value of the second direction according to the phase difference values of the matched pixels.

In an embodiment, the sub-luminance map obtaining unit is further configured to determine sub-pixel points at the same position from each pixel point to obtain a plurality of sub-pixel point sets, where positions of the sub-pixel points included in each sub-pixel point set in the pixel points are the same; for each sub-pixel point set, acquiring a brightness value corresponding to the sub-pixel point set according to the brightness value of each sub-pixel point in the sub-pixel point set; and generating a sub-brightness map according to the brightness value corresponding to each sub-pixel set.

The division of each module in the depth map generating apparatus is only used for illustration, and in other embodiments, the depth map generating apparatus may be divided into different modules as needed to complete all or part of the functions of the depth map generating apparatus.

Fig. 16 is a schematic internal structure diagram of an electronic device in one embodiment. As shown in fig. 16, the electronic apparatus includes a processor and a memory connected by a system bus. Wherein, the processor is used for providing calculation and control capability and supporting the operation of the whole electronic equipment. The memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The computer program may be executed by a processor for implementing a depth map generation method or a 3D modeling method provided in the above embodiments. The internal memory provides a cached execution environment for the operating system computer programs in the non-volatile storage medium. The electronic device may be a mobile phone, a tablet computer, or a personal digital assistant or a wearable device, etc.

The implementation of each module in the depth map generation apparatus provided in the embodiment of the present application may be in the form of a computer program. The computer program may be run on a terminal or a server. The program modules constituted by the computer program may be stored on the memory of the terminal or the server. Which when executed by a processor, performs the steps of the method described in the embodiments of the present application.

The process of the electronic device implementing the depth map generating method is as described in the above embodiments, and is not described herein again.

The embodiment of the application also provides a computer readable storage medium. One or more non-transitory computer-readable storage media containing computer-executable instructions that, when executed by one or more processors, cause the processors to perform the steps of the depth map generation method or the 3D modeling method.

A computer program product containing instructions which, when run on a computer, cause the computer to perform a depth map generation method.

Any reference to memory, storage, database, or other medium used by embodiments of the present application may include non-volatile and/or volatile memory. Suitable non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM), which acts as external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), synchronous Link (Synchlink) DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and bus dynamic RAM (RDRAM).

The above examples only express several embodiments of the present application, and the description thereof is more specific and detailed, but not to be construed as limiting the scope of the present application. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, and these are all within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims

1. A depth map generation method is applied to electronic equipment, the electronic equipment comprises at least two common cameras and at least one time of flight (TOF) camera, and the method comprises the following steps:

if the texture direction of the original image is judged to be consistent with the arrangement direction of the two common cameras in the electronic equipment, calculating the depth information of the original image through a TOF camera in the electronic equipment;

generating a depth map of the original image from the depth information.

2. The method of claim 1, further comprising:

when the accuracy of the depth information of a preset area in an original image calculated by a TOF camera in the electronic equipment is lower than a preset accuracy threshold, calculating first depth information of the preset area in the original image according to the target phase difference;

acquiring depth information of other regions except the preset region from the depth information of the original image calculated by the TOF camera in the electronic equipment;

and synthesizing the first depth information of a preset area in the original image with the depth information of other areas except the preset area to obtain the depth information of the original image.

3. The method according to claim 1, wherein after the determining whether the texture direction of the original image is consistent with the arrangement direction of two common cameras used for calculating the depth information in the electronic device, the method comprises:

and if the texture direction of the original image is judged to be inconsistent with the arrangement direction of two common cameras used for calculating the depth information in the electronic equipment, calculating the depth information of the original image through the two common cameras in the electronic equipment.

4. The method of claim 3, further comprising:

when the accuracy of the depth information of a preset region in the original image calculated by the two ordinary cameras in the electronic equipment is lower than a preset accuracy threshold, calculating first depth information of the preset region in the original image according to the target phase difference and calculating second depth information of the preset region in the original image according to a TOF camera in the electronic equipment;

screening out depth information with higher accuracy from first depth information and second depth information of a preset area in the original image to serve as target depth information of the preset area in the original image;

acquiring depth information of other areas except the preset area from the depth information of the original image calculated by the two ordinary cameras in the electronic equipment;

and synthesizing the target depth information of a preset area in the original image with the depth information of other areas except the preset area to obtain the depth information of the original image.

5. The method of claim 1, wherein the electronic device comprises an image sensor comprising a plurality of pixel groups arranged in an array, each pixel group comprising M x N pixels arranged in an array; each pixel point corresponds to one photosensitive unit; each pixel point comprises a plurality of sub pixel points which are arranged in an array, wherein M and N are both natural numbers which are more than or equal to 2;

the calculating of the phase difference in the first direction and the phase difference in the second direction for the original image includes:

for each pixel point group in the original image, acquiring a sub-brightness graph corresponding to the pixel point group according to the brightness value of the sub-pixel point at the same position of each pixel point in the pixel point group;

generating a target brightness map according to the sub-brightness map corresponding to each pixel point group;

and calculating the phase difference in the first direction and the phase difference in the second direction according to the target brightness map.

6. The method of claim 5, wherein calculating the phase difference in the first direction and the phase difference in the second direction from the target luminance map comprises:

performing segmentation processing on the target brightness image to obtain a first segmentation brightness image and a second segmentation brightness image, and determining the phase difference value of the mutually matched pixels according to the position difference of the mutually matched pixels in the first segmentation brightness image and the second segmentation brightness image;

and determining the phase difference value in the first direction and the phase difference value in the second direction according to the phase difference values of the mutually matched pixels.

7. The method according to claim 5, wherein the obtaining the sub-luminance graph corresponding to the pixel point group according to the luminance value of the sub-pixel point at the same position of each pixel point in the pixel point group comprises:

determining sub-pixel points at the same position from each pixel point to obtain a plurality of sub-pixel point sets, wherein the positions of the sub-pixel points in the pixel points included in each sub-pixel point set are the same;

for each sub-pixel point set, acquiring a brightness value corresponding to the sub-pixel point set according to the brightness value of each sub-pixel point in the sub-pixel point set;

and generating the sub-brightness graph according to the brightness value corresponding to each sub-pixel point set.

8. A method of 3D modeling, the method comprising:

when the object distance in the original image is larger than or equal to a preset object distance threshold value, judging whether the texture direction of the original image is consistent with the arrangement direction of two common cameras used for calculating depth information in the electronic equipment; if the texture direction of the original image is judged to be consistent with the arrangement direction of the two common cameras in the electronic equipment, calculating the depth information of the original image through a TOF camera in the electronic equipment;

generating a depth map of the original image from the depth information;

and 3D modeling is carried out according to the depth map of the original image.

9. An apparatus for generating a depth map, the apparatus comprising:

the target phase difference calculation module is used for calculating a phase difference in a first direction and a phase difference in a second direction for an original image when an object distance in the original image is smaller than a preset object distance threshold value, a preset included angle is formed between the first direction and the second direction, the preset object distance threshold value is a minimum object distance within an overlapped field angle range of a camera in electronic equipment, and a target phase difference is determined from the phase difference in the first direction and the phase difference in the second direction;

the depth map generation module is used for judging whether the texture direction of the original image is consistent with the arrangement direction of two common cameras used for calculating depth information in the electronic equipment or not when the object distance in the original image is larger than or equal to a preset object distance threshold value; if the texture direction of the original image is judged to be consistent with the arrangement direction of the two ordinary cameras in the electronic equipment, the depth information of the original image is calculated through a TOF camera in the electronic equipment; generating a depth map of the original image from the depth information.

10. An electronic device comprising a memory and a processor, the memory having stored thereon a computer program, wherein the computer program, when executed by the processor, causes the processor to perform the steps of the depth map generation method of any of claims 1 to 7 or the steps of the 3D modeling method of claim 8.

11. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 7 or the steps of the 3D modeling method according to claim 8.