CN110335211B

CN110335211B - Method for correcting depth image, terminal device and computer storage medium

Info

Publication number: CN110335211B
Application number: CN201910550733.5A
Authority: CN
Inventors: 杨鑫
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2019-06-24
Filing date: 2019-06-24
Publication date: 2021-07-30
Anticipated expiration: 2039-06-24
Also published as: CN110335211A

Abstract

The embodiment of the application discloses a method for correcting a depth image, terminal equipment and a computer storage medium, wherein the method comprises the following steps: acquiring an original image corresponding to a target object and a main color image and a secondary color image corresponding to the target object; the original image is obtained according to the acquisition of a target object by a time of flight (TOF) sensor, and the main color image and the secondary color image are obtained according to the acquisition of the target object by the double cameras; determining first depth information and a first confidence coefficient corresponding to a target object by using a preset double-shot algorithm according to the main color image and the auxiliary color image; determining second depth information corresponding to the target object according to the original image; determining an error data region in the first depth information based on the first depth information, the second depth information and the first confidence level; and correcting the error data area through the second depth information and the main color image to obtain target depth information, and obtaining a depth image according to the target depth information.

Description

Method for correcting depth image, terminal device and computer storage medium

Technical Field

The present application relates to the field of image processing technologies, and in particular, to a method for correcting a depth image, a terminal device, and a computer storage medium.

Background

With the rapid development of intelligent terminals, terminal devices such as mobile phones, palm computers, digital cameras, video cameras and the like have become an essential tool in the life of users, and great convenience is brought to various aspects of the life of the users. Existing terminal devices basically have a photographing function, and users can photograph various images using the terminal devices.

When an image with a blurring effect is captured, it is generally necessary for a terminal device to be configured with two cameras. Depth (depth) information is acquired through the two cameras, although the structure is simple, the hardware power consumption is low, and the resolution is high, the defects still exist, for example, the adaptability to scenes without textures, repeated textures, overexposure, underexposure and the like is poor, the accuracy of the acquired depth information is low, and the portrait blurring effect is influenced.

Disclosure of Invention

The main purpose of the present application is to provide a method for correcting a depth image, a terminal device, and a computer storage medium, which can repair the phenomenon that depth information in a double-shot portrait mode makes mistakes in depth in regions such as no texture, repeated texture, overexposure, and underexposure, thereby improving accuracy of depth in the double-shot portrait mode, and further optimizing accuracy of portrait blurring.

In order to achieve the purpose, the technical scheme of the application is realized as follows:

in a first aspect, an embodiment of the present application provides a method for correcting a depth image, where the method includes:

acquiring an original image corresponding to a target object and a main color image and a secondary color image corresponding to the target object; the original image is obtained according to the acquisition of a target object by a time of flight (TOF) sensor, and the main color image and the secondary color image are obtained according to the acquisition of the target object by two cameras;

determining first depth information and a first confidence coefficient corresponding to the target object by using a preset double-shot algorithm according to the main color image and the secondary color image; determining second depth information corresponding to the target object according to the original image;

determining an erroneous data region in the first depth information based on the first depth information, the second depth information, and the first confidence level;

and correcting the error data area through the second depth information and the main color image to obtain target depth information, and obtaining a depth image according to the target depth information.

In a second aspect, an embodiment of the present application provides a terminal device, where the terminal device includes: an acquisition unit, a determination unit, and a correction unit, wherein,

the acquisition unit is configured to acquire an original image corresponding to a target object and a primary color image and a secondary color image corresponding to the target object; the original image is obtained according to the acquisition of a target object by a TOF sensor, and the main color image and the secondary color image are obtained according to the acquisition of the target object by two cameras;

the determining unit is configured to determine first depth information and a first confidence coefficient corresponding to the target object by using a preset bi-shooting algorithm according to the main color image and the secondary color image; determining second depth information corresponding to the target object according to the original image; and further configured to determine an erroneous-data region in the first depth information based on the first depth information, the second depth information, and the first confidence level;

the correcting unit is configured to perform correction processing on the error data area through the second depth information and the main color image to obtain target depth information, and obtain a depth image according to the target depth information.

In a third aspect, an embodiment of the present application provides a terminal device, where the terminal device includes: a memory and a processor; wherein the content of the first and second substances,

the memory for storing a computer program operable on the processor;

the processor is configured to execute the method for correcting a depth image according to the first aspect when the computer program is executed.

In a fourth aspect, the present application provides a computer storage medium storing a depth image correction program, which when executed by at least one processor implements the depth image correction method according to the first aspect.

According to the depth image correction method, the terminal device and the computer storage medium provided by the embodiment of the application, an original image corresponding to a target object and a main color image and a secondary color image corresponding to the target object are obtained; the original image is obtained according to the acquisition of a target object by a TOF sensor, and the main color image and the secondary color image are obtained according to the acquisition of the target object by two cameras; then, according to the main color image and the secondary color image, determining first depth information and a first confidence coefficient corresponding to the target object by using a preset double-shot algorithm; determining second depth information corresponding to the target object according to the original image; determining an error data area in the first depth information based on the first depth information, the second depth information and the first confidence degree; finally, correcting the error data area through the second depth information and the main color image to obtain target depth information, and obtaining a depth image according to the target depth information; in this way, the first depth information is optimized through the second depth information, and the phenomenon that depth information makes mistakes in regions such as no texture, repeated texture, overexposure and underexposure in the double-shot portrait mode can be repaired, so that the accuracy of depth in the double-shot portrait mode is improved; in addition, the target depth information is mainly used for blurring the main color image, the accuracy of portrait blurring can be optimized, and the portrait blurring effect is improved.

Drawings

Fig. 1 is a schematic diagram of an explosive structure of a TOF camera according to an embodiment of the present disclosure;

fig. 2 is a schematic structural diagram of a double-shot blurring process according to an embodiment of the present disclosure;

fig. 3 is a schematic flowchart of a method for correcting a depth image according to an embodiment of the present disclosure;

fig. 4 is a schematic diagram of a hardware structure of a terminal device according to an embodiment of the present application;

FIG. 5 is a schematic diagram illustrating comparison of effects of epipolar line rectification provided by an embodiment of the present application;

fig. 6 is a schematic diagram illustrating an effect of a bi-camera parallax calculation according to an embodiment of the present application;

fig. 7 is a schematic diagram of a model for calculating depth information according to an embodiment of the present disclosure;

fig. 8 is a detailed flowchart of a method for correcting a depth image according to an embodiment of the present disclosure;

fig. 9 is a schematic diagram illustrating comparison of human image blurring effects provided in the embodiment of the present application;

fig. 10 is a schematic structural diagram of a terminal device according to an embodiment of the present application;

fig. 11 is a schematic hardware structure diagram of another terminal device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application.

In recent years, optical studies have been made more and more intensively due to rapid development of Time of Flight (TOF) technology. TOF is widely applied to terminal devices such as smart phones, palm computers, tablet computers, digital cameras and the like as a Three-dimensional (3D) imaging technology, can realize applications such as distance measurement, Three-dimensional modeling, photographing blurring and motion sensing games, and can also realize related applications of AR glasses in cooperation with Augmented Reality (AR) technology.

In general, a TOF camera may be composed of a light emitting module and a light receiving module. The light emitting module is also called a laser transmitter, a TOF transmitter, a transmitting illumination module, etc., and the light receiving module is also called a detector, a TOF receiver, a photosensitive receiving module, etc. Specifically, the light emitting module emits modulated near-infrared light, the modulated near-infrared light is reflected after encountering a shot object, then the light receiving module calculates the time difference or the phase difference between light emission and reflection, and then the time difference or the phase difference is converted to obtain the distance of the shot object, so that depth information is generated.

Referring to fig. 1, a schematic diagram of a blasting structure of a TOF camera provided in an embodiment of the present application is shown. As shown in fig. 1, the TOF camera 10 includes a light emitting module 110 and a light emitting receiving module 120; the light Emitting module 110 is composed of a diffuser (diffuser), a Photodiode (PD), a Vertical Cavity Surface Emitting Laser (VCSEL), a ceramic package, and the like; the light receiving module 120 is composed of a lens, a 940nm narrow-band filter, a TOF Sensor (TOF Sensor), and the like. Those skilled in the art will appreciate that the constituent structure shown in fig. 1 does not constitute a limitation of TOF camera, which may include more or fewer components than shown, or some components in combination, or a different arrangement of components.

It is understood that TOF can be divided into Direct-TOF (D-TOF) and Indirect-TOF (I-TOF) according to the result of the acquired signal. The D-TOF acquires the time difference, and the I-TOF acquires the phase shift (for example, the specific gravity of charges or voltages in different phases) of different target return signals, so as to calculate the distance of the shot object and generate depth information.

In addition, the I-TOF may be classified into a pulse Modulation (Pulsed Modulation) scheme and a Continuous Wave Modulation (Continuous Wave Modulation) scheme according to a Modulation scheme. The mainstream approach currently used by most manufacturers' terminal equipment is the continuous wave modulated indirect TOF scheme (which can be expressed as CW-I-TOF). For the CW-I-TOF scheme, each pixel comprises 2 capacitors, the light emitting module emits 4 square wave pulses with the pulse period delta t; and the light receiving module has a phase delay when receiving the pulse, each window is phase-delayed by 90 degrees, namely, one quarter of the pulse period (indicated by delta t/4), so that the phase delays are respectively 0 degrees, 180 degrees, 90 degrees and 270 degrees, which is also called a four-phase method. During exposure, the two capacitors of each pixel are charged in turn, the exposure time is equalized, and the difference of the exposure amount of the two capacitors is recorded as Q1, Q2, Q3 and Q4 respectively; the phase difference can be calculated by using the relationship between the charge difference and the flight phase

Obtaining the distance D of the shot object through the phase difference recalculation; wherein the content of the first and second substances,

when the angle corresponding to the distance of the object to be shot exceeds 2 pi, two phases with different frequencies are needed to solve the real distance. Assuming that the two phase values obtained are used separately

And

show that

Is extended to

Will be provided with

Is extended to

Then there will be a true distance such that the difference between the two corresponding distances is minimal, and the true distance can be determined.

TOF is used as an active depth sensor, and has been widely applied to terminal devices such as mobile phones, for example, a manufacturer uses TOF as a rear depth sensor. But the resolution of the TOF sensor is low, and the TOF sensor cannot be directly adapted to applications such as blurring and matting which have high requirements on the accuracy of the foreground edge. Therefore, the two-shot scheme is mainly used at present.

The double-camera scheme can comprise a main camera and an auxiliary camera, the main camera and the auxiliary camera can be RGB cameras, wherein RGB represents colors of three channels of Red (Red, R), Green (Green, G) and Blue (Blue, B), the colors of the three channels are mixed or overlapped according to different proportions, and all colors perceived by human vision in an image can be obtained. The double-shot scheme is used as portrait blurring and has become a standard matching function of the terminal device, the double-shot blurring process is shown in fig. 2, and the double-shot blurring function is completed through four steps of double-shot marking, epipolar line correction, stereo matching, scene blurring and the like. However, although the depth information obtained by the double-shot scheme has more advantages, such as simple structure, low hardware power consumption and high resolution of the depth information, most of indoor and outdoor scenes can be considered; however, the double shot scheme has poor adaptability to scenes such as no texture, repeated texture, overexposure, underexposure and the like, so that depth information generated by the double shot scheme in regions such as no texture, repeated texture, overexposure, underexposure and the like may have errors, thereby causing blurring errors.

The embodiment of the application provides a method for correcting a depth image, which is applied to terminal equipment. Obtaining an original image corresponding to a target object and a main color image and a secondary color image corresponding to the target object; the original image is obtained according to the acquisition of a target object by a TOF sensor, and the main color image and the secondary color image are obtained according to the acquisition of the target object by two cameras; then, according to the main color image and the secondary color image, determining first depth information and a first confidence coefficient corresponding to the target object by using a preset double-shot algorithm; determining second depth information corresponding to the target object according to the original image; determining an error data area in the first depth information based on the first depth information, the second depth information and the first confidence degree; finally, correcting the error data area through the second depth information and the main color image to obtain target depth information, and obtaining a depth image according to the target depth information; in this way, the first depth information is optimized through the second depth information, and the phenomenon that depth information makes mistakes in regions such as no texture, repeated texture, overexposure and underexposure in the double-shot portrait mode can be repaired, so that the accuracy of depth in the double-shot portrait mode is improved; in addition, the target depth information is mainly used for blurring the main color image, the accuracy of portrait blurring can be optimized, and the portrait blurring effect is improved.

Embodiments of the present application will be described in detail below with reference to the accompanying drawings.

Referring to fig. 3, a schematic flow chart of a method for correcting a depth image according to an embodiment of the present application is shown. As shown in fig. 3, the method may include:

s301: acquiring an original image corresponding to a target object and a main color image and a secondary color image corresponding to the target object; the original image is obtained according to the acquisition of a target object by a TOF sensor, and the main color image and the secondary color image are obtained according to the acquisition of the target object by two cameras;

it should be noted that the method is applied to a terminal device, and the terminal device includes components such as a TOF sensor and dual cameras (a main camera and a sub camera). Therefore, an original image corresponding to the target object can be acquired through the TOF sensor, and a main color image and an auxiliary color image corresponding to the target object can be acquired through the double-camera acquisition, so that the subsequent calculation of depth information is facilitated.

It is also noted that the terminal device may be implemented in various forms. For example, the terminal devices described in the present application may include mobile terminals such as mobile phones, tablet computers, notebook computers, palmtop computers, Personal Digital Assistants (PDAs), wearable devices, Digital cameras, video cameras, and the like, and stationary terminals such as Digital TVs, desktop computers, and the like; the embodiments of the present application are not particularly limited.

In some embodiments, for S301, the acquiring an original image corresponding to a target object and a primary color image and a secondary color image corresponding to the target object may include:

s301 a: collecting the target object through a TOF sensor to obtain an original image corresponding to the target object;

s301 b: acquiring the target object through two cameras to obtain a main color image of the target object under the main camera and a secondary color image of the target object under the secondary camera; wherein, two cameras include main camera and vice camera.

It should be noted that, through the acquisition of the target object by the TOF sensor, an original image corresponding to the target object, such as a group of RAW images, can be obtained; acquiring a target object through two cameras to obtain a corresponding main color image under a main camera, such as an RGB main image; and a corresponding secondary color image under the secondary camera, such as an RGB secondary image. In this way, depth information corresponding to a double-shooting mode can be obtained according to the main color image and the auxiliary color image acquired by the double cameras, and the embodiment of the application is represented by the first depth information; depth information corresponding to the TOF mode can be obtained according to the original image acquired by the TOF sensor, and the embodiment of the application is represented by second depth information.

For example, refer to fig. 4, which shows a schematic diagram of a hardware structure of a terminal device provided in an embodiment of the present application. As shown in fig. 4, the terminal device may include an Application Processor (AP), a main camera, a sub camera, a TOF sensor, and a Laser (Laser) transmitter; wherein, the AP side includes a First Image Signal Processor (ISP 1), a Second Image Signal Processor (ISP 2) and a Mobile Industry Processor Interface (MIPI); in addition, a preset algorithm, such as a preset double shot algorithm, a preset calibration algorithm, etc., is further placed on the AP side, and the embodiment of the present application is not particularly limited.

With reference to the terminal device shown in fig. 4, in the dual shooting mode, when the terminal device performs image acquisition on a target object, the AP side may connect two paths of cameras through two ISPs, respectively, to obtain two paths of RGB data, and simultaneously, frame synchronization and 3A synchronization in the dual shooting mode are ensured; the 3A synchronization includes synchronization of Auto Focus (AF), Auto Exposure (AE), and Auto White Balance (AWB). In fig. 4, a main camera collects a target object, and sends an obtained main color image (one path of RGB data) to an ISP 1; the target object is collected by the secondary camera, and the obtained secondary color image (the other path of RGB data) is sent to the ISP 2. In addition, the terminal device may also ensure the exposure timing requirements of the Laser (Laser) and the Infrared (IR) by driving an Integrated Circuit (IC), and require that the IR exposure and the RGB exposure corresponding to the main camera are synchronous; specifically, the synchronization may be implemented using a software synchronization method or a hardware synchronization method. And calculating to obtain first depth information corresponding to the target object by combining a preset algorithm placed on the AP side. In the TOF mode, the terminal device may acquire a group of RAW maps through a TOF sensor to acquire second depth information corresponding to the target object. Therefore, the depth of the first depth information and the depth of the second depth information in the main shooting coordinate system can be fused subsequently, the error area in the first depth information in the double shooting mode can be corrected, and the purpose of improving the accuracy of the depth is achieved.

S302: determining first depth information and a first confidence coefficient corresponding to the target object by using a preset double-shot algorithm according to the main color image and the secondary color image; determining second depth information corresponding to the target object according to the original image;

it should be noted that the first confidence level is used to characterize the accuracy of the first depth information, and the preset bi-camera algorithm is used to represent a preset algorithm or model based on the stereo matching of the dual cameras. Specifically, according to the main color image and the sub color image, the first depth information and the first confidence coefficient are calculated through a preset double-shot algorithm, so that the first depth information and the first confidence coefficient in a double-shot mode can be obtained; the depth information is calculated according to the original image, and second depth information in a TOF mode can be acquired; in this way, the first depth information in the bi-shooting mode can be subsequently optimized according to the second depth information in the TOF mode.

It should be noted that, the first depth information is calculated according to the main color image and the sub-color image, and the obtained first depth information is originally in the main shooting coordinate system, so that the coordinate system conversion of the first depth is not needed; the second depth information is calculated from the original image, and the obtained second depth information is in the TOF coordinate system, so that coordinate system conversion needs to be performed on the second depth to align the second depth into the main shooting coordinate system.

Generally, the main shooting resolution is high, and the resolution for generating depth information is also high, and for example, the resolution is 2584 × 1938 by taking 400 ten thousand cameras as an example; whereas TOF resolution is lower, the resolution of generating depth information is lower, for example, the resolution is 320 × 240; that is, the resolution of the first depth information is higher than the resolution of the second depth information. Therefore, after the second depth information is aligned to the main shooting coordinate system, pixel points corresponding to the second depth information are sparse, and some sparse effective pixel points can be provided for follow-up.

S303: determining an erroneous data region in the first depth information based on the first depth information, the second depth information, and the first confidence level;

it should be noted that the erroneous data region refers to a region formed by pixels with depth errors in the first depth information, and the erroneous data region is usually located in a low confidence region of the first depth information; wherein the low confidence region is determined by the first confidence.

Under the double-shot mode, depth errors exist in areas such as no texture and repeated texture of the first depth information; in order to improve the depth accuracy in the dual-shooting mode, the embodiment of the application needs to determine the erroneous data area, so that the subsequent depth calibration of the erroneous data area is facilitated. In this way, after the first depth information and the second depth information are both aligned to the main shooting coordinate system, according to the first confidence, a low confidence region in the first depth information can be determined; then, in the low confidence region, through the first depth information and the second depth information, an error data region in the first depth information can be calculated, so that subsequent calibration can be performed on the error data region.

S304: and correcting the error data area through the second depth information and the main color image to obtain target depth information, and obtaining a depth image according to the target depth information.

It should be noted that after the error data region in the first depth information is obtained, interpolation and repair processing may be performed on the pixel points in the error data region through the second depth information, so as to obtain new depth information. The error data area in the first depth information is obtained by interpolation and repair processing of the second depth information, and the non-error data area in the first depth information retains the original first depth information, so that new depth information is obtained for the first depth information; as can be seen, the new depth information is obtained by fusing the first depth information and the second depth information; in order to weaken the trace of artificial synthesis, new depth information can be filtered through the main color image, the finally output depth information is target depth information, and a required depth image can be obtained according to the target depth information, so that the phenomenon that the depth information makes mistakes in regions depth such as no texture, repeated texture, overexposure and underexposure in a double-shot mode is solved, and the accuracy of the depth information is improved.

Further, in some embodiments, after S304, the method may further include:

and performing blurring processing on the main color image according to the depth image to obtain a target image.

It should be noted that, the blurring processing is performed on the main color image according to the acquired depth image, so that a required target image can be obtained; wherein the target image may be a shot image. In addition, because the algorithm of the embodiment of the application has higher complexity, the depth image correction method is mainly applied to a photographing mode of double-shot portrait blurring and cannot be applied to a preview mode. Specifically, the embodiment of the application mainly aims at the application of blurring and matting and the like, and optimizes the depth information of the double-shot portrait by combining the advantages of the TOF. Therefore, the target image is obtained by blurring the main color image according to the acquired depth image, so that the target image restores the phenomenon that depth information in a double-shot portrait mode makes mistakes in areas such as no texture, repeated texture, overexposure and underexposure, the accuracy of depth in the double-shot portrait mode is improved, and the accuracy of portrait blurring is optimized.

The embodiment provides a method for correcting a depth image, which is applied to a terminal device. Acquiring an original image corresponding to a target object and a main color image and a secondary color image corresponding to the target object; the original image is obtained according to the acquisition of a target object by a TOF sensor, and the main color image and the secondary color image are obtained according to the acquisition of the target object by two cameras; then, according to the main color image and the secondary color image, determining first depth information and a first confidence coefficient corresponding to the target object by using a preset double-shot algorithm; determining second depth information corresponding to the target object according to the original image; determining an error data area in the first depth information based on the first depth information, the second depth information and the first confidence degree; finally, correcting the error data area through the second depth information and the main color image to obtain target depth information, and obtaining a depth image according to the target depth information; therefore, the second depth information is obtained by the TOF mode, the first depth information is obtained by the double-shot mode, and the first depth information is optimized through the second depth information, so that the phenomenon that depth information in the double-shot portrait mode makes mistakes in regions such as no texture, repeated texture, overexposure, underexposure and the like can be repaired, the optimization of the TOF on the double-shot portrait mode is realized, and the accuracy of depth in the double-shot portrait mode is improved; in addition, the target depth information is mainly used for blurring the main color image, the accuracy of portrait blurring can be optimized, and the portrait blurring effect is improved.

In another embodiment of the present application, distortion is introduced due to lens accuracy and process of the camera to cause image distortion; in the double-shooting mode, the optical axes of the main camera and the auxiliary camera are not parallel; at this time, distortion correction and epipolar line correction processing are also required for the main color image and the sub color image before calculating the first depth information in the bi-shooting mode. Therefore, in some embodiments, for S302, the determining, according to the main color image and the sub color image, the first depth information and the first confidence degree corresponding to the target object by using a preset bi-shooting algorithm may include:

s302 a: carrying out distortion correction processing on the main color image to obtain a corrected main color image;

s302 b: carrying out distortion correction and epipolar line correction processing on the secondary color image to obtain a corrected secondary color image;

note that the imaging process of the main camera or the sub camera is actually a process of converting coordinate points of the world coordinate system to the main camera coordinate system. Distortion correction of the primary and secondary color images is required because distortion is introduced by the lens accuracy and process of the camera (so-called distortion, in particular, when a straight line in the world coordinate system is converted to another coordinate system and is not a straight line any more), thereby causing image distortion. In addition, in order to achieve that the optical axes of the main camera and the sub camera are completely parallel, so that the heights of the same pixel point of the target object in the main color image and the sub color image are consistent, epipolar correction needs to be performed on the sub color image, for example, a Bouguet epipolar correction algorithm can be adopted. Specifically, since the optical axes (also referred to as baselines) between the main camera and the sub-camera before correction are not parallel, the object of epipolar line correction is that the optical axes between the main camera and the sub-camera are perfectly parallel; thus, after the distortion correction and the epipolar correction, the images of binocular Vision can be parallel in accordance with the same Field of view (FOV) standard.

It should be noted that after the main color image and the sub-color image are obtained, the main color image and the sub-color image may be scaled according to a preset ratio to obtain a low-resolution color image; and then carrying out distortion correction and epipolar line correction on the low-resolution color image according to the calibration parameters corresponding to the double cameras so as to obtain a corrected main color image and a corrected secondary color image. The calibration parameters corresponding to the two cameras can be obtained by calculation according to a preset calibration algorithm, such as a Zhangyingyou calibration method; or may be obtained from direct supply from the manufacturer or supplier of the dual cameras. In addition, the preset ratio is a ratio value preset according to the target resolution, and in practical application, the preset ratio is set according to an actual situation, and the embodiment of the present application is not particularly limited.

Because polar lines between the main camera and the auxiliary camera arranged on the terminal equipment are not parallel, the height of the same pixel point in the target object in the main color image is not consistent with the height of the pixel point in the auxiliary color image, and after polar line correction is carried out, the height of the same pixel point in the target object in the main color image is consistent with the height of the same pixel point in the auxiliary color image. Therefore, when the main color image and the auxiliary color image are subjected to stereo matching, matched pixel points only need to be searched on the same line.

Exemplarily, refer to fig. 5, which shows a schematic diagram comparing effects of epipolar line rectification provided by the embodiments of the present application. In fig. 5, before epipolar line correction is performed on the main camera and the sub camera, the optical axes of the two are not parallel, as shown in (a); at this time, for the pixel 1 of the target object, the height in the obtained main color image does not coincide with the height of the sub color image, as shown in (b); in order to be able to calculate the first depth information, it is necessary to perform epipolar line correction on the main camera and the sub-camera, so that after the epipolar line correction, the optical axes of the main camera and the sub-camera are completely parallel, as shown in (c); at this time, the height of the pixel 1 in the primary color image is identical to the height of the pixel 1 in the secondary color image, as shown in (d); therefore, when the main color image and the auxiliary color image are subjected to stereo matching, matched pixel points only need to be searched on the same line, and the efficiency is greatly improved.

S302 c: for each pixel in the target object, based on the corrected main color image and the corrected sub-color image, determining first depth information corresponding to each pixel and a first confidence corresponding to each pixel by using a preset double-shot algorithm; wherein the first confidence level is used to characterize the accuracy of the first depth information.

It should be noted that after the corrected main color image and the corrected sub-color image are acquired, the first depth information and the first confidence corresponding to each pixel in the target object may be determined according to the corrected main color image and the corrected sub-color image; wherein the first depth information and the first confidence are in pixel units.

Further, in some embodiments, for S302c, the determining the first depth information corresponding to each pixel based on the corrected primary color image and the corrected secondary color image may include:

performing parallax matching calculation on the corrected main color image and the corrected auxiliary color image through a double-shot matching algorithm to obtain a parallax value corresponding to each pixel;

and performing depth conversion on the parallax value through a first preset conversion model to obtain first depth information corresponding to each pixel.

It should be noted that the bi-shooting matching algorithm is a preset algorithm or model for calculating the parallax, and belongs to a classical algorithm for calculating the parallax in a preset bi-shooting algorithm; the bi-shooting Matching algorithm may be a Semi-Global Matching (SGM) algorithm, or a Cross-Scale Cost Aggregation (CSCA) algorithm, and the embodiment of the present application is not limited in particular. Exemplarily, refer to fig. 6, which shows an effect schematic diagram of a bi-camera parallax calculation provided by the embodiment of the present application. In fig. 6, it can be seen that the parallax effect map shown in (c) is finally obtained by performing parallax matching calculation on the two images of (a) and (b).

In addition, the first preset conversion model is a preset model for parallax depth conversion, and generally refers to a triangulation model for calculating depth information by using a parallax value and preset imaging parameters; the preset imaging parameters in the embodiment of the present application may include a baseline distance and a focal length. For example, the first predetermined conversion model may be Z ═ Baseline ═ focal/disparity, where Z denotes depth information, Baseline denotes a distance from a base line or an optical axis, focal denotes a focal length, and disparity denotes a disparity value; however, the first predetermined conversion model is not particularly limited in the embodiment of the present application.

For example, refer to fig. 7, which shows a schematic diagram of a model for calculating depth information according to an embodiment of the present application. As shown in FIG. 7, O_RIs the position of the main camera, O_TIs the position of the sub-camera, O_RAnd O_TThe distance between is the baseline distance, denoted b; p is the position of the target object, P₁For the terminal device to acquire the image point obtained when the target object P is acquired through the main camera, P₁' is an image point, x, obtained when the terminal device collects the target object P through the auxiliary camera_RIs a target object's image point P₁Coordinates in the main color image, x_TIs a target object's image point P₁' coordinates in the secondary color image,f is the focal length between the main camera and the auxiliary camera. At this time, it can be known from the similar triangle

Further, it can obtain

Where d is the disparity value. Therefore, the terminal device only needs to know the baseline distance b, the focal length f and the parallax value d, and then can convert the distance according to the first predetermined conversion model (for example, according to the first predetermined conversion model

) And calculating first depth information corresponding to each pixel point.

The determination of the first confidence level may also be performed after the corrected primary color image and the corrected secondary color image are acquired. In this embodiment of the present application, the first confidence may be calculated according to a matching similarity cost between the corrected main color image and the corrected sub-color image, or may be calculated according to a texture gradient difference corresponding to the main color image and the sub-color image, which is not specifically limited in this embodiment of the present application.

Further, in some embodiments, for S302c, the determining a first confidence corresponding to each pixel based on the corrected primary color image and the corrected secondary color image may include:

performing matching similarity calculation on the corrected main color image and the corrected auxiliary color image to obtain matching similarity cost corresponding to each pixel;

and determining a first confidence corresponding to each pixel based on the matching similarity cost.

It should be noted that, by performing matching similarity calculation on the corrected main color image and the corrected sub-color image, matching similarity costs corresponding to each pixel in the target object can be obtained; the specific calculation mode of matching the similarity cost may be set according to an actual situation in practical application, and the embodiment of the present application is not particularly limited.

It should be further noted that after the matching similarity cost is obtained, the first confidence corresponding to each pixel may be further determined according to the matching similarity cost. Specifically, the terminal device may set a cost threshold, and compare the matching similarity cost obtained for each pixel with the cost threshold to determine a first confidence level; for example, when the matching similarity cost corresponding to the pixel is greater than the cost threshold, it indicates that the corrected primary color image and the corrected secondary color image still have a greater probability of being a false match for the pixel, and the first confidence corresponding to the pixel is lower. In addition, the matching similarity calculation is carried out on each pixel, so that the minimum matching similarity cost and the second minimum matching similarity cost can be obtained; if the minimum matching similarity cost is closer to the next smallest matching similarity cost, it may also indicate that the first confidence corresponding to the pixel is lower.

calculating a first texture gradient corresponding to each pixel under the corrected main color image;

based on the first texture gradient, a first confidence corresponding to each pixel is determined.

It should be noted that the first confidence level may also relate to texture richness. According to the corrected main color image, a first texture gradient corresponding to each pixel can be calculated; thus, from the first texture gradient, a first confidence level may be determined. In practical application, the calculation mode of the specific texture gradient may be set according to practical situations, and the embodiment of the present application is not particularly limited.

The embodiment provides a method for correcting a depth image, which is applied to a terminal device. The specific implementation of the foregoing embodiment is elaborated in detail in this embodiment, and it can be seen that, by the technical scheme of this embodiment, the phenomenon that depth information is erroneous in regions depth such as no texture, repeated texture, overexposure, underexposure and the like in the dual-shot portrait mode can be repaired, so that optimization of the TOF on the dual-shot portrait mode is achieved, and accuracy of depth in the dual-shot portrait mode is improved; in addition, the target depth information is mainly used for blurring the main color image, the accuracy of portrait blurring can be optimized, and the portrait blurring effect is improved.

In another embodiment of the present application, the first depth information is calculated according to the main color image and the sub-color image, and the obtained first depth information is originally in the main shooting coordinate system, so that the coordinate system conversion of the first depth is not needed; the second depth information is calculated from the original image, and the obtained second depth information is in the TOF coordinate system, so that coordinate system conversion needs to be performed on the second depth to align the second depth into the main shooting coordinate system. Therefore, in some embodiments, for S302, the determining, according to the original image, second depth information corresponding to the target object may include:

s302 d: obtaining initial depth information of each pixel in the target object under a TOF coordinate system according to the original image;

it should be noted that, the TOF sensor is used to acquire the target object, and obtain an original image (for example, a group of RAW images) corresponding to the target object; thus, the initial depth information of each pixel in the target object under the TOF coordinate system can be obtained by calculating the depth information according to the original image; the calculation method of the initial depth information may adopt a four-phase method.

S302 e: and converting the coordinate system of the initial depth information through a second preset conversion model to obtain second depth information of each pixel in the main shooting coordinate system.

It should be noted that the second preset conversion model is a preset model for coordinate system conversion, such as converting coordinates in the TOF coordinate system into the main shooting coordinate system. In this way, the initial depth information in the TOF coordinate system can be converted into the second depth information in the main camera coordinate system according to the second preset conversion model, so as to realize pixel alignment.

Further, in some embodiments, prior to S302e, the method may further include:

calibrating the TOF sensor and the double cameras according to a preset calibration algorithm to obtain calibration parameters;

correspondingly, the performing coordinate system conversion on the initial depth information through a second preset conversion model to obtain second depth information of each pixel in the main shooting coordinate system may include:

and converting the initial depth information into a main shooting coordinate system based on the calibration parameters and a second preset conversion model to obtain second depth information of each pixel in the main shooting coordinate system.

It should be noted that before performing pixel alignment, dual camera calibration needs to be performed on the TOF sensor and the dual cameras. The calibration parameter may be obtained by calculation according to a preset calibration algorithm, such as a Zhangyingyou calibration method; the two-camera system can also be obtained by direct supply from manufacturers or suppliers of the two-camera system, and the embodiment of the application is not particularly limited.

Therefore, after the calibration parameters are obtained, the initial depth information can be converted into the main shooting coordinate system according to the calibration parameters and the second preset conversion model, and the second depth information of each pixel in the main shooting coordinate system is obtained, so that the pixel alignment of the first depth information and the second depth information is realized, the subsequent fusion of the first depth information and the second depth information is facilitated, and the correction of an error data area in the first depth information is realized. Note that, in general, the main-shooting resolution is larger, and the resolution of generating depth information thereof is generally larger than that of TOF generating depth information; therefore, after the initial depth information is aligned to the main shooting coordinate system, pixels in the obtained second depth information are sparse, and the method is favorable for providing some sparse effective pixel points for subsequent depth information fusion.

In still another embodiment of the present application, for an erroneous data region in the first depth information, the erroneous data region is generally located in a low confidence region of the first depth information, and may be specifically determined by judging a difference between the first depth information and the second depth information. Thus, in some embodiments, for S303, the determining an erroneous-data region in the first depth information based on the first depth information, the second depth information, and the first confidence level may include:

s303 a: determining a low confidence region in the first depth information according to the first confidence;

it should be noted that the low confidence region in the first depth information may be determined by a first confidence, and the first confidence is related to the matching similarity cost and the texture richness. That is, the low confidence region in the first depth information may be determined according to a matching similarity cost between the pixel in the first depth information and the pixel in the second depth information, and the low confidence region in the first depth information may also be determined according to a texture gradient of the pixel in the first depth information and a texture gradient of the pixel in the second depth information.

In addition, it is assumed that the first confidence threshold is a determination value for measuring whether the first confidence belongs to a low confidence; in this way, the judgment can be performed according to the first confidence threshold value acquired in advance. Specifically, the first confidence is compared to a first confidence threshold; when the first confidence is smaller than the first confidence threshold, it is indicated that the pixel corresponding to the first confidence belongs to the low confidence region, so that the low confidence region in the first depth information can be obtained.

S303 b: calculating a difference value between first depth information corresponding to each pixel to be judged and second depth information corresponding to the pixel to be judged in an effective neighborhood of the pixel to be judged aiming at each pixel to be judged in the low confidence coefficient area;

it should be noted that the effective neighborhood in the second depth information refers to a region closest to the pixel to be determined, and the size of the effective neighborhood is limited, for example, may be 5 × 5 or 7 × 7; however, the size of the effective neighborhood can be set according to actual conditions in practical application, and the embodiment of the present application is not particularly limited.

In this way, an effective pixel point associated with the second depth information exists in the effective neighborhood, so that the difference value between the first depth information corresponding to the pixel to be judged and the second depth information corresponding to the effective pixel point can be calculated; and whether the pixel to be judged is an error point or not is further determined according to the size of the difference value.

S303 c: comparing the difference value with a preset difference value threshold value;

it should be noted that the preset difference threshold is a preset determination value for determining whether a pixel to be determined is an error point. Thus, after step S303c, when the difference is greater than the preset difference threshold value according to the comparison result of the difference with the preset difference threshold value, step S303d is performed; when the difference is not greater than the preset difference threshold, step S303e is performed.

S303 d: when the difference value is larger than a preset difference value threshold value, marking the pixel to be judged as an error point, and obtaining an error data area in the first depth information according to the marked error point;

s303 e: and when the difference is not greater than a preset difference threshold, reserving the first depth information corresponding to the pixel to be judged to obtain a reserved data area in the first depth information.

It should be noted that, in the dual shooting mode, depth of the first depth information in the regions without texture, repeated texture, etc. is erroneous; in order to improve the accuracy of depth in the bi-shooting mode, in the embodiment of the present application, the error data area needs to be determined, so as to distinguish the error data area from the original data area for the first depth information. Specifically, after the first depth information and the second depth information are both aligned to the main shooting coordinate system, according to the first confidence, a low confidence region in the first depth information can be determined; then in the low confidence coefficient area, calculating the difference value of the first depth information corresponding to the pixel to be judged and the second depth information corresponding to the effective neighborhood of the pixel to be judged; comparing the difference value with a preset difference value threshold value; when the difference value is larger than a preset difference value threshold value, the pixel to be judged can be marked as an error point, and an error data area in the first depth information is obtained according to the marked error point; when the difference is not greater than the preset difference threshold, it is indicated that the first depth information corresponding to the pixel to be determined is correct, and at this time, the first depth information corresponding to the pixel to be determined needs to be retained, so as to obtain a retained data area in the first depth information, that is, an original data area in the first depth information, and thus, the first depth information is divided into an error data area and an original data area.

Further, in some embodiments, the method may further comprise:

s303 f: and if the effective pixel point associated with the second depth information does not exist in the effective neighborhood of the pixel to be judged, the step of comparing the difference value with a preset difference value threshold value is not executed.

Note that after the initial depth information generated by TOF is aligned to the main shooting coordinate system, pixels in the obtained second depth information are sparse; thus, for the second depth information, there may be no valid pixel point associated with the second depth information in the valid neighborhood of the pixel to be determined. That is to say, when there is no effective pixel point corresponding to the second depth information in the effective neighborhood of the pixel to be determined, the step of comparing the difference value with the preset difference value threshold cannot be executed at this time, and the first depth information corresponding to the pixel to be determined can be retained.

Further, in some embodiments, for S304, performing a correction process on the erroneous data area through the second depth information and the main color image to obtain target depth information may include:

s304 a: for each error point in the error data area, performing weighted interpolation calculation on the error data area through second depth information corresponding to each error point, and replacing the error data area in the first depth information with the calculated depth information to obtain new depth information;

s304 b: and filtering the new depth information according to the main color image to obtain the target depth information.

It should be noted that the error points included in the error data region may utilize second depth information corresponding to effective pixel points in an effective neighborhood, and may also perform weighted interpolation calculation on the error data region by combining color similarity and weight of spatial distance, at this time, the calculated depth information may be obtained; replacing the error data area in the first depth information by using the calculated depth information to obtain new depth information; as can be seen, the new depth information is obtained by fusing the first depth information and the second depth information; in order to weaken the trace of artificial synthesis, the main color image can be used as a guide to filter new depth information, so that the trace of artificial synthesis is weakened, a depth image can be obtained according to the output target depth information, and the correction processing of the error data area is realized.

The Filter processing methods include guided Filter (Guide Filter), DT Filter (Domain Transform Filter), weighted median Filter (Weight Medium Filter), and the like; in practical application, the setting may be performed according to actual situations, and the embodiment of the present application is not particularly limited.

Exemplarily, refer to fig. 8, which shows a detailed flowchart of a method for correcting a depth image according to an embodiment of the present application. As shown in fig. 8, the terminal device acquires a RAW image through a TOF sensor, which may be composed of a group of RAW maps; the terminal equipment acquires a main color image through the main camera, and acquires an auxiliary color image through the auxiliary camera; then distortion correction processing is carried out on the main color image, distortion correction and epipolar line correction processing are carried out on the secondary color image, and the corrected main color image and the corrected secondary color image can be obtained respectively; matching the corrected main color image and the corrected auxiliary color image by using a double-shot matching algorithm to obtain first depth information and a first confidence coefficient; then, depth information calculation is carried out on the original image, so that initial depth information can be obtained; the initial depth information is in a TOF coordinate system and needs to be converted into a main shooting coordinate system, so that second depth information is obtained, and pixel alignment of the second depth information and the first depth information is realized; determining a low confidence coefficient region in the first depth information according to the first confidence coefficient, and then calculating a difference value between the first depth information and the second depth information in the low confidence coefficient region; judging whether the difference is larger than a preset difference threshold value or not, namely judging the safety of the difference; when the difference is larger than a preset difference threshold, the difference is unsafe, the pixel to be judged is marked as an error point at the moment, an error data area in the first depth information is obtained, and then the second depth information is utilized to carry out correction processing (such as repair, filling, interpolation processing and the like) on the error data area; when the difference value is not greater than the preset difference value threshold value, the difference value is safe, and at the moment, first depth information corresponding to the pixel to be judged can be reserved; and finally, fusing the two images and outputting a final depth image.

After the depth image is acquired, the main color image can be subjected to blurring processing according to the depth image, and accuracy of portrait blurring can be improved. Referring to fig. 9, a schematic diagram illustrating a comparison of portrait blurring effects provided by the embodiment of the present application is shown. As shown in fig. 9, (a) and (b) both are background repeated texture regions, where (a) a portrait blurring effect in the bi-shot mode is provided, and (b) a portrait blurring effect in the bi-shot mode + TOF mode is provided; therefore, the portrait blurring effect in the double shot mode + TOF mode is better.

In yet another embodiment of the present application, since the TOF mode is poor outdoors, a large number of holes exist in the second depth information, and the TOF mode is not suitable for performing the depth image correction method according to the embodiment of the present application. Therefore, in some embodiments, after the determining, according to the original image, the second depth information corresponding to the target object, the method further includes:

determining a second confidence corresponding to the target object according to the original image; wherein the second confidence level is used to characterize the accuracy of the second depth information;

determining the number of holes in the second depth information based on the second confidence;

and if the number of the holes is larger than a preset hole threshold value, not executing the correction method of the depth image.

It should be noted that, because the TOF mode has a poor effect outdoors, the number of holes or the determination of the hole rate may be increased. If a large number of holes or low-confidence regions exist in the second depth information in the TOF mode, the depth image correction method (specifically, fusion of the first depth information and the second depth information) described in the embodiment of the present application may not be executed at this time, and the depth image is still acquired in the normal bi-shooting mode. Specifically, assuming that the preset hole threshold is a determination value used for measuring whether the number of holes is excessive, the number of holes in the second depth information may be determined according to the second confidence; if the number of the holes is larger than the preset hole threshold, it indicates that the number of the holes is too large, and at this time, the depth image correction method of the embodiment of the present application may not be executed, and only the normal double shot mode is adopted to obtain the depth image.

In addition, the resolution of the depth information generated by the TOF sensor in the terminal equipment is low, and the depth information is mainly used for correcting depth errors of relatively large areas; for the scenes with hollowing and rich depth levels, the correction effect is not good. Therefore, the depth image correction method according to the embodiment of the present application is mainly applied to use in the double-shot portrait mode.

Based on the same inventive concept of the foregoing embodiment, refer to fig. 10, which shows a schematic structural diagram of another terminal device 100 provided in the embodiment of the present application. As shown in fig. 10, the terminal device 100 may include: an acquisition unit 1001, a determination unit 1002, and a correction unit 1003, wherein,

the acquiring unit 1001 is configured to acquire an original image corresponding to a target object and a primary color image and a secondary color image corresponding to the target object; the original image is obtained according to the acquisition of a target object by a TOF sensor, and the main color image and the secondary color image are obtained according to the acquisition of the target object by two cameras;

the determining unit 1002 is configured to determine, according to the main color image and the sub color image, first depth information and a first confidence degree corresponding to the target object by using a preset bi-shooting algorithm; determining second depth information corresponding to the target object according to the original image; and further configured to determine an erroneous-data region in the first depth information based on the first depth information, the second depth information, and the first confidence level;

the correcting unit 1003 is configured to perform correction processing on the error data area through the second depth information and the main color image to obtain target depth information, and obtain a depth image according to the target depth information.

In the foregoing solution, referring to fig. 10, the terminal device 100 may further include a blurring unit 1004 configured to perform blurring processing on the main color image according to the depth image to obtain a target image.

In the above scheme, referring to fig. 10, the terminal device 100 may further include an acquisition unit 1005 configured to acquire the target object through a TOF sensor, and acquire an original image corresponding to the target object; the target object is collected through the two cameras, and a main color image corresponding to the target object under the main camera and a secondary color image corresponding to the target object under the secondary camera are obtained; wherein, two cameras include main camera and vice camera.

In the above solution, the correcting unit 1003 is further configured to perform distortion correction processing on the main color image to obtain a corrected main color image; and carrying out distortion correction and epipolar correction on the secondary color image to obtain a corrected secondary color image;

the determining unit 1002 is specifically configured to determine, for each pixel in the target object, first depth information corresponding to each pixel and a first confidence corresponding to each pixel based on the corrected main color image and the corrected sub-color image by using a preset bi-shooting algorithm; wherein the first confidence level is used to characterize the accuracy of the first depth information.

In the above scheme, referring to fig. 10, the terminal device 100 may further include a calculating unit 1006 and a converting unit 1007, wherein,

the calculating unit 1006 is configured to perform parallax matching calculation on the corrected main color image and the corrected sub-color image through a bi-shooting matching algorithm to obtain a parallax value corresponding to each pixel;

the conversion unit 1007 is configured to perform depth conversion on the disparity value through a first preset conversion model to obtain first depth information corresponding to each pixel.

In the above solution, the calculating unit 1006 is further configured to perform matching similarity calculation on the corrected main color image and the corrected sub-color image to obtain a matching similarity cost corresponding to each pixel;

the determining unit 1002 is further configured to determine a first confidence corresponding to each pixel based on the matching similarity cost.

In the above solution, the calculating unit 1006 is further configured to calculate a first texture gradient corresponding to each pixel in the corrected main color image;

the determining unit 1002 is further configured to determine a first confidence corresponding to each pixel based on the first texture gradient.

In the above scheme, the conversion unit 1007 is further configured to obtain initial depth information of each pixel in the target object in a TOF coordinate system according to the original image; and converting the coordinate system of the initial depth information through a second preset conversion model to obtain second depth information of each pixel in the main shooting coordinate system.

In the above scheme, the calculating unit 1006 is further configured to calibrate between the TOF sensor and the dual cameras according to a preset calibration algorithm, so as to obtain calibration parameters;

the conversion unit 1007 is specifically configured to convert the initial depth information into a main shooting coordinate system based on the calibration parameter and a second preset conversion model, so as to obtain second depth information of each pixel in the main shooting coordinate system.

In the above scheme, referring to fig. 10, the terminal device 100 may further include a determining unit 1008, wherein,

the determining unit 1002 is further configured to determine a low confidence region in the first depth information according to the first confidence;

the calculating unit 1006 is further configured to calculate, for each pixel to be determined in the low confidence region, a difference between first depth information corresponding to each pixel to be determined and second depth information corresponding to a valid neighborhood of the pixel to be determined;

the judging unit 1008 is configured to compare the difference value with a preset difference value threshold; and when the difference value is larger than a preset difference value threshold value, marking the pixel to be judged as an error point, and obtaining an error data area in the first depth information according to the marked error point.

In the foregoing solution, the determining unit 1008 is further configured to, when the difference is not greater than a preset difference threshold, reserve the first depth information corresponding to the pixel to be determined, and obtain a reserved data area in the first depth information.

In the above solution, the determining unit 1008 is further configured to not perform the step of comparing the difference with a preset difference threshold if there is no valid pixel point associated with the second depth information in the valid neighborhood of the pixel to be determined.

In the foregoing solution, the correcting unit 1003 is specifically configured to, for each error point in the error data area, perform weighted interpolation calculation on the error data area through second depth information corresponding to each error point, and replace the error data area in the first depth information with the calculated depth information to obtain new depth information; and filtering the new depth information according to the main color image to obtain the target depth information.

In the above solution, the determining unit 1002 is further configured to determine, according to the original image, a second confidence corresponding to the target object; wherein the second confidence level is used to characterize the accuracy of the second depth information; determining the number of holes in the second depth information based on the second confidence coefficient;

the determining unit 1008 is further configured to not perform the depth image correction method if the number of the holes is greater than a preset hole threshold.

It is understood that in this embodiment, a "unit" may be a part of a circuit, a part of a processor, a part of a program or software, etc., and may also be a module, or may also be non-modular. Moreover, each component in the embodiment may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware or a form of a software functional module.

Based on the understanding that the technical solution of the present embodiment essentially or a part contributing to the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a storage medium, and include several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the method of the present embodiment. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

Accordingly, the present embodiments provide a computer storage medium storing a depth image correction program that when executed by at least one processor implements the method of any of the preceding embodiments.

Based on the above-mentioned constituent structure of the terminal device 100 and the computer storage medium, referring to fig. 11, which shows a specific hardware structure of the terminal device 100 provided in the embodiment of the present application, the specific hardware structure may include: a communication interface 1101, a memory 1102, and a processor 1103; the various components are coupled together by a bus system 1104. It is understood that the bus system 1104 is used to enable communications among the components for connection. The bus system 1104 includes a power bus, a control bus, and a status signal bus in addition to the data bus. For clarity of illustration, however, the various buses are designated as the bus system 1104 in FIG. 11. Among them, the communication interface 1101 is used for receiving and sending signals in the process of sending and receiving information to and from other external devices;

a memory 1102 for storing a computer program operable on the processor 1103;

a processor 1103 configured to, when running the computer program, perform:

acquiring an original image corresponding to a target object and a main color image and a secondary color image corresponding to the target object; the original image is obtained according to the acquisition of a target object by a TOF sensor, and the main color image and the secondary color image are obtained according to the acquisition of the target object by two cameras;

It will be appreciated that the memory 1102 in the subject embodiment can be either volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. The non-volatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), or a flash Memory. Volatile Memory can be Random Access Memory (RAM), which acts as external cache Memory. By way of illustration and not limitation, many forms of RAM are available, such as Static random access memory (Static RAM, SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic random access memory (Synchronous DRAM, SDRAM), Double Data Rate Synchronous Dynamic random access memory (ddr Data Rate SDRAM, ddr SDRAM), Enhanced Synchronous SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 1102 of the systems and methods described herein is intended to comprise, without being limited to, these and any other suitable types of memory.

The processor 1103 may be an integrated circuit chip having signal processing capability. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in software form in the processor 1103. The Processor 1103 may be a general-purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, or discrete hardware components. The various methods, steps, and logic blocks disclosed in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory 1102, and the processor 1103 reads the information in the memory 1102 and performs the above-mentioned method in combination with the hardware thereof.

It is to be understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For a hardware implementation, the Processing units may be implemented within one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), general purpose processors, controllers, micro-controllers, microprocessors, other electronic units configured to perform the functions described herein, or a combination thereof.

For a software implementation, the techniques described herein may be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. The software codes may be stored in a memory and executed by a processor. The memory may be implemented within the processor or external to the processor.

Optionally, as another embodiment, the processor 1103 is further configured to, when running the computer program, perform the method of any of the preceding embodiments.

Alternatively, as another embodiment, the terminal device 100 may include an application processor, a main camera, a sub camera, an infrared emitter, and a laser emitter; wherein the application processor may be configured to perform the method of any of the preceding embodiments when running the computer program.

It should be noted that, in the present application, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The above-mentioned serial numbers of the embodiments of the present application are merely for description and do not represent the merits of the embodiments.

The methods disclosed in the several method embodiments provided in the present application may be combined arbitrarily without conflict to obtain new method embodiments.

Features disclosed in several of the product embodiments provided in the present application may be combined in any combination to yield new product embodiments without conflict.

The features disclosed in the several method or apparatus embodiments provided in the present application may be combined arbitrarily, without conflict, to arrive at new method embodiments or apparatus embodiments.

The above description is only for the specific embodiments of the present application, but the scope of the present application is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present application, and shall be covered by the scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for correcting a depth image, the method comprising:

determining first depth information and a first confidence coefficient corresponding to the target object by using a preset double-shot algorithm according to the main color image and the secondary color image; determining second depth information corresponding to the target object according to the original image; wherein the first confidence level is used to characterize the accuracy of the first depth information;

2. The method of claim 1, wherein after the deriving the depth image from the target depth information, the method further comprises:

3. The method of claim 1, wherein the obtaining of the original image corresponding to the target object and the primary and secondary color images corresponding to the target object comprises:

collecting the target object through a TOF sensor to obtain an original image corresponding to the target object;

acquiring the target object through two cameras to obtain a main color image of the target object under the main camera and a secondary color image of the target object under the secondary camera; wherein, two cameras include main camera and vice camera.

4. The method of claim 1, wherein determining the first depth information and the first confidence level corresponding to the target object by using a preset bi-shooting algorithm according to the main color image and the secondary color image comprises:

carrying out distortion correction processing on the main color image to obtain a corrected main color image;

carrying out distortion correction and epipolar line correction processing on the secondary color image to obtain a corrected secondary color image;

and for each pixel in the target object, based on the corrected main color image and the corrected secondary color image, determining first depth information corresponding to each pixel and a first confidence corresponding to each pixel by using a preset bi-shooting algorithm.

5. The method of claim 4, wherein determining the first depth information for each pixel based on the corrected primary color image and the corrected secondary color image comprises:

6. The method of claim 4, wherein determining a first confidence level for each pixel based on the corrected primary color image and the corrected secondary color image comprises:

7. The method of claim 4, wherein determining a first confidence level for each pixel based on the corrected primary color image and the corrected secondary color image comprises:

8. The method of claim 1, wherein determining second depth information corresponding to the target object according to the original image comprises:

obtaining initial depth information of each pixel in the target object under a TOF coordinate system according to the original image;

and converting the coordinate system of the initial depth information through a second preset conversion model to obtain second depth information of each pixel in the main shooting coordinate system.

9. The method according to claim 8, wherein before the coordinate system converting the initial depth information by the second predetermined conversion model to obtain the second depth information of each pixel in the main camera coordinate system, the method further comprises:

correspondingly, the performing coordinate system conversion on the initial depth information through a second preset conversion model to obtain second depth information of each pixel in the main shooting coordinate system includes:

10. The method of claim 1, wherein determining the region of erroneous data in the first depth information based on the first depth information, the second depth information, and the first confidence level comprises:

determining a low confidence region in the first depth information according to the first confidence;

calculating a difference value between first depth information corresponding to each pixel to be judged and second depth information corresponding to the pixel to be judged in an effective neighborhood of the pixel to be judged aiming at each pixel to be judged in the low confidence coefficient area;

comparing the difference value with a preset difference value threshold value;

and when the difference is larger than a preset difference threshold value, marking the pixel to be judged as an error point, and obtaining an error data area in the first depth information according to the marked error point.

11. The method of claim 10, wherein after said comparing said difference to a preset difference threshold, said method further comprises:

and when the difference is not greater than a preset difference threshold, reserving the first depth information corresponding to the pixel to be judged to obtain a reserved data area in the first depth information.

12. The method of claim 10, further comprising:

and if the effective pixel point associated with the second depth information does not exist in the effective neighborhood of the pixel to be judged, the step of comparing the difference value with a preset difference value threshold value is not executed.

13. The method of claim 1, wherein the performing the correction processing on the error data region through the second depth information and the main color image to obtain target depth information comprises:

for each error point in the error data area, performing weighted interpolation calculation on the error data area through second depth information corresponding to each error point, and replacing the error data area in the first depth information with the calculated depth information to obtain new depth information;

and filtering the new depth information according to the main color image to obtain the target depth information.

14. The method according to any one of claims 1 to 13, wherein after the determining second depth information corresponding to the target object from the original image, the method further comprises:

15. A terminal device, characterized in that the terminal device comprises: an acquisition unit, a determination unit, and a correction unit, wherein,

the determining unit is configured to determine first depth information and a first confidence coefficient corresponding to the target object by using a preset bi-shooting algorithm according to the main color image and the secondary color image; determining second depth information corresponding to the target object according to the original image; and further configured to determine an erroneous-data region in the first depth information based on the first depth information, the second depth information, and the first confidence level; wherein the first confidence level is used to characterize the accuracy of the first depth information;

16. A terminal device, characterized in that the terminal device comprises: a memory and a processor; wherein the content of the first and second substances,

the memory for storing a computer program operable on the processor;

the processor, when running the computer program, is configured to perform the method of any of claims 1 to 14.

17. A computer storage medium storing a depth image correction program that when executed by at least one processor implements the method of any one of claims 1 to 14.