KR20240078316A - Apparatus and method for estimating dept - Google Patents

Abstract

The present disclosure relates to an apparatus and method for estimating depth, comprising receiving a Time Of Flight (TOF) image, a left image, and a right image of a scene, determining a first reliability of each TOF pixel of the TOF image. It may include confirming and estimating a depth map of the scene based on the TIOF image, the left image, and the right image according to the first reliability.

Description

Apparatus and method for estimating depth {APPARATUS AND METHOD FOR ESTIMATING DEPT}

The following embodiments relate to the field of image processing, and particularly to depth estimation methods, devices, electronic devices, and storage media.

Efficient depth estimation plays an important role in augmented reality. By analyzing the input sensor image, depth information of the surrounding scene is inferred to support various follow-up tasks.

For example, various 3D virtual objects can be drawn based on the 3D information of scene objects. At the same time, obtaining 3D information of the surrounding scene also has important implications for many other tasks, such as robot obstacle avoidance and grasping. Therefore, a more accurate and efficient depth estimation method is needed.

The purpose of the present invention is to provide an apparatus and method for estimating depth.

A depth estimation method according to an embodiment of the present invention, comprising: receiving a Time Of Flight (TOF) image, a left image, and a right image of a scene; Confirming a first reliability of each TIOF pixel of the TIOF image; And it may include estimating a depth map of the scene based on the TIOF image, the left image, and the right image according to the first reliability.

At this time, the step of estimating the depth map of the scene based on the TFO image, the left image, and the right image according to the first reliability includes selecting the TFO pixel of the first reliability that satisfies a predetermined requirement. Confirming the quantity; If the quantity of TFO pixels of the first reliability meets a first threshold requirement, estimating a depth map of the scene based on the TFO image, the left image, and the right image; and if the quantity of TIOF pixels of the first reliability does not meet the first threshold requirement, estimate the depth map of the scene based on the left image and the right image without using the TFO image. It may include steps.

At this time, the step of checking the first reliability of each ThiofF pixel of the Thiofe image includes projecting each Thiofe pixel of the Thiofe image onto the left image and the right image, respectively; For each Thiof projection point after projection, calculating a second reliability of a Thiof pixel corresponding to each Thiof projection point on the corresponding Thiof projection point scan line along a first direction; And calculating the first reliability of the TIOF pixel corresponding to each TIOF projection point along a second direction opposite to the first direction based on a reliability calculation result in the first direction. .

At this time, the step of calculating the second reliability and the step of calculating the first reliability include the difference in image characteristics of each TIOF projection point in the left image and the right image and the difference in image characteristics of each TIOF projection point on the corresponding projection point scan line. Based on the third reliability of the TIOF pixel corresponding to the TIOF projection point with similar image characteristics where the distance between the projection points is within a preset range, the second reliability of the TFO pixel corresponding to each TFO projection point and the first reliability can be calculated.

At this time, estimating the depth map of the scene based on the TIOF image, the left image, and the right image includes performing first stereo matching of the left image and the right image; predicting a fourth reliability of each TIOF pixel based on the TFO image and the first stereo matching result using a first neural network; and performing a second stereo matching of the left image and the right image based on the TIOF image and the fourth reliability to estimate a first depth map of the scene.

At this time, the step of performing the first stereo matching of the left image and the right image includes, when the quantity satisfies the first threshold requirement and the second threshold requirement, the first reliability and TIOF image performing the first stereo matching of the left image and the right image based on; or if the quantity meets the first threshold requirement and does not meet the second threshold requirement, perform the first stereo matching of the left image and the right image in a situation where the TIOF image is not used. It may include steps.

At this time, the step of predicting the fourth reliability of each TFO pixel based on the TFO image and the first stereo matching result using the first neural network includes first information, second information, and third information. and predicting the fourth reliability of each TIOF pixel by using at least one piece of information as an input to the first neural network, wherein the first information is applied to each TFO pixel determined according to the TFO image. It is the difference between the corresponding disparity value and the disparity value of each TIOF pixel determined through the first stereo matching, and the second information is the image characteristic of each TOF pixel at the projection point of the left image and the right image. It is the difference, and the third information may be the difference between the projection point of each TIOF pixel of the left image and the right image and the depth value of at least one TIOF projection point with similar image characteristics in the projection point area. .

At this time, the step of estimating the first depth map of the scene by performing the second stereo matching of the left image and the right image based on the TIOF image and the fourth reliability includes: Calculating a matching cost of a candidate disparity corresponding to each of the ThioF pixels during the second stereo matching based on the value of each Thiofe pixel and the predicted fourth reliability of each Thiofe pixel; and determining a disparity value corresponding to each TIOF pixel based on the matching cost, and estimating the first depth map using the determined disparity value.

At this time, the step of estimating the depth map of the scene based on the TIOF image, the left image, and the right image includes projecting the TIOF image onto the left image and the right image, and satisfying a preset density. For the TIOF projection point area, performing interpolation based on image features of the left image and the right image to estimate a second depth map; estimating a third depth map by performing interpolation on the first depth map based on image features of the left image and the right image; And it may further include estimating a fourth depth map of the scene based on the second depth map and the third depth map.

At this time, the step of estimating the second depth map by performing interpolation based on image features of the left image and the right image for the TIOF projection point area that satisfies the preset density includes the steps of estimating the second depth map, performing interpolation based on image features of the left image and the right image for adjacent TIOF projection points spaced within a preset distance on the scan line of the point; Constructing a regular grid of TIOF projection points by sampling the interpolated TIOF projection points; and estimating the second depth map by performing interpolation based on image features of the left image and the right image for each TIOF projection point on each configured grid.

At this time, when interpolation is performed based on the image features of the left image and the right image, the depth value of the point to be interpolated can be determined based on the difference in image features and the spatial distance between the point to be interpolated and the adjacent reference point.

At this time, the step of estimating the depth map of the scene based on the left image and the right image without using the T.O.F image includes creating a fifth depth map of the scene through stereo matching of the left image and the right image. It may include a step of estimating.

At this time, the depth estimation method further includes updating the depth value of an unreliable depth value point among the estimated depth maps, wherein the estimated depth map is the first depth map, the fourth depth map, or the fourth depth map. 5 May include a depth map.

At this time, updating the depth value of the unreliable depth value point in the estimated depth map includes determining a reliable depth value point and an unreliable depth value point in the estimated depth map; predicting a depth value of the unreliable depth value point based on characteristics of the reliable depth value point and the unreliable depth value point using a second neural network; and estimating an updated depth map by performing interpolation based on image features of the left image and the right image for the area surrounding the unreliable depth value point.

At this time, the step of determining the reliable depth value points and the unreliable depth value points among the estimated depth map includes configuring a regular grid of depth value points based on the estimated depth map, and forming the regular grid of depth value points based on the estimated depth map. and determining the reliable depth value point and the unreliable depth value point in the image.

An electronic device according to an embodiment of the present invention, comprising: at least one processor; and a memory storing at least one computer-executable instruction, wherein the processor receives a Time Of Flight (TOF) image, a left image, and a right image of the scene, each TOF of the TOF image. The first reliability of the pixel may be confirmed, and the depth map of the scene may be estimated based on the TFO image, the left image, and the right image according to the first reliability.

At this time, when estimating the depth map of the scene, the processor confirms the quantity of TIOF pixels of the first reliability that satisfies a predetermined requirement, and the quantity of TFO pixels of the first reliability is determined by the first reliability. If the threshold requirement is met, a depth map of the scene is estimated based on the TIOF image, the left image, and the right image, and the quantity of TFO pixels of the first reliability is determined by the first threshold requirement. If the requirements are not met, the depth map of the scene can be estimated based on the left image and the right image in a situation where the TFO image is not used.

At this time, when determining the first reliability of each Thiof pixel of the Thiof image, the processor projects each Thiof pixel of the Thiof image onto the left image and the right image, respectively, and projects each Thiof pixel of the Thiof image after projection. For an F projection point, calculate a second reliability of a Thiof pixel corresponding to each Thiof projection point on the corresponding Thiof projection point scan line along a first direction, and along a second direction opposite to the first direction. , The first reliability of the TIOF pixel corresponding to each TIOF projection point may be calculated based on the reliability calculation result in the first direction.

At this time, when estimating the depth map of the scene based on the TIOF image, the left image, and the right image, the processor performs first stereo matching of the left image and the right image, and uses a first neural network Predict the fourth reliability of each TIOF pixel based on the TIOF image and the first stereo matching result, and predict the fourth reliability of the left image and the right image based on the TFO image and the fourth reliability. The first depth map of the scene can be estimated by performing a second stereo matching.

At this time, when the processor estimates the depth map of the scene based on the left image and the right image without using the TFO image, the processor determines the depth map of the scene through stereo matching of the left image and the right image. 5 The depth map can be estimated.

1 is a flowchart illustrating a process for estimating depth according to one embodiment.
Figure 2 is an exemplary diagram illustrating the configuration of a TIOF image sensor and a color image sensor on a head wearable device according to an embodiment.
FIG. 3 is an exemplary diagram illustrating an example of TFO pixel reliability inference based on two-way propagation according to an embodiment.
Figure 4 is a diagram illustrating stereo matching of a TIOF guide according to an embodiment.
Figure 5 is an exemplary diagram illustrating an example of reliability of transformer-based ToF pixel prediction according to an embodiment.
Figure 6 is an example diagram illustrating a projection situation of ToF points on an RGB image according to an embodiment.
Figure 7 is an exemplary diagram illustrating an example of grid points after downsampling according to an embodiment.
FIG. 8 is a diagram illustrating an example of performing interpolation guided by image features of left and right images on a regular grid, according to an embodiment.
9 shows an example of interpolation guided by image features of left and right images after random projection, according to one embodiment.
FIG. 10 is a diagram illustrating a process of updating a transformer-based unreliable depth value point according to an embodiment.
Figure 11 is a diagram illustrating an example of a process for estimating depth according to an embodiment.
Figure 12 is a block diagram showing a depth estimation device according to an embodiment.
Figure 13 is a block diagram of an electronic device according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, various changes can be made to the embodiments, so the scope of the patent application is not limited or limited by these embodiments. It should be understood that all changes, equivalents, or substitutes for the embodiments are included in the scope of rights.

The terms used in the examples are for descriptive purposes only and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the embodiments belong. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless explicitly defined in the present application, should not be interpreted in an ideal or excessively formal sense. No.

In addition, when describing with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted. In describing the embodiments, if it is determined that detailed descriptions of related known technologies may unnecessarily obscure the gist of the embodiments, the detailed descriptions are omitted.

Additionally, in describing the components of the embodiment, terms such as first, second, A, B, (a), and (b) may be used. These terms are only used to distinguish the component from other components, and the nature, sequence, or order of the component is not limited by the term. When a component is described as being “connected,” “coupled,” or “connected” to another component, that component may be directly connected or connected to that other component, but there is no need for another component between each component. It should be understood that may be “connected,” “combined,” or “connected.”

Components included in one embodiment and components including common functions will be described using the same names in other embodiments. Unless stated to the contrary, the description given in one embodiment may be applied to other embodiments, and detailed description will be omitted to the extent of overlap.

Below, according to an embodiment of the present invention An apparatus and method for estimating depth will be described in detail with reference to the attached FIGS. 1 to 13.

1 is a flowchart illustrating a process for estimating depth according to one embodiment.

Referring to FIG. 1, the depth estimation method may receive the TOF (Time Of Flight), left image, and right image of the scene in step S110, as shown in FIG. 1 (110). For example, ToF images can be collected using a ToF image sensor, left and right images can be collected using a color image sensor, or pre-stored ToF images and left and right images can be acquired directly from memory. The present disclosure does not limit the method of acquiring ToF images, left and right images. The left and right images can be RGB images, other types of color images, or infrared images, etc. For example, the ToF image sensor and color image sensor of the head wearable device may be configured as shown in FIG. 2.

Figure 2 is an exemplary diagram illustrating the configuration of a TIOF image sensor and a color image sensor on a head wearable device according to an embodiment.

Referring to FIG. 2, the head wearable device 210 that estimates depth may include a ToF image sensor 212 and color image sensors 214 and 216 located on the same line.

For example, the captured image may include a right image 224 and a left image 226 corresponding to a ToF image 222 and a stereo image pair, as shown in FIG. 2 .

Returning to the description of FIG. 1, the depth estimation method can confirm the first reliability of each ToF pixel in the ToF image (120). At this time, the first reliability may be a value measuring the accuracy or reliability of the depth value corresponding to the ToF pixel. In addition, the second reliability, third reliability, and fourth reliability mentioned in the present disclosure below are also values that measure the accuracy or reliability of the depth value corresponding to the TOF pixel.

In step 120, in the depth estimation method, each ToF pixel of the ToF image may be first projected onto the left and right images, respectively. In addition, the depth estimation method may calculate a second reliability of the ToF pixel corresponding to each ToF projection point on the ToF projection point scan line along the first direction for each ToF projection point scan line after projection. Additionally, the depth estimation method may calculate the first reliability of the ToF pixel corresponding to each ToF projection point along a second direction opposite to the first direction based on the reliability calculation result in the first direction.

In this disclosure, the method for determining the above-described first reliability is referred to as “two-way propagation-based ToF reliability inference.” In bidirectional propagation-based ToF reliability inference, when calculating the second reliability along the first direction and the first reliability along the second direction, the image feature difference of each ToF projection point in the left and right images, the difference in image characteristics on the scan line of the corresponding projection point, Based on the image feature of the distance between each ToF projection point within a predetermined range and the third reliability of the ToF pixel corresponding to the ToF projection point, the second reliability and the first reliability of the ToF pixel corresponding to each ToF projection point are calculated. You can. For example, the first direction may be left to right along the scan line of the projection point, and the second direction may be right to left along the scan line of the projection point. Alternatively, the first direction may be from top to bottom along the scan line of the projection point, and the second direction may be from bottom to top along the scan line of the projection point.

FIG. 3 is an exemplary diagram illustrating an example of TFO pixel reliability inference based on two-way propagation according to an embodiment.

Referring to FIG. 3, when the ToF image sensor and the color image sensor are located on the same line, the ToF pixels projected on the left and right images are also located on the same line. Taking the ToF pixel projected on the left image as an example, for each ToF projection point scan line, there are two ToF pixels corresponding to each ToF projection point along two directions (e.g., first left to right, then right to left). Calculate reliability.

In certain implementations, confidence calculations are performed by projecting each ToF pixel of the ToF image. Taking the left to right direction as an example, given the ToF projection point P, the difference in image features between the corresponding points of the left and right images can be defined as e0, and the already existing features within a predetermined distance in the horizontal direction (e.g. , color features), the similar points are M1, ??, Mn. A threshold for determining whether image features are similar for different image features may be set in advance. For these n points, the horizontal distance to point P can be defined as di, i=1, ??, n, and the depth difference between them and point P is bi, i=1, ??, n, The calculated reliability is ci, i=1, ??, n. Therefore, the reliability of point P can be defined as follows <Equation 1>.

[Equation 1]

At this time, Cp is the reliability of P, which is a ToF pixel of the ToF image, and a0, a1, and a2 may be preset constants. a0, a1, and a2 are all greater than 0. For example, a0, a1, and a2 may be experimental values greater than 0.

The reliability of the ToF projection point to the right of point P from left to right (the reliability of the ToF projection point is the reliability of the ToF pixel corresponding to the ToF projection point) is temporarily unknown. For example, you can temporarily set it to the default value (e.g. 1). In other words, for reliability points for which reliability has not yet been calculated, the reliability can be temporarily set to the default value. After calculating the second reliability of the ToF pixel corresponding to each ToF projection point from left to right, the first reliability of the ToF pixel corresponding to each ToF projection point is calculated from right to left. For example, as shown in Figure 3, when calculating the second reliability of point P from left to right, the second reliability of points M1 and M2 have already been calculated, but the second reliability of point M3 located to the right of point P has not been calculated yet, you can temporarily set it to the default value (e.g. 1). When calculating the first reliability of point P from right to left, the first reliability of point M3 has already been calculated. At this time, the first reliability of point P can be calculated using the first reliability of point M3 and the second reliability of points M1 and M2.

양방향 전파를 기반으로 한 ToF 픽셀의 신뢰성 추론은 신경망과 같은 방법에 비해 빠른 이점이 있어 깊이 추정의 효율성을 향상시키는데 도움이 된다. 동시에 추정된 제1 신뢰도는 왼쪽, 오른쪽 및 ToF 이미지를 기반으로 장면의 깊이 정보를 추정할지 여부를 결정하는데 사용될 수 있다.

Reliability inference of ToF pixels based on bidirectional propagation has the advantage of being faster than methods such as neural networks, which helps improve the efficiency of depth estimation. At the same time, the estimated first confidence can be used to decide whether to estimate the depth information of the scene based on the left, right, and ToF images.

Going back to the description of FIG. 1, the depth estimation method may estimate the depth map of the scene based on the left image, right image, and ToF image according to the first reliability (130).

According to an embodiment, in step 130, the depth estimation method determines the quantity of ToF pixels whose first reliability satisfies the preset requirement, and the quantity of ToF pixels whose first reliability satisfies the preset requirement is the first reliability. If the threshold requirements are met, the depth map of the scene can be estimated based on the left image, right image, and ToF image.

Conversely, in step 130, if the quantity of ToF pixels whose first reliability satisfies the preset requirement does not satisfy the first threshold requirement, the depth estimation method determines the depth of the scene based on the left and right images without using the ToF image. The map can be estimated.

For example, the depth estimation method may determine whether to determine the depth map of the scene based on the left and right and ToF images based on the first reliability.

The depth estimation method estimates the depth map of the scene based on the left and right and ToF images only in situations where the quantity of ToF pixels whose first reliability meets the preset requirements satisfies the first threshold. Otherwise, the depth map of the scene is estimated based on the ToF image. In situations where images are not used, the depth map of the scene can be estimated based on the left and right images. As a result, the depth estimation method can avoid the problem of high computational overhead due to the limited information provided by ToF images, thereby improving depth estimation efficiency.

Hereinafter, estimating the depth map of a scene based on left, right, and ToF images will be described in more detail.

Estimating the depth map of a scene based on left, right, and ToF images may include stereo matching based on the left and right images as well as the ToF image to estimate the depth map of the scene.

Research shows that both ToF cameras and stereo matching have unique advantages and disadvantages. ToF cameras have accurate range and can handle untextured areas very well. However, ToF images obtained through a ToF camera can be noisy and produce no results when the light is too strong, too dark, or at long distances. In the case of stereo matching, processing of areas without texture is poor, but it is not affected by lighting and distance. Correspondingly, this disclosure proposes a method to better fuse this stereo matching with ToF images to estimate depth information.

In one embodiment, estimating a depth map of a scene based on the left, right, and ToF images includes first, performing first stereo matching of the left and right images, and second, performing the ToF image and the first stereo matching result. predicting a fourth reliability of each ToF pixel using a first neural network based on , and finally performing a second stereo matching of the left and right images based on the ToF image and the predicted fourth reliability to obtain a first It may include the step of obtaining a depth map. For example, the first neural network may be, but is not limited to, a Transformer.

As mentioned earlier, the depth estimation method can estimate the depth map of the scene based on the left and right images and the ToF image when the quantity of ToF pixels whose first reliability satisfies a predetermined requirement satisfies the first threshold. . For example, the depth estimation method estimates the depth map of the scene based on stereo matching of the left and right images as well as the ToF image when the quantity of ToF pixels whose first reliability satisfies a predetermined requirement satisfies the first threshold. can do. Specifically, the present disclosure also provides different fusion methods depending on whether the quantity of ToF pixels whose first reliability satisfies a predetermined requirement satisfies the first threshold requirement and more satisfies the second threshold requirement. can be adopted. For example, the selection of the first and second threshold requirements may be related to the quantity of ToF pixels in the ToF image and/or the accuracy of the ToF camera used to acquire the ToF image. It can be. For example, the more ToF pixels in the ToF image and/or the higher the accuracy of the ToF camera, the higher the first threshold and the second threshold may be.

For example, in different fusion methods, before using the first neural network to predict the fourth reliability of each ToF pixel, a different method may be adopted to perform the first stereo matching of the left and right images. For example, if the quantity of ToF pixels whose confidence satisfies the predetermined requirement satisfies the first threshold requirement and the second threshold requirement, then the left image and right image are selected based on the first confidence and ToF image. The first stereo matching can be performed. or 1 If the quantity of ToF pixels whose confidence satisfies a predetermined requirement satisfies the first threshold requirement and does not meet the second threshold requirement, then in the situation where ToF images are not used, the First stereo matching may be performed. In situations where ToF images are not used, the first stereo matching of left and right images may be performed using any method known to those skilled in the art.

Here, performing the first stereo matching of the left and right images based on the first reliability and ToF images will be described. For example, a method of performing first stereo matching of left and right images based on the first reliability and ToF images includes: first, first stereo matching based on the value of each ToF pixel in the ToF image and the first reliability of each ToF pixel; It may include calculating a matching cost of a candidate disparity corresponding to each ToF pixel when performing matching, and secondly, determining a disparity value corresponding to each ToF pixel based on the calculated matching cost.

For example, when performing first stereo matching of left and right images based on the first reliability and ToF images, SGM (Semi Global Matching) can be used as a reference method for stereo matching. SGM is fast, has excellent performance, and is not affected by domain conversion. However, ToF pixels often have noise. Therefore, in this disclosure, when performing stereo matching using SGM, we propose a method of calculating the matching cost of the candidate disparity corresponding to each ToF pixel during SGM stereo matching based on the value of each ToF pixel of the ToF image and the reliability of each ToF pixel. do. For example, the weight used to calculate the matching cost can be defined based on the first reliability. For convenience of explanation, the stereo matching method considering ToF reliability of the present disclosure is referred to as “stereo matching of ToF guide” (see FIG. 4).

Figure 4 is a diagram illustrating stereo matching of a TIOF guide according to an embodiment.

Referring to FIG. 4, first, image features are extracted (422, 432) for the left image 420 and the right image 430, respectively. For example, image features for the left image 420 and right image 430 can be extracted using CNN. The depth estimation method can then run the SGM stereo matching algorithm separately for each scan line. However, when executing the SGM stereo matching algorithm in the present disclosure, the matching cost of the candidate disparity corresponding to each ToF pixel is a weight of matching cost 1 determined based on image block matching of the left and right images and matching cost 2 determined based on each ToF pixel. It can be decided based on the sum. At this time, the weight of matching cost 2 used when performing the weighted sum can be defined based on the first reliability. The higher the first reliability, the greater the weight of matching cost 2. Here, the closer the candidate disparity is to the disparity corresponding to the ToF pixel, the closer the matching cost 2 is to 0. This method can make the disparity value obtained during stereo matching more accurate.

The depth estimation method may perform first stereo matching of the left and right images based on the first reliability and ToF images (442). Additionally, the depth estimation method may predict the fourth reliability of each ToF pixel using a first neural network (eg, Transformer) based on the ToF image and the first stereo matching result (444). The fourth confidence level of each ToF pixel can be checked for each scan after projecting the ToF image onto the left and right images.

Figure 5 is an exemplary diagram illustrating an example of reliability of transformer-based ToF pixel prediction according to an embodiment.

Referring to FIG. 5, the depth estimation method uses at least one type of information below (first information 512, second information 514, Third information 516) can be used.

The first information 512 may be the difference between the disparity value corresponding to each ToF pixel determined in the ToF image and the disparity value of each ToF pixel determined through first stereo matching. The second information 514 may be the difference in image characteristics of each ToF pixel at the projection point of the left image and the right image. The third information 516 may be a depth value difference between the projection point of each ToF pixel on the left image and the right image and at least one ToF projection point similar to an image feature in the area at the projection point.

Here, the disparity value corresponding to each ToF pixel determined based on the ToF image is obtained by converting the depth value of each ToF pixel in the ToF image. The image feature difference (second information 514) at the projection point of each ToF pixel in the left image and the right image is obtained by projecting the ToF image onto the left and right images to obtain the corresponding projection point, and then calculating the image feature at the projection point. can be obtained by comparison. The depth value difference (third information 516) between the projection point of each ToF pixel on the left image and the right image and at least one ToF projection point similar to the image feature in the adjacent area of the projection point is used to store the ToF image on the left and right images. After obtaining the corresponding projection point by projection, it can be determined by searching for ToF projection points similar to image features within a predetermined distance range of the projection point on the projection point scan line and comparing the depth values between them. The first information 512, the second information 514, and the third information 516 may be connected (520, 530) and sent to the transformer encoder 540 to be encoded. The encoded result is input to the MLP (Multi-layer Perception Machine) 550 for decoding to obtain the reliability of each ToF point. The transformer encoder 540 of Figure 5 can optionally use Swin-Transformer (Liu Z, Lin Y, Cao Y, et al. Swin Transformer: Hierarchical vision transformer using shifted windows. ICCV, 2021), which is a general transformer (Transformer ), the processing speed is faster than that of

Returning to the description of FIG. 4, the depth estimation method predicts the fourth reliability of the ToF pixel using a first neural network (444), and then calculates the fourth reliability of the left image and the right image based on the ToF image and the predicted fourth reliability. 2 Stereo matching can be performed to estimate the first depth map of the scene (446).

Specifically, the depth estimation method may calculate the matching cost of the candidate disparity corresponding to each ToF pixel during the second stereo matching based on the value of each ToF pixel of the ToF image and the fourth reliability of each predicted ToF pixel. The depth estimation method may determine a disparity value corresponding to each ToF pixel based on a matching cost, and estimate a first depth map based on the determined disparity value. The method of "performing the second stereo matching of the left and right images based on the value of each ToF pixel in the ToF image and the fourth reliability of each predicted ToF pixel" is the previously described method of "performing the value of each ToF pixel in the ToF image and the fourth reliability of each ToF pixel." It is the same as the method of “performing the first stereo matching of the left and right images based on the first reliability of What was used previously is the first reliability of each ToF pixel. Therefore, further description is omitted here and related information can be found in the above description.

In the above-mentioned ToF guide stereo matching, the matching cost during stereo matching is calculated based on the ToF pixel value and the reliability of the ToF pixel, thereby improving the accuracy of depth estimation and obtaining a more accurate depth map.

Optionally, to obtain a denser and more accurate depth map, the depth estimation method projects the ToF image to the left image and the right image, and for the ToF projection point area that meets the preset density, the image of the left image and the right image estimating a second depth map by performing interpolation based on features, estimating a third depth map by performing interpolation based on image features of the left image and right image with respect to the first depth map, and estimating a second depth The method may further include estimating a fourth depth map of the scene based on the map and the third depth map. "Perform interpolation guided by the image features of the left and right images" means performing interpolation based on the image features of the left and right images, that is, taking into account the image features of the left and existing images when performing the interpolation. means performing.

Interpolation guided by image features of the left image and right image (also referred to as “interpolation of RGB guides” in this disclosure) refers to using image features of the left image and right image for interpolation. According to an embodiment, in interpolation guided by image features of the left and right images, the depth value of the point to be interpolated can be determined based on the difference in image features and the spatial distance between the point to be interpolated and adjacent reference points. For example, but not limited to, the image feature difference may be a color feature difference (also referred to as “color distance”). Interpolation of the RGB guide considers the image characteristics of the left and right images, so the accuracy of interpolation can be improved.

In an embodiment of the present disclosure, the depth estimation method determines the point to be interpolated and adjacent reference points based on the object to be interpolated. Depending on the different objects of the point to be interpolated, the reference points adjacent to the point to be interpolated represent different points. For example, if the object to be interpolated is a ToF projection point area, a reference point adjacent to the point to be interpolated may be a ToF projection point. For example, if the object to be interpolated is a depth map, a reference point adjacent to the point to be interpolated may be a depth value point. For example, the point to be interpolated may be a point in an area where ToF projection points are dense, excluding existing ToF projection points. Adjacent reference points are ToF projection points on the four corners of the rectangular grid where the points to be interpolated are located.

Due to the high computational amount involved in calculating the spatial distance (e.g., Euclidean distance), to increase the computational amount, in this disclosure, the ToF image is first projected onto the left and right images, and the ToF projection point area that meets the predetermined density is Interpolation of RGB guides on a regular grid can be performed.

Figure 6 is an example diagram illustrating a projection situation of ToF points on an RGB image according to an embodiment.

Referring to Figure 6, when projecting the ToF pixels of a ToF image onto the left and right images, the initial result is irregular grid points. That is, ToF points on the same line may still be on the same projection scan line after projection, but the distance between adjacent ToF points may no longer be regular. Therefore, in the present disclosure, interpolation guided by the image features of the left and right images is first performed for adjacent ToF projection points spaced within a certain distance on the scan line of each ToF projection point. For example, interpolate point F between points C and D. Areas with large post-projection gaps are typically very discontinuous and may have significant errors that are not visible to the ToF camera. Therefore, interpolation can only be performed in the ToF projection point area that meets the predetermined density. If the distance between two adjacent ToF projection points on the scan line is greater than a predetermined interval, interpolation is not performed. For example, the predetermined spacing may be related to the spacing between two ToF pixels in a ToF image. For example, the predetermined spacing may be a predetermined multiple (e.g., three times) of the spacing between two ToF pixels in the ToF image.

Second, the depth estimation method samples the interpolated ToF projection points to construct a regular ToF projection point grid. For example, a schematic diagram of grid points after downsampling is shown in Figure 7. Finally, interpolation guided by the image features of the left and right images is performed on each constructed grid.

Figure 7 is an exemplary diagram illustrating an example of grid points after downsampling according to an embodiment.

Referring to FIG. 7, the depth estimation method can, given four points A, B, C, and D on a grid point, infer the depth value of point F to be interpolated within a rectangle surrounded by them. For example, given the width of a small grid, the length of AB is N, and the calculation method for the depth value d _F of point F may be, for example, as shown in Equation 2 below.

[Equation 2]

here, are the spatial distances in the x- and y-axis directions between point A and point F, respectively, , are the color values at point A and point F, respectively, and a is a constant greater than 0.

In one embodiment, to speed up processing, the calculation Coefficients related to spatial distance and productive distance according to the formula of You can calculate in advance and store it in a table, and determine its value through table lookup.

Above we have provided an example of how to calculate the depth value for point F, but the method for calculating the depth value for point F is not limited to the above example, but rather the spatial distance between point F and adjacent reference points A, B, C, D and the image It can be any method that determines the depth value of point F based on feature differences.

FIG. 8 is a diagram illustrating an example of performing interpolation guided by image features of left and right images on a regular grid, according to an embodiment.

Referring to FIG. 8, (a) of FIG. 8 is an RGB image, and (b) of FIG. 8 shows interpolation results at different pixel intervals (grid widths). In order from the left, they are the sparse depth map (i.e., the depth map before interpolation), the predicted depth map (i.e., the depth map after interpolation), the actual depth map, and the difference between the actual depth map and the predicted depth map. In Figure 8, it can be seen that the narrower the grid width, the smaller the difference between the predicted depth map and the actual depth map.

9 shows an example of interpolation guided by image features of left and right images after irregular projection, according to one embodiment.

Referring to FIG. 9, (a) of FIG. 9 shows a left RGB image and a ToF image. Here, projecting the ToF image onto the left RGB image is taken as an example. Figure 9(b) shows the interpolation effect on a sparse scan line, the effect of interpolation of the RGB guide on a regular grid to fill the dense ToF projection point area, and the predicted depth map and actual depth map sequentially. It is shown as Here, the predicted depth map is the previously mentioned second depth map.

In one embodiment, the above-mentioned operation "projects the ToF image to the left image and the right image, and performs interpolation based on the image features of the left image and the right image for the ToF projection point area that meets a preset density, The task of “estimating a second depth map” can be performed before or after the previously mentioned task of “performing a first stereo matching of the left and right images without using the ToF image”, or the task of “performing a first confidence and ToF It can be executed before or after “performing the first stereo matching of the left image and the right image based on the image.” The present disclosure is not limited thereto. Additionally, when estimating the second depth map, the depth estimation method may estimate the third depth map by performing interpolation guided by image features of the left and right images on the first depth map. Finally, the fourth depth map of the scene can be further estimated based on the second depth map and the third depth map. A more accurate depth map can be obtained through additional fusion of the depth map.

For example, the depth estimation method may obtain a fourth depth map by performing a weighted average of the depth values of the area surrounding a reliable ToF pixel among the second depth map and the third depth map. At this time, the weight during the weighted average varies depending on the fourth reliability of the ToF pixel. If there is a corresponding ToF pixel for which the fourth reliability has been calculated prior to the weighted depth value, the weight of the corresponding depth value may be determined based on the fourth reliability of the ToF pixel. If there is no corresponding ToF pixel for which the fourth reliability was calculated prior to the weighted depth value, the depth estimation method may determine the weight of the depth value based on the fourth reliability of the nearest ToF pixel.

The depth estimation method that combines ToF image and stereo matching described above can estimate an accurate depth map by effectively integrating the advantages of both ToF image and stereo matching.

However, in some cases, the hidden area of the scene cannot be seen by both the ToF camera and the RGB camera, so the depth information of the hidden area cannot be accurately estimated. The problem of inaccurate depth information estimation due to such occlusion can occur not only in the method of estimating the depth map based on the left image, right image, and ToF image, but also in the method of estimating depth information based on the left and right images without using the ToF image. there is. For example, a method of estimating depth information of a scene based on the left and right images in a situation where ToF images are not used is to create a fifth depth map of the scene through stereo matching of the left and right images in a situation where ToF images are not used. It may include an acquisition step. However, the depth value of the occluded area in the estimated depth map may not be accurate enough. In response to this, the present disclosure additionally proposes updating the depth value corresponding to this occluded area to obtain a more accurate depth map. Therefore, according to an embodiment, the depth estimation method may further include updating the depth value of an unreliable depth point among the estimated depth maps, where the estimated depth map is a first depth map, a fourth Includes a depth map or a fifth depth map.

Specifically, first, the depth estimation method may first determine reliable depth value points and unreliable depth value points of the estimated depth map in the update operation. Second, the depth estimation method can predict the depth value of the unreliable depth value point using a second neural network based on the characteristics of the reliable depth value point and the unreliable depth value point. Finally, the depth estimation method can estimate an updated depth map by performing interpolation guided by image features of the left and right images for the area around the unreliable depth value point. The depth map update method can effectively utilize global semantic information for depth inference, so a more accurate depth map can be estimated. According to an embodiment, the second neural network may be a transformer, but is not limited thereto.

FIG. 10 is a diagram illustrating a process of updating a transformer-based unreliable depth value point according to an embodiment.

Referring to FIG. 10, in order to improve the update speed, the present disclosure constructs a grid of regular depth value points based on the estimated depth map, and determines reliable depth value points and unreliable depth value points on the regular grid. there is. According to an embodiment, when the estimated depth map is the first depth map mentioned above, the depth estimation method may first perform interpolation of the RGB guide for the first depth map. Then, the depth estimation method may construct a regular grid by sampling depth value points for the first depth map. And, the depth estimation method can determine reliable and unreliable depth value points on the grid.

The depth estimation method can construct a regular grid by directly sampling the fourth depth map and determine reliable and unreliable depth value points on the grid when the estimated depth map is the aforementioned fourth depth map. there is. The method of configuring a regular grid has been described previously and will not be described further here.

The depth estimation method can configure a regular grid and then determine reliable depth value points (P1 to PN in FIG. 10) and unreliable depth value points (Q1 to QM in FIG. 10) on the regular grid. Generally, in the case of an unreliable depth value point, the adjacent distance between the corresponding ToF projection points is relatively large, or the feature difference between the corresponding left and right images is large. Therefore, based on this, unreliable depth value points can be determined, and depth value points other than unreliable depth value points are reliable depth value points.

The depth estimation method determines reliable and unreliable depth points on a regular grid, and then locates features (e.g., depth features or image features corresponding to depth value points, etc.) of the reliable depth points on the regular grid. It is connected to information (1010), further linearly embedded (102), and finally input to a transformer encoder (1030) for encoding.

In addition, the depth estimation method concatenates (1040) the features and location information of unreliable depth value points on a regular grid and further encodes the results obtained by linear embedding (1050) with a transformer encoder (1030). By entering 1060, the depth value of the unreliable depth value point (D1 to DM in FIG. 10) can be predicted or decoded.

Finally, the depth estimation method can obtain an updated depth map by performing interpolation guided by image features of the left and right images for the area around the unreliable depth value point. The interpolation method guided by the image features of the left and right images has been described previously and will not be repeated here. Because depth values are updated on regular grid points, processing speed can be effectively improved.

Above, the depth estimation method according to an embodiment of the present disclosure has been described with reference to FIGS. 1 to 10. Hereinafter, for convenience of understanding, an example of a depth estimation method will be briefly described with reference to FIG. 11.

Figure 11 is a diagram illustrating an example of a process for estimating depth according to an embodiment.

Referring to FIG. 11, the depth estimation method acquires (1110) one ToF image and two RGB images (left and right RGB images) at the current time t, and then uses the aforementioned two-way propagation-based ToF reliability inference method. A first confidence level for each ToF pixel can be quickly determined (1120).

Additionally, the depth estimation method can check whether the quantity of ToF pixels (quantity of reliable points) of the first reliability that satisfies preset requirements is greater than s0 (1130).

As a result of the confirmation in step 1130, if the quantity of ToF pixels (quantity of trustworthy points) of the first reliability that meets the preset requirements is greater than s0, the depth estimation method is performed by fusing the ToF image with stereo matching. The depth map of the scene can be estimated (1150). For example, in this fusion method, the depth estimation method first projects the ToF image on the left RGB image and the right RGB image, respectively, and performs interpolation of the RGB guide in the area where ToF projection points are concentrated (area where ToF points are concentrated). (1152). Next, the depth estimation method can perform stereo matching of the left and right RGB images based on the first reliability and the ToF image (1154). Then, based on the ToF image and the stereo matching result, the fourth reliability of each ToF pixel can be predicted using Transformer (1156). Finally, the depth estimation method can perform stereo matching of the left and right RGB images based on the ToF image and the predicted fourth reliability (1158).

As a result of the confirmation in step 1130, if the number of reliable points is too small (s0 or less), the depth estimation method directly performs stereo matching (1142).

The depth estimation method can confirm the case where the quantity of reliable points is not greater than s0 but greater than s1 (1144).

As a result of the confirmation in step 1144, if the quantity of reliable points is not greater than s0 but greater than s1, the depth estimation method can estimate the depth map of the scene by fusing the ToF image with stereo matching. This fusion method can project the ToF image onto the RGB image after performing stereo matching of the left and right images in a situation where the ToF image is not used, and can perform interpolation of the RGB guide in an area where ToF projection points are dense (1146) . Next, the depth estimation method can predict the fourth reliability of each ToF pixel using a Transformer based on the ToF image and the stereo matching result (1156). Finally, the depth estimation method can perform stereo matching of the left and right images based on the ToF image and the predicted fourth reliability (1158). Here the interpolation operations and stereo matching are relatively independent. In both processing methods, the interpolation task can be performed first before stereo matching, or the stereo matching task can be performed first before interpolation.

The depth map obtained in the above process can be further updated for unreliable depth value points (1160), and through this, an updated depth map can be obtained. Specifically, this update can be performed on grid points. For example, as described in FIG. 10, the depth estimation method may first determine unreliable depth value points (unreliable grid points) on a regular grid (1162).

Additionally, the depth estimation method can use a Transformer to infer the depth value of an unreliable grid point (1162). Finally, the depth estimation method can estimate an updated depth map by performing interpolation of the RGB guide for the area around the unreliable grid point (1166).

The depth estimation method according to an embodiment of the present disclosure can be applied to augmented reality. For example, it can be applied to the rendering of virtual objects in augmented reality, the interaction of virtual objects with real space, or image projection correction in VST (Video See Through).

For example, the depth estimation method according to an embodiment of the present disclosure can be applied to head wearable smart devices such as smart glasses. Smart glasses may include a ToF image sensor and a pair of color image sensors. After acquiring the ToF image of the scene through the ToF image sensor and the left and right images of the scene through the color image sensor, the depth map of the scene can be estimated using the depth estimation method according to an embodiment of the present disclosure. Using the estimated depth map, smart glasses can determine the depth value of each object in the scene, display virtual objects on or around objects expected to be interacted with according to the determined depth value, and allow the wearer to play the game. You can further interact with virtual objects displayed as . Through the combination of virtual objects and objects in real space, the wearer can experience an immersive feeling.

Above, the depth estimation method according to an embodiment of the present disclosure has been described by combining FIGS. 1 to 11. According to the above-described depth estimation method, the depth information of the scene can be estimated more accurately and efficiently.

Figure 12 is a block diagram showing a depth estimation device according to an embodiment.

Referring to FIG. 12 , the depth estimation device 1200 may include an image acquisition unit 1210, a reliability determination unit 1220, and a depth estimation unit 1230. Specifically, the image acquisition unit 1210 is configured to obtain a time-of-flight ToF image, a left image, and a right image of the scene. The reliability determination unit 1220 is configured to determine the first reliability of each ToF pixel in the ToF image. The depth estimation unit 1230 is configured to estimate the depth map of the scene based on the left image, right image, and ToF image according to the first reliability. Optionally, the depth estimation device 1200 may further include an update unit (not shown) for updating the depth value of an unreliable depth value point in the estimated depth map.

The depth estimation method shown in FIG. 1 can be performed by the depth estimation device 1200 shown in FIG. 12, and the image acquisition unit 1210, reliability determination unit 1220, and depth estimation unit 1230 each have 110 Since steps 120 and 130 are performed, all relevant details related to the operations performed by each configuration in Figure 12 may refer to the corresponding descriptions in Figures 1 to 11, and will not be repeated further here.

In addition, when introducing the depth estimation device 1200, it was divided into an image acquisition unit 1210, a reliability determination unit 1220, and a depth estimation unit 1230 to perform the corresponding processing separately, but the image acquisition unit 1210 , it is obvious to those skilled in the art that the processing performed by the reliability determination unit 1220 and the depth estimation unit 1230 may be performed in the depth estimation device 1200 without division or clear boundaries between configurations. Additionally, the depth estimation device 1200 may further include other components such as an image preprocessor, memory, etc.

Figure 13 is a block diagram of an electronic device according to an embodiment.

Referring to FIG. 13, the electronic device 1300 includes at least one memory 1310 and at least one processor 1320, wherein the at least one memory 1310 stores computer-executable instructions and performs computer execution. The enabling instructions, when executed by at least one processor 1320, cause at least one processor 1320 to execute a depth estimation method according to an embodiment of the present disclosure.

At least one configuration among the plurality of configurations can be implemented through an AI model. AI-related functions can be performed by non-volatile memory, volatile memory, and processors.

A processor may include one or more processors. At this time, one or more processors may be a general-purpose processor (e.g., central processing unit (CPU), application processor (AP), etc.) or a pure graphics processing unit (e.g., graphics processing unit (GPU), visual processing unit (VPU)), and/ Alternatively, it may be a dedicated AI processor (e.g., neural processing unit (NPU)).

One or more processors control the processing of input data according to predefined operation rules or artificial intelligence (AI) models stored in non-volatile memory and volatile memory. Provides predefined action rules or artificial intelligence models through training or learning. Here, provision by learning means applying a learning algorithm to a plurality of learning data to obtain an AI model with predefined operation rules or desired characteristics. This learning may be performed on the device itself where AI according to the embodiment is performed, and/or may be implemented by a separate server/device/system.

A learning algorithm is a method of inducing, allowing, or controlling a target device (e.g., a robot) to determine or predict the target device by training it using a plurality of learning data. Examples of such learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

According to the present invention, in an image processing method executed in an electronic device, an output image after processing the target area can be obtained by using an input image as input data for an AI model.

AI models can be obtained through training. Here, 'gaining through training' means training an artificial intelligence model configured to execute predefined operating rules or necessary characteristics (or goals) by training the basic artificial intelligence model with multiple training data through a training algorithm.

For example, an AI model may consist of multiple neural network layers. Each layer has multiple weight values, and the calculation of one layer is performed based on the calculation results of the previous layer and multiple weights of the current layer. Examples of neural networks include convolutional neural networks (CNN), deep neural networks (DNN), recurrent neural networks (RNN), restricted Boltzmann machines (RBM), deep belief networks (DBN), bidirectional recurrent deep neural networks (BRDNN), and generative correspondence networks ( GAN) and deep Q networks. For example, the electronic device may be a PC computer, tablet device, PDA, smartphone, or other device capable of executing the aforementioned instruction set. Here, the electronic device does not have to be a single electronic device, but may be a collection of devices or circuits that can individually or jointly execute the command (or set of commands). The electronic device may also be part of an integrated control system or system manager, or may consist of a portable electronic device that interfaces locally or remotely (e.g., via wireless transmission).

In electronic devices, a processor may include a central processing unit (CPU), graphics processing unit (GPU), programmable logic device, dedicated processor system, microcontroller, or microprocessor. By way of example and not limitation, the processor may further include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, and the like.

The processor can execute instructions or code stored in memory, which can further store data. Commands and data may also be transmitted and received over a network via a network interface, which may use any known transmission protocol.

Memory may be integrated with the processor, for example by arranging RAM or flash memory within an integrated circuit microprocessor or the like. Additionally, memory may include independent devices such as external disk drives, storage arrays, or other storage devices that may be used in any database system. The memory and processor are operatively coupled or communicate with each other through I/O ports, network connections, etc., allowing the processor to read files stored in the memory.

Additionally, the electronic device may further include a video display (eg, liquid crystal display) and a user interaction interface (eg, keyboard, mouse, touch input device, etc.). All components of an electronic device may be connected to each other via buses and/or networks.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may store program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. , can be saved. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

Although the embodiments have been described with limited drawings as described above, those skilled in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the following claims.

Claims (20)

In the depth estimation method,
Receiving a Time Of Flight (TOF) image, a left image, and a right image of the scene;
Confirming a first reliability of each TIOF pixel of the TIOF image; and
Estimating a depth map of the scene based on the TFO image, the left image, and the right image according to the first reliability.
Depth estimation method including.
According to paragraph 1,
Estimating a depth map of the scene based on the TFO image, the left image, and the right image according to the first reliability,
confirming the quantity of TIOF pixels of the first reliability that satisfies a predetermined requirement;
If the quantity of TFO pixels of the first reliability meets a first threshold requirement, estimating a depth map of the scene based on the TFO image, the left image, and the right image; and
If the quantity of TIOF pixels of the first reliability does not meet the first threshold requirement, estimating the depth map of the scene based on the left image and the right image without using the TFO image. step
Depth estimation method including.
According to paragraph 1,
The step of checking the first reliability of each TIOF pixel of the TIOF image is,
Projecting each TIOF pixel of the TIOF image onto the left image and the right image, respectively;
For each Thiof projection point after projection, calculating a second reliability of a Thiof pixel corresponding to each Thiof projection point on the corresponding Thiof projection point scan line along a first direction; and
Calculating the first reliability of a TIOF pixel corresponding to each TIOF projection point along a second direction opposite to the first direction based on a reliability calculation result in the first direction.
Depth estimation method including.
According to paragraph 3,
The step of calculating the second reliability and the step of calculating the first reliability are,
The difference in image characteristics of each TIOF projection point in the left image and the right image and the distance between each TIOF projection point on the corresponding projection point scan line correspond to similar TIOF projection points with image characteristics within a preset range. Based on the third reliability of the TIOF pixel, calculating the second reliability and the first reliability of the TIOF pixel corresponding to each TIOF projection point.
Depth estimation method.
According to paragraph 2,
The step of estimating the depth map of the scene based on the TIOF image, the left image, and the right image,
performing first stereo matching of the left image and the right image;
predicting a fourth reliability of each TIOF pixel based on the TFO image and the first stereo matching result using a first neural network; and
Estimating a first depth map of the scene by performing a second stereo matching of the left image and the right image based on the TIOF image and the fourth reliability.
Depth estimation method including.
According to clause 5,
The step of performing the first stereo matching of the left image and the right image includes:
If the quantity satisfies the first threshold requirement and the second threshold requirement, performing the first stereo matching of the left image and the right image based on the first reliability and T.O.F image; or
If the quantity meets the first threshold requirement and does not meet the second threshold requirement, performing the first stereo matching of the left image and the right image in a situation where the TIOF image is not used. step
Depth estimation method including.
According to clause 5,
Predicting the fourth reliability of each TFO pixel based on the TFO image and the first stereo matching result using the first neural network,
Predicting the fourth reliability of each TIOF pixel by using at least one piece of information among first information, second information, and third information as an input to the first neural network.
Contains,
The first information is,
It is the difference between the disparity value corresponding to each TIOF pixel determined according to the TIOF image and the disparity value of each TIOF pixel determined through the first stereo matching,
The second information is,
It is the difference in image characteristics of each TIOF pixel at the projection point of the left image and the right image,
The third information is,
The difference between the projection point of each Thiofe pixel of the left image and the right image and the depth value of at least one Thiofe projection point with similar image features in the projection point area,
Depth estimation method.
According to clause 5,
Estimating the first depth map of the scene by performing the second stereo matching of the left image and the right image based on the T.O.F image and the fourth reliability includes,
Calculating a matching cost of a candidate disparity corresponding to each TIOF pixel during the second stereo matching based on the value of each TIOF pixel of the TIOF image and the fourth reliability of each predicted TIOF pixel. step; and
Determining a disparity value corresponding to each TIOF pixel based on the matching cost, and estimating the first depth map using the determined disparity value.
Depth estimation method including.
According to clause 5,
The step of estimating the depth map of the scene based on the TIOF image, the left image, and the right image,
The TIOF image is projected onto the left image and the right image, and interpolation is performed based on the image features of the left image and the right image for a TIOF projection point area that satisfies a preset density to obtain a second depth. estimating a map;
estimating a third depth map by performing interpolation on the first depth map based on image features of the left image and the right image; and
Estimating a fourth depth map of the scene based on the second depth map and the third depth map.
A depth estimation method further comprising:
According to clause 9,
The step of estimating the second depth map by performing interpolation based on image features of the left image and the right image for the TIOF projection point area that satisfies the preset density includes,
performing interpolation based on image features of the left image and the right image for adjacent TIOF projection points spaced apart within a preset distance on the scan line of each TIOF projection point;
Constructing a regular grid of TIOF projection points by sampling the interpolated TIOF projection points; and
Estimating the second depth map by performing interpolation based on image features of the left image and the right image for each TIOF projection point on each constructed grid.
Depth estimation method including.
According to clause 9,
Performing interpolation based on the image features of the left image and the right image,
Determines the depth value of the point to be interpolated based on the spatial distance between the point to be interpolated and adjacent reference points and the difference in image features.
Depth estimation method.
According to paragraph 2,
The step of estimating the depth map of the scene based on the left image and the right image without using the T.O.F image,
Comprising the step of estimating a fifth depth map of the scene through stereo matching of the left image and the right image.
Depth estimation method.
According to any one of paragraphs 8, 9, and 12,
Step of updating the depth value of an unreliable depth value point in the estimated depth map.
It further includes,
The estimated depth map is,
Containing the first depth map, the fourth depth map, or the fifth depth map
Depth estimation method.
According to clause 13,
The step of updating the depth value of the unreliable depth value point in the estimated depth map includes:
determining reliable depth value points and unreliable depth value points among the estimated depth map;
predicting a depth value of the unreliable depth value point based on characteristics of the reliable depth value point and the unreliable depth value point using a second neural network; and
For the area surrounding the unreliable depth value point, performing interpolation based on image features of the left image and the right image to estimate an updated depth map.
Depth estimation method including.
According to clause 14,
The step of determining the reliable depth value point and the unreliable depth value point among the estimated depth map includes:
Constructing a regular grid of depth value points based on the estimated depth map, and determining the reliable depth value points and the unreliable depth value points on the regular grid.
Depth estimation method including.
In electronic devices,
at least one processor; and
Memory that stores at least one computer executable instruction
Including,
The processor,
Receive a Time Of Flight (TOF) image, a left image, and a right image of a scene, determine a first reliability of each TOF pixel of the TOF image, and determine the TOF image according to the first reliability; Estimating the depth map of the scene based on the left image and the right image
Electronic devices.
According to clause 16,
The processor,
When estimating the depth map of the scene,
Confirming the quantity of TIOF pixels of the first reliability that meets predetermined requirements,
If the quantity of TIOF pixels of the first reliability meets a first threshold requirement, estimate a depth map of the scene based on the TFO image, the left image, and the right image,
If the quantity of TFO pixels of the first reliability does not meet the first threshold requirement, a depth map of the scene is created based on the left image and the right image in a situation where the TFO image is not used. presumed
Electronic devices.
According to clause 16,
The processor,
When determining the first reliability of each ThiofF pixel of the Thiofe image,
Projecting each TIOF pixel of the TIOF image onto the left image and the right image, respectively,
For each Thiof projection point after projection, calculate a second reliability of the Thiof pixel corresponding to each Thiof projection point on the corresponding Thiof projection point scan line along the first direction;
Calculating the first reliability of a TIOF pixel corresponding to each TIOF projection point based on a reliability calculation result in the first direction along a second direction opposite to the first direction.
Electronic devices.
According to clause 17,
The processor,
When estimating the depth map of the scene based on the TIOF image, the left image, and the right image,
Perform first stereo matching of the left image and the right image,
Predicting a fourth reliability of each TIOF pixel based on the TIOF image and the first stereo matching result using a first neural network,
Performing a second stereo matching of the left image and the right image based on the TIOF image and the fourth reliability to estimate the first depth map of the scene
Electronic devices.
According to clause 17,
The processor,
When estimating the depth map of the scene based on the left image and the right image without using the T.O.F image,
Estimating a fifth depth map of the scene through stereo matching of the left image and the right image.
Electronic devices.