WO2023225235A1

WO2023225235A1 - Method for predicting depth map via multi-view stereo system, electronic apparatus and storage medium

Info

Publication number: WO2023225235A1
Application number: PCT/US2023/022791
Authority: WO
Inventors: Changjiang Cai; Pan JI; Qingan Yan; Yi Xu
Original assignee: Innopeak Technology Inc
Current assignee: Innopeak Technology Inc
Priority date: 2022-05-19
Filing date: 2023-05-18
Publication date: 2023-11-23
Anticipated expiration: 2024-11-19
Also published as: CN119173906A

Abstract

A method for predicting depth map via multi-view stereo system, an electronic apparatus and a storage medium are provided. The method for predicting depth maps via the multi-view stereo system performed on the electronic apparatus comprises: extracting source features from source images, and extracting an aggregated reference feature from a reference image; constructing a cost volume based on the source features and the aggregated reference feature; performing an iterative estimation having multiple iterations based on the cost volume to obtain a series of index fields, the iterative estimation comprising: for each iteration, predicting a residual value through a gated recurrent neural network by using the last index field, and obtaining the current index field according to the residual value and the last index field; and estimating a first depth map based on the final index field.

Description

METHOD FOR PREDICTING DEPTH MAP VIA MULTI-VIEW STEREO SYSTEM, ELECTRONIC APPARATUS AND STORAGE MEDIUM CROSS-REFERENCE TO RELATED APPLICATION This application claims the priority benefit of US provisional application serial no. 63/343,910, filed on May 19, 2022. The entirety of the above-mentioned patent application is hereby incorporated by reference herein and made a part of this specification. Technical Field [0001] The disclosure relates to a three-dimensional (3D) reconstruction technology, and in particular to a method for predicting depth map via the multi-view stereo system, an electronic apparatus and a non-transitory computer readable storage medium. Related Art [0002] Multi-view stereo (MVS) aims to recover dense 3D geometry from multiple images captured from different viewpoints with calibrated cameras. It is a fundamental problem in computer vision and has wide applications ranging from autonomous driving, remote sensing, augmented reality, to robotics. Following the seminal MVSNet, many learning-based methods have been proposed, achieving great improvements against their traditional counterparts, in terms of accuracy or efficiency. [0003] Most of learning-based MVS methods rely on traditional plane-sweeping approach to generate a cost volume by comparing the convolutional neural network (CNN) features of reference image and source images at several depth planes, and then apply 2D or 3D convolutional encoder-decoder architectures to aggregate and regularize the cost volume. [0004] The 2D CNN methods use multi-level features as the skip connections to help decode the cost volume for depth regression. Even though the skip-connections improve the depth maps, they weaken the role of cost volume and the geometry knowledge embedded therein to some extent. Hence, 2D CNN methods suffer from degraded generalization when testing on unseen domains. [0005] The 3D CNN methods use soft-argmin to regress the depth map as the expectation from the cost volume distribution, and hence cannot predict the best candidate but instead an averaged one when dealing with a flat or multi-modal distribution caused by textureless, repeated, or occluded regions, etc. Therefore, when this situation happens, the soft-argmin cannot well classify to target the best candidate, but instead gives an average prediction, by aggregating the unseen best candidate, the worst ones and the ones in the middle. [0006] Therefore, how to improve the accuracy and generalization ability of the multi-view stereo is one of the problems to be solved at present. SUMMARY OF INVENTION [0007] The disclosure provides a method for predicting depth map via a multi-view stereo system, electronic apparatus and storage medium to predict the depth accurately. [0008] A method for predicting depth map via a multi-view stereo system performed on an electronic apparatus according to the disclosure, comprises: extracting source features from a plurality of source images, and extracting an aggregated reference feature from a reference image, wherein the reference image is one of a plurality of captured images of the same scene with different camera poses, and the source images are remaining captured images; constructing a cost volume based on the source features and the aggregated reference feature; performing an iterative estimation having a plurality of iterations based on the cost volume to obtain a series of index fields, the iterative estimation comprising: for each of iterations, predicting a residual value through a recurrent neural network by using an index field obtained last time, and obtaining an index field of current iteration according to the residual value and the index field obtained last time; and estimating a first depth map based on the index field obtained finally in the series of index fields. [0009] An electronic apparatus comprises: a storage, configured to store a program; and a processor, coupled to the storage, and configured to execute the program to: extract source features from a plurality of source images, and extracting an aggregated reference feature from a reference image, wherein the reference image is one of a plurality of captured images of the same scene with different camera poses, and the source images are remaining captured images; construct a cost volume based on the source features and the aggregated reference feature; perform an iterative estimation having a plurality of iterations based on the cost volume to obtain a series of index fields, the iterative estimation comprising: for each of iterations, predicting a residual value through a recurrent neural network by using an index field obtained last time, and obtaining an index field of current iteration according to the residual value and the index field obtained last time; and estimate a first depth map based on the index field obtained finally in the series of index fields. [0010] Based on the above description, the disclosure exploits a recurrent neural network to iteratively optimize a final index fields obtained based on the cost volume for depth regression. Accordingly, the depth can be predicted via learning to recurrently index an asymmetric cost volume, leading to improve accuracy and generalization. BRIEF DESCRIPTION OF DRAWINGS [0011] The accompanying drawings are included to provide a further understanding of the invention, and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention. [0012] FIG. 1 is a block diagram of a user equipment according to an embodiment of the disclosure. [0013] FIG. 2 is a flowchart of a method for predicting depth map via the multi-view stereo system according to an embodiment of the disclosure. [0014] FIG.3 is a schematic diagram illustrating feature extraction and cost volume construction according to an embodiment of the disclosure. [0015] FIG. 4 is a schematic diagram illustrating depth prediction using iterative estimation according to an embodiment of the disclosure. [0016] FIG.5 is a schematic diagram illustrating depth prediction using iterative estimation and residual pose estimation according to an embodiment of the disclosure. DESCRIPTION OF EMBODIMENTS [0017] FIG. 1 is a block diagram of a user equipment according to an embodiment of the disclosure. Please refer to FIG. 1, an electronic apparatus 100 is an apparatus having an arithmetic capability. The electronic apparatus 100 may be, for example, a server, a desktop computer, a laptop computer, a network computer, a workstation, a personal digital assistant (PDA), a tablet personal computer (PC), and the like. [0018] The electronic apparatus 100 comprises a processor 110 and a storage 120. The processor 110 is coupled to the storage 120. The processor 110 is, for example, a central processing unit (CPU), a physics processing unit (PPU), a programmable microprocessor, an embedded control chip, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or other similar devices. [0019] The storage 120 is, for example, any type of fixed or removable random-access memory (RAM), read-only memory (ROM), flash memory, hard disk drive, other similar apparatuses, or a combination of the apparatuses. The storage 120 stores a plurality of code fragments, and the code fragments are executed by the processor 110 after being installed, so as to execute the method for predicting depth map via the multi-view stereo system. [0020] In the following embodiments, depth prediction is implemented by learning to optimize the cost volume directly to iteratively estimate a series of index fields. To facilitate the optimization, the cost volume in pixel- and frame levels are improved respectively. In the pixel level, a transformer layer is asymmetrically applied to the reference view (but not to source views) to build an asymmetric cost volume to store more accurate matching similarity cues. In the frame level, a residual pose network is provided to rectify the camera poses, which are usually obtained via Visual SLAM and inevitably contain noise, to more accurately backward warp the reference features to match its counterparts in source views. [0021] FIG. 2 is a flowchart of a method for predicting depth map via the multi-view stereo system according to an embodiment of the disclosure. In the embodiment, the method for predicting depth map would be roughly divided into feature extraction, cost volume construction and depth estimation. Details will be discussed below. [0022] Please refer to FIG. 1 and FIG. 2. In step S205, the processor 110 extracts source features from a plurality of source images, and extracts an aggregated reference feature from a reference image. In one embodiment, at least one imaging sensor (ex. calibrated camera) is used to capture a set of captured images from different viewpoints of the same scene with known camera poses, and the set of captured images is denoted by I={I₀, I₁, I₂, …, I_N-2, I_N-1}. One of the captured images is used as the reference image, and other remaining captured images are used as the source images. For example, the reference image is the first captured image denoted by I₀ and the source images are other captured images I₁ to I_N-1 denoted by . Then, feature extraction is

performed to extract source features (denoted by f^S) from the source images I^S, and extract the aggregated reference feature (denoted by ) from the reference image I0.

[0023] Next, in step S210, the processor 110 constructs a cost volume based on the source features and the aggregated reference feature. In one embodiment, the cost volume is defined on a 3D view frustum attached to the camera in perspective projection, and it is generated by plane- sweep stereo. [0024] FIG.3 is a schematic diagram illustrating feature extraction and cost volume construction according to an embodiment of the disclosure. Please refer to FIG.3, a feature extractor 310 is configured for the source features, and a feature extractor 320 and a transformer layer 330 are configured for the reference image. In one embodiment, the feature extractors 310 and 320 are based on PairNet. The PairNet is a lightweight feature pyramid network (FPN) on top of first fourteen layers of MnasNet. The transformer layer 330 may be implemented by four-head self- attention with positional encoding. The feature extractor 310 is used for extracting source features from the source images I^S. The feature extractor 320 is

used for extracting a matching feature f₀ from the reference image I₀ and the transformer layer 330 is used to construct the aggregated reference feature based on the matching feature f₀.

[0025] Specifically, in one embodiment, the reference image is spatially scaled down

until 1/32 scale, and recovered up to 1/2 scale, resulting in multi-scale features

(s=2, 4, 8, 16 and F₀=32 for feature channels). In the feature extractor 320, a

fusion layer G is provided to aggregate 1/2 scale feature f_0,2 and other features (f_0,4, f_0,8, f_0,16) into the matching feature f₀ at 1/4 scale, as: . The fusion layer G is

a sequence of operation of Conv_3×3 function, batch normalization, rectified linear unit (ReLU), and Conv_1×1. Herein, ↓x and ↑x are downsampling and upsampling by scale x respectively,

is concatenation along channel dimension, and with F₁=128. Similarly, the

feature extractor 310 (with shared weights as that for I₀) is also applied to source images I^S to extract a set of source features f^S. [0026] Besides the local pixel-wise features (matching feature f₀) extracted from feature extractor 320, the transformer layer 330 is used to leverage global long-range information to better guide the feature matching. The transformer layer 330 is applied to the matching feature (local feature) f0 of the reference image I0, to construct the aggregated reference feature

as equation (1):

where σ(∙) is the softmax operator, ω_α is a learned scalar weight that is initialized to zero, and W^Q, W^K and W^V are the projections matrices for query, key and values, with h=4 for multi- head attention.

and

. [0027] The final output (reference feature

) of the transformer layer 330 contains both local and global information, which are balanced by the scalar weight ω_α, to enhance the following cost volume construction. [0028] The transformer layer 330 is only applied to the reference image I₀, while the source features

still possess the local representations from the feature extractor 310. The asymmetric employment of the transformer layer 330 provides the capability to better balance the high-frequency features (by high-pass CNNs) and the low-frequency features by self-attention. The high-frequency features are beneficial to image matching at local and structural regions, while the low-frequency ones, with noisy information suppressed by spatial smoothing (serving as a low-pass filter) of the transformer layer 330, provides more global context cues for robust matching, especially for the areas full of low-texture, repeated patterns, and occlusion, etc. Therefore, the network architecture containing the feature extractor 320 and the transformer layer 330 can learn where to rely on global features over local features, and vice versa. [0029] Next, the source features f^S and the aggregated reference feature are used to

construct the cost volume C₀. In one embodiment, the cost volume C₀ is generated by running the traditional plane-sweep stereo which uniformly samples M₀=64 depth hypotheses in the inverse depth space, s.t.1/d ~ U(d_min, d_max). Here, dmin and dmax are the near and far planes of the 3D frustum respectively. [0030] For a given depth hypothesis d and known camera intrinsic matrices K={K_i} (i=0, 1, 2, …, N-1) and original relative poses between the reference image I0 and the

source images Ii. Herein, R_0.i represents a rotation matrix between the reference image I0 and the source images I_i, and t_0,i represents a translation vector between the reference image I₀ and the source images I_i.

is a collection of . θ1 represents the relative pose the

reference image I0 and the source image I₁, θ₂ represents the relative pose the reference image I0 and the source image I₂, …, θ_N-1 represents the relative pose the reference image I₀ and the source image I_N-1. A cost map is computed by warping source feature fi into the reference image I0 and calculating the similarity between the aggregated reference feature and the warped feature .

A homography is used as a backward 2D grid sampling to generate the warped feature

. Here, B0 represents a set of depth planes,

represents the original set of relative poses. Specifically, a pixel in the reference image I0 will be warped to its counterpart in

source image I_i as equation (2):

[0031] Then the warped feature

is bilinearly sampled from as Given the

warped feature

and the aggregated reference feature

, the cost volume is formulated as equation (3):

[0032] The cost volume C₀ is constructed for all depth planes ,

resulting in a 3D tensor, denoted as

[0033] After constructing the cost volume C₀, in step S215, the processor 110 performs an iterative estimation having a plurality of iterations based on the cost volume C₀ to obtain a series of index fields. Then, in step S220, the processor 110 estimates a first depth map based on the index field obtained finally in the series of index fields. In the iterative estimation, for each of iterations, a residual value is predicted through a recurrent neural network (ex. gated recurrent unit (GRU)) by using an index field obtained last time, and an index field of current iteration is obtained according to the residual value and the index field obtained last time. For example, in the first iteration, an updated index filed for the current iteration is obtained using the predicted residual value and the initial index field. In each of other iterations except the first iteration, an updated index filed for the current iteration is obtained using the predicted residual value and the index field from last iteration. In one embodiment, the index field of the current iteration is obtained by adding the residual value and the index field obtained last time. [0034] In the embodiment, the processor 110 learns to index the cost volume C₀ by approaching correct depth planes per pixel via an index field. The proposed iterative estimate of index field (a grid of indices to identify the depth planes) enables the learning to be anchored at the cost volume domain. Specifically, it recurrently predicts the residual value, so as to retrieve cost features for the next iteration in a descent direction of cost volume, but also to directly index (i.e., sampling via linear interpolation) depth planes to render a depth map, which is iteratively optimized to approach the ground truth depth, making the system end-to-end trainable. [0035] FIG. 4 is a schematic diagram illustrating depth prediction using iterative estimation according to an embodiment of the disclosure. In the embodiment, in order to optimize the dense stereo matching problem, semi-global matching (SGM) is adopted. In SGM, the cost volume C₀ is iteratively aggregated by summing the costs (of all 1D minimum cost paths that end in pixel p at disparity d) when traversing from pixel p-r to pixel p in a direction r (out of sixteen directions) and the best disparity at each pixel p is given by d*(p)=argmind(C’(p, d)), with C’ being the aggregated cost volume. Similar to SGM, in the embodiment, the argmin is replaced with softargmin when predicting the disparity from the cost volume C₀, but still i) the update direction r when traversing from pixel p-r to p needs to be predefined, and ii) the softargmin focuses on measuring the distance of the expectation of disparity map to the ground truth disparity, and hence cannot handle multi-modal distributions in C₀ well. [0036] Please refer FIG.4, the architecture for predicting depth map further includes a context feature network 410 and a GRU-based module 420. The context feature network 410 is used to extract a context feature from the reference image I₀. For example, the context feature

network 410 consists of several residual blocks. And the GRU-based module 420 is used to implicitly optimize the cost volume C₀. A sequence of index fields are estimated by unrolling the optimization problem to iterative updates. [0037] The GRU-based module 420 estimates a series of index fields in

a plurality of iterations from an initial index field . Specifically, before performing the

iterations, an initial value is determined from the cost volume C₀ by sof-argmin function as the initial index field

. Then at each iteration t (assuming T iterative updates, t=1, 2, ... , T), the index field is estimated as a grid of indices to iteratively better approach (i.e., closer to the ground truth) depth planes having a lower matching cost. [0038] In one embodiment, a four-layer matching pyramid is built by repeated pooling

the cost volume C₀ along the depth dimension with kernel 2. To index the matching pyramid, lookup operator L is defined for retrieving cost features from the cost volume C₀ according

to the index field obtained last time. For example, in the iteration 1, the lookup operator L retrieves cost features from the cost volume C₀ according to the index field , in the iteration

2 the lookup operator L retrieves cost features from the cost volume C₀ according to the

index field and in the iteration T the lookup operator L retrieves cost features from

the cost volume C₀ according to the index field . [0039] The index field

, the retrieved cost features and the context feature

are

concatenated, and fed into the GRU module 420, together with a latent hidden state h_t. The GRU module 420 outputs a residual value Δϕt-1 and a new hidden state h_t+1:

[0040] The first depth map D_T at iteration T is estimated by sampling the depth hypotheses via linear interpolation given the index field ϕT obtained finally. In one embodiment, assuming the index field is at 1/4 resolution, a upsampling operator U (ex. a convex combination of a 3×3 neighbors) is used to upsample the index field to full resolution. For example, weight mask

is predicted from the hidden state ht using two convolutional layers and

softmax is performed over the weights of those 9 neighbors. The final high resolution index field (upsampled index field

) is obtained by taking a weighted combination over the 9 neighbors, and reshaping to the resolution H×W. Convex combination can be implemented using the einsum function in PyTorch. [0041] In one embodiment, a set of depth planes is used to construct

the cost volume C₀. A small M₀ helps reduce the computation and space. The spatial upsampling is performed for the depth planes B₀ to obtain depth planes

. The depth index fields are adjusted by a scale . To mimic the convex

combination before mentioned, a similar weighted summation is applied along the depth dimension when sampling depth from depth planes B₁. Specifically, another mask W₁ ∈

predicted from the hidden state using three convolutional layers, and further reshaped to . Given a pixel p, and the upsampled index field , the final depth (first

depth map DT) is estimated as equation (4):

where, the neighbors are aggregated within a radius r=4 centered at the index field

for a given pixel p, and ⌊ ⌋ gives a greatest integer less than or equal to i, and [i] means to index the depth planes B₁ via linear interpolation, due to index i being a real number. [0042] The iterative estimate of the index field enables the learning to be directly anchored at the cost volume domain. This indexing paradigm differentiates our approach from other depth estimation methods, such as the convolution-based depth regression which fuses cost volume and the skipped multi-level features by 2D CNNs, or soft-argmin based depth regression after cost volume aggregation and regularization by 3D CNNs. [0043] In another embodiment, an accurate cost volume benefits the GRU-based iterative optimization. The quality of the generated cost volume C₀ is not only determined by the aggregated reference feature and the source features f^S (for which an asymmetric employment

of the transformer layer is proposed), but also by the homography warping. In practice, the camera poses are usually obtained by visual SLAM (simultaneous localization and mapping) algorithms and inevitably contains noise. Accordingly, a residual pose network is provided to rectify the camera poses for accurately backward warping the reference feature to match the corresponding features in the source images. [0044] FIG.5 is a schematic diagram illustrating depth prediction using iterative estimation and residual pose estimation according to an embodiment of the disclosure. The embodiment shown in FIG. 5 is based on the architecture of FIG. 4. The system architecture (including feature extractors 310 and 320, the transformer layer 330, context feature extractor 410, GRU-based iterative optimization layers 420, and the residual pose network 520, etc.) are trained to minimize the loss between the predicted depth maps and the ground truth depth. [0045] Herein, in view of inaccuracy of poses between reference image and source images, a residual pose network 520 is incorporated to correct the relative poses. This essentially rectifies the cost volume in the frame-level. Another round of GRU-based iterative updates will be performed based on the new cost volume and the depth maps will be predicted as the final output. [0046] After estimating the first depth map D_T (for details, please refer to FIG.4), an updated cost volume C1 is constructed based on a residual pose adjusted by the first depth map DT. Then, the iterative estimation based on the updated cost volume C₁ is re-performed to obtain another series of index fields and a second depth map

is estimated based on the index

field (ϕ"_T) obtained finally in the another series of index fields. The evaluation of the second depth map is analogous to the evaluation of the first depth map D_T. [0047] Before constructing the updated cost volume C1, the source images I^S are warped respectively to obtain a plurality of warped source images (i.e.,

warped , and the residual pose network 520 is used to encode the reference image

I₀ and the warped source images with at least one of the first depth map DT, a camera pose (i.e.

the original relative poses

) and a ground truth depth

, and obtaining the residual pose

. Here, represents a collection of relative poses between the reference image I₀ and the source images I₁ to I_N-1. In the embodiment, the residual pose network 520 can be implemented using an image-net pre-trained residual neural network (ResNet) backbone. [0048] Specifically, given a depth map DT (currently estimated) at iteration t in the first iterative estimation and the ground truth depth D_gt, a source image I_i is warped into the reference image I₀ through the equation (2) with (noisy) ground truth camera poses, i.e., original relative poses Θ₀ and DT or Dgt. A probability prob(DT)=0.6 during the network training is used to randomly select one between D_T and D_gt, while the depth map D_T is always chosen during network inference, i.e., probability prob(D_T)=1 during the network inference. The input to the residual pose network 520 is the concatenated result obtained by concatenating the reference image I₀ and

warped source images , and the output is an axis-angle representation, which is further converted

to a residual rotation matrix (residual pose) for an updated relative pose θ . Here, ∆θ_i

is the residual pose of the relative pose θi (i=1, …, N-1). The collection of residual poses for each pair of each source image and the reference image, and perform the

rectification as . The collection of updated poses is used to implement

an updated homography

( , ), and a more accurate cost volume, i.e. updated cost volume C₁ is obtained using the updated homography

based on the equations (2) and (3). [0049] Then, the iterative estimation is re-performed based on the updated cost volume C₁ to obtain another series of index fields. Similar to that described in shown in FIG.4, an initial value is determined from the updated cost volume C₁ by the sof-argmin function as the initial index field . Then at each iteration t (assuming T iterative updates, t=1, 2, ... , T), a residual value

is obtained through the GRU-based module 420 by inputting the cost features, retrieved from the updated cost volume C₁ according to the index field ϕ"_t-1 obtained last time, and the context feature . Afterwards, the index field ϕ"_t is estimated by adding the residual value and the index

field obtained last time. [0050] Extensive experiments on real-world MVS datasets are conducted to show that the disclosure achieves state-of-the-art performance in terms of both within-dataset evaluation and cross-dataset generalization. [0051] The disclosure is implemented in PyTorch, and trained end-to-end with a mini-batch size of 8 per NVIDIA RTX A6000 GPU. During training, the AdamW optimizer is used and gradients are clipped to the range [−1, 1]. When generating the cost volume by plane-sweep stereo, the plane hypotheses number is set as M₁=256. The same hyperparameters are adopted for the context network and 3-level GRU architecture. In the training schedule, the network architecture of the disclosure is trained for 20 epochs, with an initial learning rate of 1e-4 and decayed by half at epoch 4^th and 8^th, respectively. For fair comparison, the baselines PairNet and IterMVS are trained on the same training samples of ScanNet for 20 epochs, using the official codes and Adam Optimizer. For the baseline PairNet, the suggested learning rate scheduler in their paper is used. And for the baseline Iter-MVS, a learning rate of 1e-4 is used, which is decayed by half at epoch 4^th and 8^th. The disclosure can make more accurate and sharp depth prediction, especially for regions near boundary and edges. For both near and far objects, our method outperforms the baselines IterMVS and Pair-Net. [0052] To sum up, embodiments of the disclosure provide a method for predicting depth map via multi-view stereo system, an electronic apparatus and a storage medium. The disclosure estimates the depth maps from a set of posed multi-view images (one as the reference and others as the sources images), and exploits learning-to-optimize paradigm that iteratively indexes a plane- sweeping cost volume and regresses depth maps via convolutional GRUs. The cost volume construction is further improved via the incorporation of a transformer block to the reference image and a residual pose network to correct the relative poses. [0053] Since the cost volume plays a paramount role in encoding the multi-view geometry, the disclosure aims to improve its construction both in pixel- and frame-levels. In the pixel level, a transformer layer is introduced to the reference image (but not to the source images) to break the symmetry of the Siamese network (which is typically used in MVS to extract image features). Such an asymmetric volume allows the network to extract global features from the reference image to predict its depth map. Moreover, in the frame-level, in view of inaccuracy of poses between reference and source images, a residual pose network is incorporated to correct the relative poses. This essentially rectifies the cost volume in the frame-level. [0054] The extensive experiments on ScanNet, DTU, 7-Scenes, and RGB-D Scenes V2 have demonstrated the superior accuracy and cross-dataset generalization capability of our method. [0055] It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed embodiments without departing from the scope of the disclosure. In view of the foregoing, it is intended that the disclosure covers modifications and variations provided that they fall within the scope of the following claims and their equivalents.

Claims

WHAT IS CLAIMED IS: 1. A method for predicting depth map via a multi-view stereo system performed on an electronic apparatus, comprising: extracting source features from a plurality of source images, and extracting an aggregated reference feature from a reference image, wherein the reference image is one of a plurality of captured images of a same scene with different camera poses, and the source images are remaining captured images; constructing a cost volume based on the source features and the aggregated reference feature; performing an iterative estimation having a plurality of iterations based on the cost volume to obtain a series of index fields, the iterative estimation comprising: for each of iterations, predicting a residual value through a recurrent neural network by using an index field obtained last time, and obtaining an index field of current iteration according to the residual value and the index field obtained last time; and estimating a first depth map based on the index field obtained finally in the series of index fields.

2. The method for predicting depth map via the multi-view stereo system according to claim 1, further comprising: extracting a context feature from the reference image by using a context feature network; wherein the iterative estimation further comprises: before performing the iterations, determining an initial value from the cost volume by soft- argmin function as an initial index field of the series of index fields; and for each of the iterations, comprising: retrieving cost features from the cost volume according to the index field obtained last time; inputting the cost features and the context feature to the recurrent neural network and obtaining the residual value; and obtaining the index field of the current iteration by adding the residual value and the index field obtained last time.

3. The method for predicting depth map via the multi-view stereo system according to claim 1, wherein after estimating the first depth map, the method further comprises: constructing a updated cost volume based on a residual pose adjusted by the first depth map; re-performing the iterative estimation based on the updated cost volume to obtain another series of index fields; and estimating a second depth map based on the index field obtained finally in the another series of index fields.

4. The method for predicting depth map via the multi-view stereo system according to claim 3, wherein before calculating the updated cost volume, the method further comprises: warping the source images to obtain a plurality of warped source images; and using a residual pose network to encode the reference image and the warped source images with at least one of the first depth map, a camera pose and a ground truth depth, and obtaining the residual pose.

5. The method for predicting depth map via the multi-view stereo system according to claim 1, wherein extracting the aggregated reference feature from the reference image comprises: extracting a matching feature from the reference image by a feature extractor; and constructing the aggregated reference feature based on the matching feature by a transformer layer.

6. The method for predicting depth map via the multi-view stereo system according to claim 1, wherein estimating the first depth map comprises: estimating the first depth map by sampling a plurality of depth planes via linear interpolation with the index field obtained finally in the series of index fields.

7. An electronic apparatus, comprising: a storage, configured to store a program; and a processor, coupled to the storage, and configured to execute the program to: extract source features from a plurality of source images, and extracting an aggregated reference feature from a reference image, wherein the reference image is one of a plurality of captured images of a same scene with different camera poses, and the source images are remaining captured images; construct a cost volume based on the source features and the aggregated reference feature; perform an iterative estimation having a plurality of iterations based on the cost volume to obtain a series of index fields, the iterative estimation comprising: for each of iterations, predicting a residual value through a recurrent neural network by using an index field obtained last time, and obtaining an index field of current iteration according to the residual value and the index field obtained last time; and estimate a first depth map based on the index field obtained finally in the series of index fields.

8. The electronic apparatus according to claim 7, wherein the processor is configured to execute the program to: extract a context feature from the reference image by using a context feature network, wherein the iterative estimation comprises: before performing the iterations, determine an initial value from the cost volume by soft- argmin function as an initial index field of the series of index fields; and for each of the iterations, comprising: retrieving cost features from the cost volume according to the index field obtained last time; inputting the cost features and the context feature to the recurrent neural network and obtaining the residual value; and obtaining the index field of the current iteration by adding the residual value and the index field obtained last time.

9. The electronic apparatus according to claim 7, wherein the processor is configured to execute the program to: construct a updated cost volume based on a residual pose adjusted by the first depth map; re-perform the iterative estimation based on the updated cost volume to obtain another series of index fields; and estimate a second depth map based on the index field obtained finally in the another series of index fields.

10. The electronic apparatus according to claim 9, wherein the processor is configured to execute the program to: warp the source images to obtain a plurality of warped source images; and use a residual pose network to encode the reference image and the warped source images with at least one of the first depth map, a camera pose and a ground truth depth, and obtaining the residual pose.

11. The electronic apparatus according to claim 7, wherein the processor is configured to execute the program to: extract a matching feature from the reference image; and construct the aggregated reference feature based on the matching feature by a transformer layer.

12. The electronic apparatus according to claim 7, wherein the processor is configured to execute the program to: estimate the first depth map by sampling a plurality of depth planes via linear interpolation with the index field obtained finally in the series of index fields.

13. A non-transitory computer readable storage medium, wherein the non-transitory computer readable storage medium records at least one programmed instruction, and after being loaded into an electronic apparatus, the at least one programmed instruction performs the following steps: extracting source features from a plurality of source images, and extracting an aggregated reference feature from a reference image, wherein the reference image is one of a plurality of captured images of a same scene with different camera poses, and the source images are remaining captured images; constructing a cost volume based on the source features and the aggregated reference feature; performing an iterative estimation having a plurality of iterations based on the cost volume to obtain a series of index fields, the iterative estimation comprising: for each of iterations, predicting a residual value through a recurrent neural network by using an index field obtained last time, and obtaining an index field of current iteration according to the residual value and the index field obtained last time; and estimating a first depth map based on the index field obtained finally in the series of index fields.