JP2023035903A - Depth estimation method and depth estimation device - Google Patents

Abstract

To provide a technique for acquiring a high-quality depth map.SOLUTION: A depth estimation method includes acquiring a plurality of depth maps and outputting a single output depth map by combining the plurality of depth maps while reducing an average difference in distance value between adjacent pixels, compared to a case of directly using distance values contained in the plurality of depth maps.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to depth estimation methods and depth estimation devices.

An image captured by an imaging device such as a camera represents the luminance information of a subject, while a depth map (also called a depth image) detected by a ranging sensor such as a ToF (Time of Flight) sensor or LiDAR (Light Detection and Ranging) is used. ) represents the distance or depth information between the ranging sensor and the subject. Such a depth map can be used, for example, for photo processing of captured images, object detection for autonomous operation of vehicles, robots, and the like.

With the advancement of AI (Artificial Intelligence) technology, a depth estimation model has been developed that estimates a depth map representing the depth, that is, the distance between an object and an imaging device, from an image acquired from the imaging device. For example, MiDaS (https://github.com/intel-isl/MiDaS) and DPT (https://github.com/intel-isl/DPT) are known as depth estimation models for monocular images.

On the other hand, along with the recent sophistication of mobile terminals such as smartphones and tablets, ranging sensors such as ToF (Time of Flight) sensors and LiDAR (Light Detection and Ranging) sensors have come to be installed in mobile terminals. is coming. For example, Patent Literature 1 discloses a processing system that aligns depth maps acquired by a ToF sensor and a stereo camera and outputs an optimized depth map.

Japanese Patent Application Laid-Open No. 2020-042772

However, depth maps acquired with ToF sensors, while typically having accurate distance values, may contain many missing pixels. On the other hand, while depth estimation models based on deep neural networks output globally consistent depth maps, they often fail to obtain accurate distance values and are unable to read fine textures.

For this reason, simply complementing missing pixels with a depth map obtained from a camera, as in the technique of Patent Document 1, is not sufficient for the difference between a pixel area having a distance value obtained by the ToF sensor and other pixel areas. Boundaries stand out unnaturally. In other words, a high-quality depth map cannot be acquired.

In view of the above problems, one problem of the present disclosure is to provide a technique for obtaining a high-quality depth map.

One aspect of the present disclosure is obtaining multiple depth maps and reducing the average difference in distance values of adjacent pixels compared to directly using the distance values contained in the multiple depth maps. Combining multiple depth maps to output an output depth map.

According to the present disclosure, it is possible to provide a technique for obtaining a high-quality depth map.

FIG. 3 is a schematic diagram illustrating depth estimation processing according to one embodiment of the present disclosure; 1 is a block diagram illustrating a depth estimation system according to one embodiment of the disclosure; FIG. [0014] Fig. 4 illustrates a depth map according to one embodiment of the disclosure; 1 is a block diagram showing a hardware configuration of a depth estimation device according to one embodiment of the present disclosure; FIG. 1 is a block diagram showing the functional configuration of a depth estimation device according to an embodiment of the present disclosure; FIG. 4 is a flowchart illustrating depth estimation processing according to one embodiment of the present disclosure; FIG. 4 is a schematic diagram illustrating depth estimation processing according to another embodiment of the present disclosure;

Embodiments of the present disclosure will be described below with reference to the drawings.

In the following examples, a depth map or depth image (hereinafter collectively referred to as a depth map) inferred by a depth estimation model from an image (for example, an RGB image) of a measurement target area and a depth map obtained from a ranging sensor and according to a cost function including constraints described below. For example, the depth estimation apparatus of the present disclosure can be used to implement depth completion that complements a depth map acquired by a ranging sensor to an image level equivalent to an RGB image. Note that throughout this specification, a depth map is two-dimensional data having a distance value for each pixel.

[Overview]
To summarize an embodiment of the present disclosure, which will be described later, as shown in FIG. A depth map P inferred from the RGB image of the measurement target area is synthesized to generate a synthesized depth map O. FIG. At this time, the depth estimation apparatus 100 uses the cost function to convert the distance value of the depth map T to the depth map T if the distance value of the corresponding pixel exists in the depth map T for each pixel of the depth map O. When the distance value of the corresponding pixel does not exist in the depth map T, the distance value of the depth map O is matched with the distance value of the depth map P, and the depth map O The depth map O is constructed so that the distance value of the pixel in question approaches the distance values of the neighboring pixels.

In order to realize such synthesis, the depth estimation device 100
(Constraint 1) When the depth map T has a distance value corresponding to the pixel of interest, the distance value of the pixel of interest in the depth map O is brought closer to the distance value of the depth map T.
(Constraint 2) When the depth map T does not have a distance value corresponding to the pixel of interest, the distance value of the pixel of interest in the depth map O is brought closer to the distance value in the depth map P.
(Constraint 3) Bring the distance value of the pixel of interest close to the distance value of the neighboring pixels of the pixel of interest.
The depth map T and the depth map P are synthesized according to a cost function including three constraints:

According to the depth estimation apparatus 100 of the embodiment described later, the distance value of the depth map T is used for pixels with no missing distance value in the depth map T, and the pixels with missing distance values in the depth map T are , the depth map O is constructed using the distance values of the depth map P. As shown in FIG. As a result, a highly accurate depth map O can be acquired globally. Further, since the depth map O is configured so that the distance value of each pixel of the depth map O approaches the distance value of the adjacent pixels, the depth map O smoothed between the adjacent pixels can be obtained.

[Depth estimation system]
First, referring to FIGS. 2-4, a depth estimation system according to one embodiment of the present disclosure will be described. FIG. 2 is a block diagram illustrating a depth estimation system according to one embodiment of the disclosure.

As shown in FIG. 2, the depth estimation system 10 has a camera 20, a ToF sensor 30, a preprocessing device 40 and a depth estimation device 100. As shown in FIG.

The camera 20 captures an image of the measurement target area and generates an RGB image of the measurement target area. For example, the camera 20 may be a monocular camera and generates a monocular RGB image of a measurement target area including a subject. The generated RGB image is passed to the preprocessing device 40 . However, the depth estimation system according to the present disclosure is not limited to camera 20 and may comprise any other type of imaging device that images the measurement target area. Also, the depth estimation system according to the present disclosure is not limited to RGB images and may acquire or process other forms of image data that can be converted into depth maps by preprocessor 40 and inference engine 41 .

The ToF sensor 30 detects the distance (depth) between each subject in the measurement target area and the ToF sensor 30, and generates ToF data or a ToF image (hereinafter collectively referred to as ToF data). The generated ToF data is passed to the preprocessing device 40 . However, depth estimation systems according to the present disclosure are not limited to ToF sensors 30 and may comprise any other suitable type of ranging sensor capable of generating a depth map, such as a LiDAR sensor. Ranging data corresponding to the type of sensor may be obtained.

The preprocessing device 40 preprocesses the RGB image acquired from the camera 20 and acquires a depth map P as an inference result by the inference engine 41 . Here, the inference engine 41 receives an RGB image as an input, and outputs a depth map P indicating the distance (depth) between each subject in the measurement target area and the camera 20 . For example, the inference engine 41 can be an existing depth estimation model such as MiDaS, DPT, or a model trained (eg, distilled) from any one or more existing depth estimation models. good too. Also, the inference engine 41 may be installed in the preprocessing device 40, or may be installed in an external server (not shown), and the inference result may be passed to the preprocessing device 40 via a network.

Specifically, the preprocessing device 40 inputs the acquired RGB image to the inference engine 41 and acquires a depth map P as an inference result. Typically, the depth map P is globally consistent, but does not always represent accurate distance values, and may not represent fine textures. The preprocessor 40 may then rescale the depth map P to match the size of the ToF data T. FIG.

On the other hand, the preprocessing device 40 may perform preprocessing such as noise removal on the acquired ToF data. For example, the preprocessor 40 performs opening processing on the ToF data to remove isolated pixels. This is because pixels with isolated distance values are likely to be noise. Further, the preprocessing device 40 may preprocess the background pixels of the ToF data so as to bring them closer to the depth map P. FIG. In general, the range that can be suitably ranged by the ToF sensor 30 is a range of several meters, and the distant portion of the ToF data is corrected to be closer to the distance value of the corresponding portion of the depth map P obtained by the inference engine 41. may be

Also, the preprocessing device 40 may refer to the ToF data and adaptively bring the vicinity of the center of the inference result closer to the foreground. ToF data is typically characterized by an inability to capture hyper-near objects or objects of a particular color. Therefore, if the ToF data T and the depth map P are synthesized without preprocessing, the scale of the depth map O is adjusted to the distant view portion, and the object in the central portion becomes the distant view. The preprocessing device 40 passes the ToF data T and the depth map P preprocessed in this manner to the depth estimation device 100 .

The depth estimation device 100 synthesizes the ToF data T and the depth map P obtained from the preprocessing device 40 according to the cost function to generate a synthesized depth map O. FIG. A cost function according to one embodiment of the present disclosure is:
(Constraint 1) When the ToF data T has a distance value corresponding to the pixel of interest, the distance value of the pixel of interest in the depth map O is brought close to the ToF data T distance value.
(Constraint 2) When the ToF data T does not have a distance value corresponding to the pixel of interest, the distance value of the pixel of interest in the depth map O is brought closer to the distance value of the depth map P.
(Constraint 3) Bring the distance value of the pixel of interest close to the distance value of the neighboring pixels of the pixel of interest.
It may include the three constraints of That is, the depth estimation apparatus 100 uses the distance value of the depth map T for pixels with no missing distance value in the ToF data T, and uses the depth map T for pixels with no missing distance value in the depth map T. Construct a depth map O using the distance values of P. Furthermore, the depth estimation apparatus 100 configures the depth map O such that the distance value of each pixel of the depth map O is brought closer to the distance value of the adjacent pixels. This makes it possible to acquire a smoothed depth map O with high accuracy globally.

For example, as shown in FIG. 3, when an inferred depth map P and ranged ToF data T are acquired for the measurement target area, the depth estimation apparatus 100 performs the following according to the above-described cost function: A synthesized depth map O as shown can be obtained. As can be observed from FIG. 3, the depth map O is considered to reproduce the depth of each object in the measurement target region better than both the ToF data T and the depth map P. In FIG.

Here, the depth estimation device 100 may be implemented by a computing device such as a smart phone, tablet, or personal computer, and may have a hardware configuration as shown in FIG. 4, for example. That is, the depth estimation device 100 comprises a storage device 101, a processor 102, a user interface (UI) device 103 and a communication device 104 interconnected via a bus B. FIG.

Programs or instructions for realizing various functions and processes described later in the depth estimation device 100 may be downloaded from any external device via a network or the like, or stored in a CD-ROM (Compact Disk-Read Only Memory), flash It may be provided from a removable storage medium such as a memory.

The storage device 101 is implemented by one or more non-transitory storage mediums such as random access memory, flash memory, hard disk drive, etc., and is capable of executing programs or instructions together with installed programs or instructions. Stores files, data, etc. used for

The processor 102 includes one or more CPUs (Central Processing Units), GPUs (Graphics Pr
processing unit), processing circuit, or the like. The processor 102 executes various functions and processes of the depth estimation device 100 described later according to data such as a program stored in the storage device 101, instructions, and parameters necessary for executing the program or instructions.

The user interface (UI) device 103 may be composed of an input device such as a keyboard, mouse, camera, and microphone, an output device such as a display, speaker, headset, and printer, and an input/output device such as a touch panel. An interface with the estimating device 100 is implemented. For example, the user operates the depth estimation apparatus 100 by operating a GUI (Graphical User Interface) displayed on a display or touch panel with a keyboard, mouse, or the like.

The communication device 104 is implemented by various communication circuits that execute communication processing with a communication network such as an external device, the Internet, and a LAN (Local Area Network).

However, the hardware configuration described above is merely an example, and the depth estimation device 100 according to the present disclosure may be implemented with any other appropriate hardware configuration. For example, some or all of camera 20 , ToF sensor 30 and preprocessing device 40 may be incorporated into depth measuring device 100 .

[Depth estimation device]
Next, with reference to FIG. 5, a depth estimation device 100 according to one embodiment of the present disclosure will be described. FIG. 5 is a block diagram showing the functional configuration of the depth estimation device 100 according to one embodiment of the present disclosure.

As shown in FIG. 5, depth estimation apparatus 100 has acquisition section 110 and derivation section 120 .

Acquisition unit 110 acquires a first depth map acquired by a ranging sensor for a measurement target area and a second depth map inferred from an image of the measurement target area by a trained inference engine. That is, the acquisition unit 110 acquires the ToF data T and the depth map P from the preprocessing device 40 and transfers them to the derivation unit 120 .

Here, the ToF data T may be data preprocessed by the preprocessing device 40 with respect to the detection result of the ToF sensor 30 . For example, the preprocessing may be an opening process for removing noise, a correction process for a background portion, or the like.

Further, the depth map P may be data obtained by resizing the inference result of the inference engine 41 for the RGB image captured by the camera 20 so as to match the size of the ToF data T. FIG. For example, the ToF data T and depth map P may be resized into two-dimensional data with a width of 224 pixels and a height of 168 pixels.

A derivation unit 120 derives a third depth map from the first depth map and the second depth map according to a cost function. where the cost function is
a first constraint to bring the distance value of the pixel of interest in the third depth map closer to the distance value of the first depth map when a distance value corresponding to the pixel of interest exists in the first depth map;
a second constraint for bringing the distance value of the pixel of interest in the third depth map closer to the distance value of the second depth map when the distance value corresponding to the pixel of interest does not exist in the first depth map;
a third constraint for bringing the distance value of the pixel of interest closer to the distance values of neighboring pixels of the pixel of interest;
including.

として定式化されうる。ここで、ｘは深度マップＯであり、ＴはＴｏＦデータＴであり、Ｐは深度マップＰであり、ＩはＲＧＢ画像である。また、ｗ_０，ｗ_１，ε，Ｍはパラメータであり、∇は勾配を求める演算子である。導出部１２０は、式（１）を最小化する深度マップｘを求め、これを深度マップＯとする。 Specifically, the derivation unit 120 synthesizes the ToF data T and the depth map P according to the cost function including the first to third constraints to generate the synthesized depth map O. FIG. In one embodiment, the cost function is

can be formulated as where x is the depth map O, T is the ToF data T, P is the depth map P, and I is the RGB image. Also, w ₀ , w ₁ , ε and M are parameters, and ∇ is an operator for obtaining a gradient. The deriving unit 120 obtains the depth map x that minimizes the expression (1), and sets this as the depth map O. FIG.

は、第１の制約に関するものであり、ＴｏＦデータＴに距離値が存在する画素については、最終出力ｘの当該画素がＴｏＦデータＴの距離値に一致することを要請するものである。 Here, the first term on the right side of equation (1)

relates to the first constraint, which requires that, for pixels for which the ToF data T has a distance value, the pixel in the final output x matches the ToF data T distance value.

は、第２の制約に関するものであり、ＴｏＦデータＴに距離値が欠損している画素については、最終出力ｘの当該画素が深度マップＰの距離値に一致することを要請するものである。 Also, the second term on the right side of equation (1)

relates to the second constraint, which requires that pixels in the final output x whose distance values are missing in the ToF data T match the distance values in the depth map P. FIG.

は、第３の制約に関するものであり、分子は、最終出力ｘの注目画素と当該注目画素の隣接画素との距離値が近くなること、すなわち、平滑化を要請するものである。なお、分母は、撮像された被写体と背景部分との間のエッジ領域における分子の平滑化の効果を弱めるためのものである。 And the third term on the right side of equation (1)

is related to the third constraint, and the numerator requests that the distance values between the pixel of interest in the final output x and the neighboring pixels of the pixel of interest be close, that is, smoothing is required. Note that the denominator is for weakening the smoothing effect of the numerator in the edge region between the imaged subject and the background portion.

The parameters w ₀ and w ₁ are positive weights to balance the influence of the three terms (in particular, w ₀ is set to less than 1 to emphasize the ToF data T over the inferred depth map P). may be used). Also, the parameter M is a positive weight that defines how much the smoothing effect is weakened in edge regions. Furthermore, the parameter ε is a small positive constant to avoid division by zero. Note that (∇I) ² can be defined as obtaining differential images in each of the RGB channels and averaging them in the channel direction.

とすると、Ｇはｘに依存しないため、予め計算可能である。このとき、式（１）のコスト関数は、以下のように書き換えることができる。

式（２）は１次式の２乗和の形式を有するため、Ｅ（ｘ）を最小にするｘを線形方程式の最小二乗解として以下のように厳密に求めることができる。 The derivation unit 120 can obtain x that minimizes the expression (1) as follows. For simplicity of explanation,

Then G does not depend on x and can be calculated in advance. At this time, the cost function of Equation (1) can be rewritten as follows.

Since equation (2) has the form of a sum of squares of linear equations, x that minimizes E(x) can be strictly determined as a least-squares solution of a linear equation as follows.

が成り立てばよい。ただし、

であり、画素（ｉ，ｊ）が２次元データの右端又は下端にあって、ｘ_{ｉ，ｊ＋１}又はｘ_{ｉ＋１，ｊ}が定義されない状況では、その未定義変数が出現する１次の項はゼロとされる。 Now consider the condition of x that satisfies E(x)=0. This is true if the linearity of each term in E(x) is zero. Therefore, for any pixel (i,j),

should be established. however,

, and in a situation where pixel (i,j) is at the right end or bottom end of two-dimensional data and x _i,j+1 or x _i+1,j is not defined, the first-order term in which the undefined variable appears is zero. be done.

の行列表現として表すことができる。ここで、ｘの画素は、ラスタスキャン順に１次元配列される。この線形方程式は、変数の数よりも条件式の数が多く、ｏｖｅｒ－ｄｅｔｅｒｍｉｎｅｄな系となっており、厳密解は存在せず、最小二乗解を求めることが妥当である
。この最小二乗解は、Ｅ（ｘ）を最小（ゼロ）にする厳密解に一致する。従って、導出部１２０は、式（４）の最小二乗解を求めることによって、コスト関数Ｅ（ｘ）を最小にするｘを求めることができる。 Formula (3) is

can be expressed as a matrix representation of Here, the pixels of x are one-dimensionally arranged in raster scan order. Since this linear equation has more conditional expressions than variables, it is an over-determined system, there is no exact solution, and it is appropriate to obtain a least-squares solution. This least-squares solution corresponds to the exact solution that minimizes (zero) E(x). Therefore, the derivation unit 120 can obtain x that minimizes the cost function E(x) by obtaining the least-squares solution of Equation (4).

ここで、最終出力ｘの距離値は未定である。また、Ｔ_１，２及びＴ_２，１がブランクになっているが、これは、当該画素の距離値が欠損していることを意味する。 As a specific example, consider the following ToF data T, depth map P, coefficient G, and final output x.

Here, the distance value of the final output x is undecided. Also, T _1,2 and T _2,1 are blank, which means that the distance value for that pixel is missing.

となる。また、ｗ_０＝０．０１と設定されている場合、第２項は、

となる。さらに、第３項は、

となる。 For these inputs, the first term on the right side of the cost function of equation (2) is

becomes. Also, when w ₀ =0.01, the second term is

becomes. Furthermore, Section 3

becomes.

すなわち、コスト関数Ｅ（ｘ）は、ｘに関する１次式の２乗和になっていることがわかる。このため、以下の線形方程式の最小二乗解によってコストを最小化する厳密解ｘを導出することができる。

Putting these together, the cost function E(x) is as follows.

That is, it can be seen that the cost function E(x) is the sum of squares of linear expressions regarding x. Therefore, the exact solution x that minimizes the cost can be derived by the least-squares solution of the following linear equation.

The least-squares solution of equation (5) can be easily obtained by singular value decomposition or the like. In this way, the derivation unit 120 can derive x that minimizes the cost function E(x) in a reasonable computation time by obtaining the least-squares solution of equation (5), and the ToF data T and A depth map O synthesized from the depth map P can be obtained.

[Depth estimation processing]
Next, with reference to FIG. 6, depth estimation processing according to one embodiment of the present disclosure will be described. The depth estimation process is executed by the depth estimation device 100 described above. More specifically, one or more processors 102 of the depth estimation device 100 execute one or more programs or It may be realized by executing instructions. For example, the depth estimation process can be started by the user of the depth estimation apparatus 100 activating an application or the like related to the process.

FIG. 6 is a flowchart illustrating depth estimation processing according to one embodiment of the present disclosure.

As shown in FIG. 6, in step S101, the depth estimation apparatus 100 obtains a depth map P and ToF data T inferred from the RGB image I of the measurement target area. Specifically, the camera 20 captures an image of the measurement target area to obtain an RGB image I, the ToF sensor 30 measures the measurement target area, and the distance between the ToF sensor 30 and each object in the measurement target area is Obtain ToF data indicating distance.

Next, the preprocessing device 40 preprocesses the acquired ToF data and acquires ToF data T. FIG. The preprocessing device 40 also generates a depth map P from the RGB image I using the inference engine 41 . For example, the ToF data T may be obtained by performing opening processing, correction processing, etc. on the ToF data obtained from the ToF sensor 30 . Also, the depth map P may be obtained by performing a resizing process on the inference result of the inference engine 41 so as to match the size of the ToF data T. FIG.

The ToF data T and depth map P obtained in this way are provided to the depth estimation device 100 .

In step S102, the depth estimation apparatus 100 synthesizes the ToF data T and the depth map P according to the cost function, and derives the synthesized depth map O. FIG. For example, the cost function is
(Constraint 1) When the ToF data T has a distance value corresponding to the pixel of interest, the distance value of the pixel of interest in the depth map O is brought close to the ToF data T distance value.
(Constraint 2) When the ToF data T does not have a distance value corresponding to the pixel of interest, the distance value of the pixel of interest in the depth map O is brought closer to the distance value of the depth map P.
(Constraint 3) Bring the distance value of the pixel of interest close to the distance value of the neighboring pixels of the pixel of interest.
It may include the three constraints of

として定式化されてもよい。ここで、ｘは深度マップＯであり、ＴはＴｏＦデータＴであり、Ｐは深度マップＰであり、ＩはＲＧＢ画像である。また、ｗ_０，ｗ_１，ε，Ｍはパラメータであり、∇は勾配を求める演算子である。深度推定装置１００は、コスト関数Ｅ（ｘ）を最小化する深度マップｘを深度マップＯとする。ここで、コスト関数Ｅ（ｘ）を最小化する深度マップｘは、Ｅ（ｘ）＝０とした場合に得られる線形方程式の最小二乗解として求めることができる。 Specifically, the cost function is

may be formulated as where x is the depth map O, T is the ToF data T, P is the depth map P, and I is the RGB image. Also, w ₀ , w ₁ , ε and M are parameters, and ∇ is an operator for obtaining a gradient. The depth estimation apparatus 100 uses the depth map O as the depth map x that minimizes the cost function E(x). Here, the depth map x that minimizes the cost function E(x) can be obtained as a least-squares solution of a linear equation obtained when E(x)=0.

According to the embodiment described above, the depth estimation device 100
(Constraint 1) When the depth map T has a distance value corresponding to the pixel of interest, the distance value of the pixel of interest in the depth map O is brought closer to the distance value of the depth map T.
(Constraint 2) When the depth map T does not have a distance value corresponding to the pixel of interest, the distance value of the pixel of interest in the depth map O is brought closer to the distance value in the depth map P.
(Constraint 3) Bring the distance value of the pixel of interest close to the distance value of the neighboring pixels of the pixel of interest.
Synthesize the depth map T obtained from the ranging sensor and the depth map P inferred from the image from the imaging device to construct a synthesized depth map O . Thereby, it is possible to obtain a depth map O that has high accuracy globally and is smoothed between adjacent pixels.

[Other Examples]
In the above embodiment, a cost function containing three constraints is used to synthesize the depth map T obtained from the ranging sensor and the depth map P inferred from the image from the imaging device, and the synthesized depth map The manner in which O is constructed has been described. However, the depth maps to be synthesized need not be the above combinations. As an example, depth estimation apparatus 100 may generate depth map P based on images captured by a stereo camera, rather than using a model trained on images from an imaging device to infer depth map P. It is known that two or more images captured by a stereo camera can acquire distance (depth) information from the camera to the subject using the principle of triangulation based on parallax between viewpoints (image capturing points). . In this embodiment, as shown in FIG. 7, the depth estimation device 100 generates a depth map P having a distance value for each pixel based on images captured by a stereo camera. The generated depth map P is combined with the depth map T acquired from the ranging sensor using the cost function including the three constraints described above, and finally the combined depth map O is output.

As mentioned above, the third term on the right hand side of equation (1) simply requests that the distance value at a pixel be closer to the distance values at its neighbors. That is, there is a demand for smoothing the distance value between pixels. According to one example, by finding the distance value that minimizes the cost function of equation (1), the distance values of adjacent pixels are smoothed while suppressing large deviations from the distance values of the input depth map. Here, smoothing means, for example, smoothing a plurality of distance values included in one object. The overall distance value may be smoothed by decreasing the difference between the distance values of some two adjacent pixels and increasing the difference between the distance values of another two adjacent pixels. Therefore, smoothing does not necessarily reduce the difference in distance values at all neighboring pixels. According to one example, the smoothing process is a process that reduces the average difference between adjacent distance values (distance values in two adjacent pixels). According to one example, when focusing on an object, the average difference of adjacent distance values contained in the object is reduced. According to another example, when looking at the entire depth map, the average difference between neighboring distance values contained in the entire depth map is reduced. According to some examples, a distance value at a pixel of the output depth map is different from any distance value at corresponding pixels of the plurality of depth maps.

Now, consider a case where a certain pixel (hereinafter referred to as the first pixel) and its adjacent pixel (hereinafter referred to as the second pixel) are both associated with one subject. In this case, the following (A), (B) and (C) are established.
(A) If both the distance value of the first pixel and the distance value of the second pixel are obtained from the depth map P, the difference between these distance values is small, and the third term on the right hand side of equation (1) calculates them by The smoothing effect is small.
(B) If both the distance value of the first pixel and the distance value of the second pixel are obtained from the depth map T, the difference between these distance values is small, and the third term on the right side of equation (1) can be used to convert them to The smoothing effect is small.
(C) However, if one of the distance value for the first pixel and the distance value for the second pixel is obtained from the depth map P and the other from the depth map T, the difference between these distance values is relatively Since it can be large, in that case the effect of smoothing by the third term on the right side of equation (1) can be large.

For the one subject, if the depth map T is simply used as a base and the missing portion of the depth map T is supplemented with the depth map P, the average difference between adjacent distance values becomes a large value. The same is true for the entire depth map. If the average difference between adjacent distance values is large, the boundaries between pixels will stand out unnaturally.

On the other hand, in the present embodiment, since the distance value is smoothed as described above, the average difference between adjacent distance values for one subject or the entire depth map becomes small. According to one example, the aforementioned distance value smoothing process is performed for each of the plurality of objects in the synthesized depth map. Note that the average difference between adjacent distance values can be reduced by a calculation method other than Equation (1).

By the way, when the first pixel relates to a subject and the second pixel relates to a background portion, the first pixel and the second pixel form an edge region. In this case, there is a significant difference between the distance value of the first pixel and the distance value of the second pixel. And the difference between the object distance value and the background part distance value should be maintained. Therefore, in the edge region, the effect of smoothing the distance value is weakened by the denominator of the third term of Equation (1). As a result, the difference in distance values in the edge region can be properly maintained without being excessively small.

According to one example, the degree of smoothing of the distance value in the object and the degree of smoothing of the distance value in the edge region can be matched in order to simplify the arithmetic processing. In this case, the denominator in the third term on the right side of Equation (1) can be omitted. In this case, for example, by setting w1 in the third term on the right side of Equation (1) to a relatively small value, excessive smoothing of the distance value in the edge region can be prevented. According to another example, as shown in equation (1), the effect of smoothing the distance value can be increased in the subject and less effective in the edge region.

The name depth estimator 100 includes the word "estimate" because it estimates the distance values by smoothing them as described above, rather than simply combining two depth maps. The depth estimation device 100 can be rephrased as a depth map synthesis device or a depth map generation device.

As depth maps to be input to the depth estimation device 100, a depth map P obtained by reasoning from an RGB image and a depth map T obtained by a distance measuring sensor are exemplified. According to another example, another depth map can be input to depth measuring device 100 . A non-limiting variation of the input depth map is shown below.

[Example 1]
For example, the depth map S obtained by stereo matching and the depth map W inferred from the wide-angle camera image can be input to the depth estimation device 100 . Stereo matching can calculate relatively accurate distance values except for occlusion areas. However, in stereo matching, only a distance value (depth) equivalent to that of a telephoto camera can be calculated, and a distance value near the edge of an image in a wide-angle camera cannot be calculated. On the other hand, according to the single camera depth estimation of the wide-angle camera, the distance value can be estimated in the entire image. That is, the number of effective pixels of the depth map S is smaller than the number of effective pixels of the depth map W. Therefore, the depth estimation apparatus 100 can create the depth map O by synthesizing the depth map S and the depth map W according to a cost function including the following three constraints.
(Constraint 1) When the depth map S has a distance value corresponding to the pixel of interest, the distance value of the pixel of interest in the depth map O is brought closer to the distance value of the depth map S.
(Constraint 2) When the depth map S does not have a distance value corresponding to the pixel of interest, the distance value of the pixel of interest in the depth map O is brought closer to the distance value in the depth map W.
(Constraint 3) Make the distance value of the pixel of interest closer to the distance value of the pixel adjacent to the pixel of interest.
In this way, a depth map O can be generated that has distance values for the entire surface, that is, for all pixels. According to one example, a combination of a wide-angle camera and a telephoto camera can provide the imaging data necessary for generating this depth map O. FIG. According to another example, the depth map can be synthesized in another way. For example, if the depth map S has a distance value, that distance value is used, and for a portion where the depth map S does not have a distance value, the distance value of the depth map W is used. good. According to one example, the distance values for the central portion of the output depth map can be taken from the depth map S, and the distance values for the peripheral portion surrounding said central portion of the output depth map can be taken from the depth map W.

[Example 2]
In Example 2, in addition to the depth map S and depth map W of Example 1, a depth map T obtained by a distance measuring sensor such as a ToF sensor is used as an input depth map. In other words, three depth maps are input to the depth estimation device 100 . Then, the depth estimation apparatus 100 synthesizes the three input depth maps based on the reliability of the input depth maps. According to an example, depth estimation apparatus 100 can synthesize depth map O based on a cost function that includes the following four constraints.
(Constraint 1) When the depth map T has a distance value corresponding to the pixel of interest, the distance value of the pixel of interest in the depth map O is brought closer to the distance value of the depth map T.
(Constraint 2) When the depth map T does not have a distance value corresponding to the pixel of interest, the distance value of the pixel of interest in the depth map O is brought closer to the distance value in the depth map S.
(Constraint 3) If the distance value corresponding to the pixel of interest does not exist in either the depth map T or the depth map S, the distance value of the pixel of interest in the depth map O is brought closer to the distance value in the depth map W.
(Constraint 4) Make the distance value of the pixel of interest closer to the distance value of the pixel adjacent to the pixel of interest.
In this example, it is assumed that the distance values of depth map T are highly reliable, the distance values of depth map S are next most reliable, and the distance values of depth map W are less reliable. Thus, depth map O can be generated by synthesizing a plurality of depth maps based on the reliability of the depth maps. According to another example, another input depth map can be employed. According to another example, the depth map can be synthesized in another way. For example, if the depth map T has a distance value, that distance value is used, and for a portion where the depth map T does not have a distance value, the distance value of the depth map S is used. An output depth map may be synthesized by using the distance values of the depth map W for a portion where no distance value exists.

[Example 3]
As input depth maps, a depth map D obtained by dual-camera parallax estimation and a depth map T obtained by a ranging sensor such as a ToF sensor can be input to the depth estimation device 100 . In dual-camera parallax estimation, occlusion areas cannot be matched, so they are generally invalid areas. When the distance values of the invalid area are filled based on the distance values around the invalid area, the invalid area becomes an area with low reliability of the distance value. Also, if there is a repeating pattern area or an area without texture, multiple points may be matched from one point, and the accuracy of the obtained distance value may drop.
That is, the depth map D is
- Region D1: (low confidence) occlusion region,
- Region D2: (low reliability) repeated pattern region or textureless region (flat region),
- Area D3: Areas other than (highly reliable) areas D1 and D2,
can be classified into
On the other hand, the depth map T obtained by a ranging sensor such as a ToF sensor is
Region T1: occlusion region caused by image alignment (low confidence);
・Area T2: (Reliability is low) A place where infrared rays do not reach (for example, a distant view),
- Region T3: (low reliability) repeated pattern region or textureless region;
• Region T4: (high reliability) includes regions other than regions T1, T2, and T3. If the direction of the reference image and the ToF or the reference image are different, the occlusion position will be in a different direction, so it can be said that they are in a complementary relationship.
In this way, the depth maps D and T each have a high-reliability region and a low-reliability region. Therefore, the depth estimation apparatus 100 can synthesize the depth map O based on a cost function including the following three constraints.
(Constraint 1) If the distance value corresponding to the pixel of interest exists in the region D3 or T4, the distance value of the pixel of interest in the depth map O is brought closer to one of these distance values.
(Constraint 2) If the distance value corresponding to the pixel of interest does not exist in the region D3 or region T4, the distance value of the pixel of interest in the depth map O is brought closer to the distance value of any one of the regions D1, D2, T1, T2, T3. .
(Constraint 3) Make the distance value of the pixel of interest closer to the distance value of the pixel adjacent to the pixel of interest.
As a result, it is possible to preferentially reflect in the depth map O the regions of which the reliability of the plurality of input depth maps is high, thereby synthesizing a highly accurate depth map.

In Example 3 above, one input depth map is classified into two regions, one with a high degree of reliability of the distance value and the other with a low degree of reliability of the distance value. According to another example, an input depth map can be divided into three or more regions with different degrees of confidence. For example, the depth map D is divided into a region D3 with a higher reliability, a region D2 with a lower reliability than the region D3, and a region D1 with a lower reliability than the region D2. For example, the depth map T is divided into a region T4 with a higher reliability, a region T3 with a lower reliability than the region T4, a region T2 with a lower reliability than the region T3, and a region T1 with a lower reliability than the region T2. According to one example, for a depth map T acquired by a ranging sensor, such as a ToF sensor, a confidence map is obtained along with the distance values. For the depth map D, on the other hand, a confidence score is assigned to each region, for example in a preprocessor. For example, the depth estimation apparatus 100 receives, as an input depth map, data in which a distance value and a reliability score of the distance value are assigned to each pixel. When the depth maps D and T are used as input depth maps, according to one example, regions T4, D3, T3, T2, T1, D2, and D1 are in descending order of reliability. The depth map O is reflected in descending order of reliability by a method equivalent to the method using functions.
A higher quality depth map can be provided by assigning a reliability level to each region of the input depth map and preferentially reflecting areas with a high level of reliability in the output depth map.

REFERENCE SIGNS LIST 10 depth estimation system 20 camera 30 ToF sensor 40 preprocessing device 100 depth estimation device 110 acquisition unit 120 derivation unit

Claims (12)

obtaining multiple depth maps;
Compared to using the distance values contained in the plurality of depth maps as they are, the plurality of depth maps are combined while reducing an average difference in distance values of adjacent pixels to output one output depth map. A depth estimation method comprising: 2. The depth estimation method of claim 1, wherein distance values at pixels of the output depth map are different than distance values at corresponding pixels of the plurality of depth maps. the plurality of depth maps includes a first depth map and a second depth map;
obtaining the first depth map using a ranging sensor;
2. The method of claim 1, comprising obtaining the second depth map using an imaging device. the plurality of depth maps includes a first depth map and a second depth map;
the first depth map is a depth map in which distance values are missing in some pixels;
2. The depth estimation method of claim 1, wherein the second depth map is a depth map without missing distance values. obtaining a first depth map with stereo matching;
obtaining a second depth map obtained by inferring from the image;
Combining a plurality of depth maps including the first depth map and the second depth map to output an output depth map. 6. The depth estimation method according to claim 5, wherein the number of effective pixels of said first depth map is smaller than the number of effective pixels of said second depth map. 6. The depth estimation method of claim 5, wherein the images are taken with a wide-angle camera. 7. The depth of claim 6, wherein distance values for a central portion of said output depth map are taken from said first depth map and peripheral portions surrounding said central portion of said output depth map are taken from said second depth map. estimation method. obtaining a third depth map obtained by the ToF sensor;
6. The depth estimation method according to claim 5, wherein said output depth map is obtained by combining said first depth map, said second depth map and said third depth map. an acquisition unit that acquires a first depth map acquired by a ranging sensor for a measurement target area and a second depth map generated from an image of the measurement target area;
a derivation unit that derives a third depth map from the first depth map and the second depth map according to a cost function;
The cost function is
a first constraint for bringing the distance value of the pixel of interest in the third depth map closer to the distance value of the first depth map when a distance value corresponding to the pixel of interest exists in the first depth map; and,
a second depth map for bringing the distance value of the pixel of interest in the third depth map closer to the distance value of the second depth map when the distance value corresponding to the pixel of interest does not exist in the first depth map; constraints and
a third constraint for bringing the distance value of the pixel of interest closer to the distance values of neighboring pixels of the pixel of interest;
A depth estimator, comprising: 11. The depth estimation device according to claim 10, wherein the derivation unit derives the third depth map such that the value of the cost function is minimized. 11. Depth estimation apparatus according to claim 10, wherein the cost function is defined to weaken the third constraint at discontinuities of distance values in the first depth map or the second depth map. .

Images