JP7106665B2 - MONOCULAR DEPTH ESTIMATION METHOD AND DEVICE, DEVICE AND STORAGE MEDIUM THEREOF - Google Patents

Description

(Cross reference to related applications)
This application is filed based on and claims priority from Chinese Patent Application No. 201810496541.6, filed on May 22, 2018, the entire disclosure of which is incorporated herein by reference. .

TECHNICAL FIELD Embodiments of the present application relate to the field of artificial intelligence, and more particularly to a monocular depth estimation method and its apparatus, apparatus and storage medium.

Monocular depth estimation is an important problem in computer vision, and the specific task of monocular depth estimation is to predict the depth of each pixel point in an image. Among them, an image composed of depth values of each pixel point is also called a depth map. Monocular depth estimation has important implications for obstacle detection, 3D scene reconstruction, and 3D scene analysis in autonomous driving. Monocular depth estimation can also indirectly improve the performance of other computer vision tasks such as object detection, target tracking and target identification.

The current problem is that a large amount of labeled data is required to train a neural network for monocular depth estimation, but the cost of obtaining labeled data is high. Although labeled data can be obtained by laser radar in outdoor environments, the obtained labeled data is very sparse, and monocular depth estimation networks trained using such labeled data have small edges without clear edges. Inability to capture accurate depth information for objects.

Embodiments of the present application provide a monocular depth estimation method and its apparatus, apparatus and storage medium.

The technical solutions of the embodiments of the present application are implemented as follows.

An embodiment of the present application comprises the steps of obtaining an image to be processed, and inputting the image to be processed into a trained monocular depth estimation network model to obtain an analysis result of the image to be processed, wherein the the monocular depth estimation network model is obtained by supervised training with the disparity map output by the first binocular matching neural network model; outputting the analysis result of the image to be processed; To provide a monocular depth estimation method comprising:

Embodiments of the present application include an acquisition module configured to acquire an image to be processed, and inputting the image to be processed into a trained monocular depth estimation network model to obtain an analysis result of the image to be processed. wherein the monocular depth estimation network model is obtained by supervised training with a disparity map output by a first binocular matching neural network model; and an output module configured to output analysis results of an image to be processed.

An embodiment of the present application is a monocular depth estimation device comprising a processor and a memory storing a computer program operable in the processor, wherein the processor performs the monocular depth estimation method provided by the embodiment of the present application when executing the program. provides a monocular depth estimation device that implements the steps in

An embodiment of the present application is a computer readable storage medium storing a computer program, which when executed by a processor, implements the steps in the monocular depth estimation method provided by the embodiment of the present application. Provide storage media.

を利用して前記損失関数を決定するステップを含み、
ここで、前記
（化８８）

は損失関数を表し、前記
（化８９）

は再構成誤差を表し、前記
（化９０）

は前記第一両眼マッチングネットワークモデルにより出力される視差マップが前記訓練後の第二両眼マッチングネットワークモデルにより出力される視差マップに比べて偏りが小さいことを表し、前記
（化９１）

は前記第一両眼マッチングネットワークモデルを制約する出力勾配が前記訓練後の第二両眼マッチングネットワークモデルの出力勾配に一致することを表し、前記
（化９２）

は強度係数を表す項目７に記載の方法。
（項目９）
さらに、式
（化９３）

、または、
（化９４）

を利用して前記再構成誤差を決定するステップを含み、
ここで、前記
（化９５）

は画像における画素の数を表し、前記
（化９６）

は前記訓練後の第二両眼マッチングネットワークモデルにより出力される遮蔽マップの画素値を表し、前記
（化９７）

は深度ラベルなしの実両眼データのうちの左画像の画素値を表し、前記
（化９８）

は深度ラベルなしの実両眼データのうちの右画像の画素値を表し、前記
（化９９）

は右画像をサンプリングしてから合成した画像の画素値を表し、前記
（化１００）

は左画像をサンプリングしてから合成した画像の画素値を表し、前記
（化１０１）

は深度ラベルなしの実両眼データのうちの左画像の第一両眼マッチングネットワークモデルによって出力された視差マップの画素値を表し、前記
（化１０２）

は深度ラベルなしの実両眼データのうちの右画像の第一両眼マッチングネットワークモデルによって出力された視差マップの画素値を表し、前記
（化１０３）

は画素点の画素座標を表す項目８に記載の方法。
（項目１０）
さらに、式
（化１０４）

、または、
（化１０５）

を利用して前記第一両眼マッチングネットワークモデルにより出力される視差マップが前記訓練後の第二両眼マッチングネットワークモデルにより出力される視差マップに比べて偏りが小さいことを決定するステップを含み、
ここで、前記
（化１０６）

は画像における画素の数を表し、前記
（化１０７）

は前記訓練後の第二両眼マッチングネットワークモデルにより出力される遮蔽マップの画素値を表し、前記
（化１０８）

は深度ラベルなしの実両眼データのうちの左画像の第一両眼マッチングネットワークモデルによって出力された視差マップの画素値を表し、前記
（化１０９）

は深度ラベルなしの実両眼データのうちの右画像の第一両眼マッチングネットワークモデルによって出力された視差マップの画素値を表し、前記
（化１１０）

は左画像の訓練後の第二両眼マッチングネットワークモデルによって出力された視差マップの画素値を表し、前記
（化１１１）

は右画像の訓練後の第二両眼マッチングネットワークモデルによって出力された視差マップの画素値を表し、前記
（化１１２）

は画素点の画素座標を表し、前記
（化１１３）

は強度係数を表す項目８に記載の方法。
（項目１１）
さらに、式
（化１１４）

、または、
（化１１５）

を利用して前記第一両眼マッチングネットワークモデルの出力勾配が前記第二両眼マッチングネットワークモデルの出力勾配に一致することを決定するステップを含み、
ここで、前記
（化１１６）

は画像における画素の数を表し、前記
（化１１７）

は深度ラベルなしの実両眼データのうちの左画像の第一両眼マッチングネットワークモデルによって出力された視差マップの勾配を表し、前記
（化１１８）

は深度ラベルなしの実両眼データのうちの右画像の第一両眼マッチングネットワークモデルによって出力された視差マップの勾配を表し、前記
（化１１９）

は左画像の訓練後の第二両眼マッチングネットワークモデルによって出力された視差マップの勾配を表し、前記
（化１２０）

は右画像の訓練後の第二両眼マッチングネットワークモデルによって出力された視差マップの勾配を表し、前記
（化１２１）

は画素点の画素座標を表す項目８に記載の方法。
（項目１２）
前記深度ラベル付きの実両眼データは左画像および右画像を含み、それに対して、前記単眼深度推定ネットワークモデルの訓練プロセスは、
前記深度ラベル付きの実両眼データのうちの左画像または右画像を訓練サンプルとして取得するステップと、
前記深度ラベル付きの実両眼データのうちの左画像または右画像に基づいて単眼深度推定ネットワークモデルを訓練するステップと、を含む項目５に記載の方法。
（項目１３）
前記深度ラベルなしの実両眼データは左画像および右画像を含み、それに対して、前記単眼深度推定ネットワークモデルの訓練プロセスは、
前記深度ラベルなしの実両眼データを前記第一両眼マッチングニューラルネットワークモデルに入力し、対応する視差マップを得るステップと、
前記対応する視差マップ、前記深度ラベルなしの実両眼データを撮影するカメラのレンズ基線長および前記深度ラベルなしの実両眼データを撮影するカメラのレンズ焦点距離に基づき、前記視差マップの対応する深度マップを決定するステップと、
前記深度ラベルなしの実両眼データのうちの左画像または右画像をサンプルデータとし、前記視差マップの対応する深度マップに基づいて単眼深度推定ネットワークモデルを教示し、それにより前記単眼深度推定ネットワークモデルを訓練するステップと、を含む項目６から１１のいずれか一項に記載の方法。
（項目１４）
前記処理対象の画像の解析結果は前記単眼深度推定ネットワークモデルにより出力される視差マップを含み、それに対して、さらに、
前記単眼深度推定ネットワークモデルにより出力される視差マップ、前記単眼深度推定ネットワークモデルに入力される画像を撮影するカメラのレンズ基線長および前記単眼深度推定ネットワークモデルに入力される画像を撮影するカメラのレンズ焦点距離に基づき、前記視差マップの対応する深度マップを決定するステップと、
前記視差マップの対応する深度マップを出力するステップと、を含む項目１２または１３に記載の方法。
（項目１５）
処理対象の画像を取得するように構成された取得モジュールと、
前記処理対象の画像を訓練された単眼深度推定ネットワークモデルに入力し、前記処理対象の画像の解析結果を得るように構成された実行モジュールであって、前記単眼深度推定ネットワークモデルは、第一両眼マッチングニューラルネットワークモデルにより出力される視差マップによる教師あり訓練によって得られたものである、実行モジュールと、
前記処理対象の画像の解析結果を出力するように構成された出力モジュールと、を含む単眼深度推定装置。
（項目１６）
さらに、取得した合成サンプルデータに基づいて第二両眼マッチングニューラルネットワークモデルを訓練し、訓練後の第二両眼マッチングニューラルネットワークモデルを取得するように構成された第一訓練モジュールと、取得した実サンプルデータに基づいて訓練後の第二両眼マッチングニューラルネットワークモデルのパラメータを調整し、第一両眼マッチングニューラルネットワークモデルを得るように構成された第二訓練モジュールと、を含む項目１５に記載の装置。
（項目１７）
さらに、合成された左画像および合成された右画像を含む深度ラベル付きの合成された両眼画像を前記合成サンプルデータとして取得するように構成された第一取得モジュールを含む項目１６に記載の装置。
（項目１８）
前記第一訓練モジュールは、前記合成された両眼画像に基づいて第二両眼マッチングニューラルネットワークモデルを訓練し、出力が視差マップおよび遮蔽マップである訓練後の第二両眼マッチングニューラルネットワークモデルを得るように構成された第一訓練ユニットを含み、ここで、前記視差マップは前記左画像における各画素点と前記右画像における対応する画素点との、画素を単位とする視差距離を表現し、前記遮蔽マップは前記左画像における各画素点の前記右画像における対応する画素点が物体により遮蔽されているかどうかを表現する項目１７に記載の装置。
（項目１９）
前記第二訓練モジュールは、取得した深度ラベル付きの実両眼データに基づいて訓練後の第二両眼マッチングニューラルネットワークモデルの教師あり訓練を行い、それによって前記訓練後の第二両眼マッチングニューラルネットワークモデルの重みを調整し、第一両眼マッチングニューラルネットワークモデルを得るように構成された第二訓練ユニットを含む項目１６に記載の装置。
（項目２０）
前記第二訓練ユニットはさらに、取得した深度ラベルなしの実両眼データに基づいて訓練後の第二両眼マッチングニューラルネットワークモデルの教師なし訓練を行い、それによって前記訓練後の第二両眼マッチングニューラルネットワークモデルの重みを調整し、第一両眼マッチングニューラルネットワークモデルを得るように構成される項目１６に記載の装置。
（項目２１）
前記第二訓練ユニットは、損失関数を使用し、前記深度ラベルなしの実両眼データに基づいて訓練後の第二両眼マッチングニューラルネットワークモデルの教師なし訓練を行い、それによって前記訓練後の第二両眼マッチングニューラルネットワークモデルの重みを調整し、第一両眼マッチングニューラルネットワークモデルを得るように構成された第二訓練コンポーネントを含む項目２０に記載の装置。
（項目２２）
さらに、式
（化１２２）

を利用して前記損失関数を決定するように構成された第一決定モジュールを含み、ここで、前記
（化１２３）

は損失関数を表し、前記
（化１２４）

は再構成誤差を表し、前記
（化１２５）

は前記第一両眼マッチングネットワークモデルにより出力される視差マップが前記訓練後の第二両眼マッチングネットワークモデルにより出力される視差マップに比べて偏りが小さいことを表し、前記
（化１２６）

は前記第一両眼マッチングネットワークモデルを制約する出力勾配が前記訓練後の第二両眼マッチングネットワークモデルの出力勾配に一致することを表し、前記
（化１２７）

は強度係数を表す項目２１に記載の装置。
（項目２３）
さらに、式
（化１２８）

、または、
（化１２９）

を利用して前記再構成誤差を決定するように構成された第二決定モジュールを含み、ここで、前記
（化１３０）

は画像における画素の数を表し、前記
（化１３１）

は前記訓練後の第二両眼マッチングネットワークモデルにより出力される遮蔽マップの画素値を表し、前記
（化１３２）

は深度ラベルなしの実両眼データのうちの左画像の画素値を表し、前記
（化１３３）

は深度ラベルなしの実両眼データのうちの右画像の画素値を表し、前記
（化１３４）

は右画像をサンプリングしてから合成した画像の画素値を表し、前記
（化１３５）

は左画像をサンプリングしてから合成した画像の画素値を表し、前記
（化１３６）

は深度ラベルなしの実両眼データのうちの左画像の第一両眼マッチングネットワークモデルによって出力された視差マップの画素値を表し、前記
（化１３７）

は深度ラベルなしの実両眼データのうちの右画像の第一両眼マッチングネットワークモデルによって出力された視差マップの画素値を表し、前記
（化１３８）

は画素点の画素座標を表す項目２２に記載の装置。
（項目２４）
さらに、式
（化１３９）

、または、
（化１４０）

を利用して前記第一両眼マッチングネットワークモデルにより出力される視差マップが前記訓練後の第二両眼マッチングネットワークモデルにより出力される視差マップに比べて偏りが小さいことを決定するように構成された第三決定モジュールを含み、ここで、前記
（化１４１）

は画像における画素の数を表し、前記
（化１４２）

は前記訓練後の第二両眼マッチングネットワークモデルにより出力される遮蔽マップの画素値を表し、前記
（化１４３）

は深度ラベルなしの実両眼データのうちの左画像の第一両眼マッチングネットワークモデルによって出力された視差マップの画素値を表し、前記
（化１４４）

は深度ラベルなしの実両眼データのうちの右画像の第一両眼マッチングネットワークモデルによって出力された視差マップの画素値を表し、前記
（化１４５）

は左画像の訓練後の第二両眼マッチングネットワークモデルによって出力された視差マップの画素値を表し、前記
（化１４６）

は右画像の訓練後の第二両眼マッチングネットワークモデルによって出力された視差マップの画素値を表し、前記
（化１４７）

は画素点の画素座標を表し、前記
（化１４８）

は強度係数を表す項目２２に記載の装置。
（項目２５）
さらに、式
（化１４９）

、または、
（化１５０）

を利用して前記第一両眼マッチングネットワークモデルの出力勾配が前記第二両眼マッチングネットワークモデルの出力勾配に一致することを決定するように構成された第四決定モジュールを含み、ここで、前記
（化１５１）

は画像における画素の数を表し、前記
（化１５２）

は深度ラベルなしの実両眼データのうちの左画像の第一両眼マッチングネットワークモデルによって出力された視差マップの勾配を表し、前記
（化１５３）

は深度ラベルなしの実両眼データのうちの右画像の第一両眼マッチングネットワークモデルによって出力された視差マップの勾配を表し、前記
（化１５４）

は左画像の訓練後の第二両眼マッチングネットワークモデルによって出力された視差マップの勾配を表し、前記
（化１５５）

は右画像の訓練後の第二両眼マッチングネットワークモデルによって出力された視差マップの勾配を表し、前記
（化１５６）

は画素点の画素座標を表す項目２２に記載の装置。
（項目２６）
前記深度ラベル付きの実両眼データは左画像および右画像を含み、それに対して、さらに、前記深度ラベル付きの実両眼データのうちの左画像または右画像を訓練サンプルとして取得し、そして前記深度ラベル付きの実両眼データのうちの左画像または右画像に基づいて単眼深度推定ネットワークモデルを訓練するように構成された第三訓練モジュールを含む項目１９に記載の装置。
（項目２７）
前記深度ラベルなしの実両眼データは左画像および右画像を含み、それに対して、さらに、前記深度ラベルなしの実両眼データを前記第一両眼マッチングニューラルネットワークモデルに入力し、対応する視差マップを得て、前記対応する視差マップ、前記深度ラベルなしの実両眼データを撮影するカメラのレンズ基線長および前記深度ラベルなしの実両眼データを撮影するカメラのレンズ焦点距離に基づき、前記視差マップの対応する深度マップを決定し、そして前記深度ラベルなしの実両眼データのうちの左画像または右画像をサンプルデータとし、前記視差マップの対応する深度マップに基づいて単眼深度推定ネットワークモデルを教示し、それにより前記単眼深度推定ネットワークモデルを訓練するように構成された第三訓練モジュールを含む項目２０から２５のいずれか一項に記載の装置。
（項目２８）
前記処理対象の画像の解析結果は前記単眼深度推定ネットワークモデルにより出力される視差マップを含み、それに対して、さらに、前記単眼深度推定ネットワークモデルにより出力される視差マップ、前記単眼深度推定ネットワークモデルに入力される画像を撮影するカメラのレンズ基線長および前記単眼深度推定ネットワークモデルに入力される画像を撮影するカメラのレンズ焦点距離に基づき、前記視差マップの対応する深度マップを決定するように構成された第五決定モジュールと、前記視差マップの対応する深度マップを出力するように構成された第一出力モジュールと、を含む項目２６または２７に記載の装置。
（項目２９）
プロセッサおよびプロセッサにおいて運用可能なコンピュータプログラムが記憶されたメモリを含む単眼深度推定機器であって、前記プロセッサは前記プログラムを実行する時に項目１から１４のいずれか一項に記載の単眼深度推定方法におけるステップを実現する単眼深度推定機器。
（項目３０）
コンピュータプログラムが記憶されたコンピュータ読み取り可能記憶媒体であって、該コンピュータプログラムはプロセッサにより実行される時に項目１から１４のいずれか一項に記載の単眼深度推定方法におけるステップを実現するコンピュータ読み取り可能記憶媒体。 In an embodiment of the present application, an image to be processed is acquired, and the image to be processed is input to a monocular depth estimation network model obtained by supervised training with a disparity map output by a first binocular matching neural network model. and obtaining an analysis result of the image to be processed, and outputting an analysis result of the image to be processed to perform monocular depth estimation using less or no depth map labeled data. The network can be trained and provides a more efficient, unsupervised, fine-tunable binocular parallax-based network method, thereby indirectly improving the effectiveness of monocular depth estimation.
For example, the present application provides the following items.
(Item 1)
obtaining an image to be processed;
inputting the image to be processed into a trained monocular depth estimation network model to obtain an analysis result of the image to be processed, wherein the monocular depth estimation network model is processed by a first binocular matching neural network model: a step obtained by supervised training with the output disparity map;
and outputting an analysis result of the image to be processed.
(Item 2)
The training process of the first binocular matching neural network model includes:
training a second binocular matching neural network model based on the obtained synthetic sample data to obtain a second binocular matching neural network model after training;
adjusting parameters of the second trained binocular matching neural network model based on the acquired real sample data to obtain a first binocular matching neural network model.
(Item 3)
3. The method of item 2, further comprising obtaining a depth-labeled synthesized binocular image including a synthesized left image and a synthesized right image as the synthesized sample data.
(Item 4)
The step of training a second binocular matching neural network model based on the obtained synthetic sample data comprises:
training a second binocular matching neural network model based on the combined binocular image to obtain a trained second binocular matching neural network model whose output is a disparity map and an occlusion map, wherein , the parallax map expresses a parallax distance in pixels between each pixel point in the left image and the corresponding pixel point in the right image, and the occlusion map expresses a pixel point in the right image for each pixel point in the left image; 4. The method of item 3 of expressing whether the corresponding pixel point in is occluded by an object.
(Item 5)
The step of adjusting the parameters of the second binocular matching neural network model after training based on the acquired real sample data to obtain a first binocular matching neural network model, comprising:
supervised training a second post-trained binocular matching neural network model based on the acquired depth-labeled real binocular data, thereby adjusting the weights of the post-trained second binocular matching neural network model. , obtaining a first binocular matching neural network model.
(Item 6)
The step of adjusting parameters of a second binocular matching neural network model after training based on the acquired real sample data to obtain a first binocular matching neural network model, further comprising:
unsupervised training of a second post-trained binocular matching neural network model based on the acquired real binocular data without depth labels, thereby adjusting the weights of the post-trained second binocular matching neural network model. , obtaining a first binocular matching neural network model.
(Item 7)
unsupervised training of a post-trained second binocular matching neural network model based on the acquired real binocular data without depth labels, thereby adjusting the weights of the post-trained second binocular matching neural network model. and obtaining a first binocular matching neural network model comprising:
unsupervised training of a post-trained second binocular matching neural network model based on said unlabeled real binocular data using a loss function, thereby said post-trained second binocular matching neural network model A method according to item 6, comprising adjusting the weights of , to obtain a first binocular matching neural network model.
(Item 8)
Furthermore, the expression
(Chemical 87)

determining the loss function using
where
(Chem. 88)

represents the loss function, and
(Chemical 89)

represents the reconstruction error, and
(Chemical 90)

represents that the disparity map output by the first binocular matching network model is less biased than the disparity map output by the second binocular matching network model after training, and
(Chemical 91)

represents that the output gradient constraining the first binocular matching network model matches the output gradient of the trained second binocular matching network model, and
(Chemical 92)

8. A method according to item 7, wherein is the strength factor.
(Item 9)
Furthermore, the expression
(Chemical 93)

,or,
(Chemical 94)

determining the reconstruction error using
where
(Chemical 95)

represents the number of pixels in the image, and
(Chemical 96)

represents the pixel values of the occlusion map output by the second binocular matching network model after training, and
(Chemical 97)

represents the pixel value of the left image of the real binocular data without a depth label, and
(Chemical 98)

represents the pixel value of the right image of the real binocular data without a depth label, and
(Chemical 99)

represents the pixel value of the synthesized image after sampling the right image, and
(Chem. 100)

represents the pixel value of the synthesized image after sampling the left image, and
(Chem. 101)

represents the pixel values of the disparity map output by the first binocular matching network model of the left image of the real binocular data without depth labels, and
(Chem. 102)

represents the pixel values of the disparity map output by the first binocular matching network model of the right image of the real binocular data without depth labels, and
(Chem. 103)

9. A method according to item 8, wherein is the pixel coordinate of the pixel point.
(Item 10)
Furthermore, the expression
(Chem. 104)

,or,
(Chem. 105)

determining that the disparity map output by the first binocular matching network model is less biased than the disparity map output by the trained second binocular matching network model using
where
(Chem. 106)

represents the number of pixels in the image, and
(Chem. 107)

represents the pixel values of the occlusion map output by the second binocular matching network model after training, and
(Chem. 108)

represents the pixel values of the disparity map output by the first binocular matching network model of the left image of the real binocular data without depth labels, and
(Chem. 109)

represents the pixel values of the disparity map output by the first binocular matching network model of the right image of the real binocular data without depth labels, and
(Chem. 110)

represents the pixel values of the disparity map output by the second binocular matching network model after training of the left image, and
(Chem. 111)

represents the pixel values of the disparity map output by the second binocular matching network model after training for the right image, and
(Chem. 112)

represents the pixel coordinates of the pixel point, and
(Chem. 113)

9. A method according to item 8, wherein is the strength factor.
(Item 11)
Furthermore, the expression
(Chem. 114)

,or,
(Chem. 115)

determining that the output gradient of the first binocular matching network model matches the output gradient of the second binocular matching network model using
where
(Chem. 116)

represents the number of pixels in the image, and
(Chem. 117)

represents the gradient of the disparity map output by the first binocular matching network model of the left image of real binocular data without depth labels, and
(Chem. 118)

represents the gradient of the disparity map output by the first binocular matching network model of the right image of real binocular data without depth labels, and
(Chem. 119)

represents the gradient of the disparity map output by the second binocular matching network model after training of the left image, said
(Chem. 120)

represents the gradient of the disparity map output by the second binocular matching network model after training on the right image, said
(Chem. 121)

9. A method according to item 8, wherein is the pixel coordinate of the pixel point.
(Item 12)
Wherein the depth-labeled real binocular data includes left and right images, the monocular depth estimation network model training process comprises:
obtaining a left image or a right image of the depth-labeled real binocular data as a training sample;
training a monocular depth estimation network model based on left or right images of the depth-labeled real binocular data.
(Item 13)
Whereas the unlabeled real binocular data includes left and right images, the monocular depth estimation network model training process comprises:
inputting the real binocular data without depth labels into the first binocular matching neural network model to obtain a corresponding disparity map;
Based on the corresponding disparity map, the lens baseline length of the camera that captures the real binocular data without the depth label, and the lens focal length of the camera that captures the real binocular data without the depth label, the corresponding disparity map determining a depth map;
taking the left image or the right image of the real binocular data without depth labels as sample data, and teaching a monocular depth estimation network model based on the corresponding depth map of the disparity map, whereby the monocular depth estimation network model 12. A method according to any one of items 6 to 11, comprising training the .
(Item 14)
The analysis result of the image to be processed includes a disparity map output by the monocular depth estimation network model, and further comprising:
A disparity map output by the monocular depth estimation network model, a lens baseline length of a camera that captures an image input to the monocular depth estimation network model, and a lens of a camera that captures an image input to the monocular depth estimation network model determining a corresponding depth map of the disparity map based on the focal length;
and outputting a corresponding depth map of the disparity map.
(Item 15)
an acquisition module configured to acquire an image to be processed;
an execution module configured to input the image to be processed into a trained monocular depth estimation network model to obtain an analysis result of the image to be processed, wherein the monocular depth estimation network model is configured to: an execution module obtained by supervised training with a disparity map output by an eye-matching neural network model;
an output module configured to output an analysis result of the image to be processed.
(Item 16)
Further, a first training module configured to train a second binocular matching neural network model based on the obtained synthetic sample data to obtain a trained second binocular matching neural network model; a second training module configured to adjust parameters of the second binocular matching neural network model after training based on the sample data to obtain the first binocular matching neural network model. Device.
(Item 17)
17. Apparatus according to item 16, further comprising a first acquisition module configured to acquire a depth-labeled synthesized binocular image including a synthesized left image and a synthesized right image as the synthesized sample data. .
(Item 18)
The first training module trains a second binocular matching neural network model based on the synthesized binocular image, and trains a second binocular matching neural network model whose output is a disparity map and an occlusion map. a first training unit configured to obtain, wherein the disparity map represents a disparity distance in pixels between each pixel point in the left image and a corresponding pixel point in the right image; 18. Apparatus according to item 17, wherein the occlusion map represents whether the corresponding pixel point in the right image for each pixel point in the left image is occluded by an object.
(Item 19)
The second training module performs supervised training of a post-trained second binocular matching neural network model based on the acquired depth-labeled real binocular data, whereby the post-trained second binocular matching neural network model is 17. Apparatus according to item 16, comprising a second training unit configured to adjust weights of the network model to obtain a first binocular matching neural network model.
(Item 20)
The second training unit further performs unsupervised training of a post-training second binocular matching neural network model based on the acquired real binocular data without depth labels, whereby the post-training second binocular matching neural network model is 17. Apparatus according to item 16, configured to adjust weights of a neural network model to obtain a first binocular matching neural network model.
(Item 21)
The second training unit uses a loss function to perform unsupervised training of a post-trained second binocular matching neural network model based on the depth-unlabeled real binocular data, whereby the post-trained second 21. Apparatus according to item 20, comprising a second training component configured to adjust the weights of the binocular matching neural network model to obtain a first binocular matching neural network model.
(Item 22)
Furthermore, the expression
(Chem. 122)

a first determination module configured to determine said loss function utilizing
(Chem. 123)

represents the loss function, and
(Chem. 124)

represents the reconstruction error, and
(Chem. 125)

represents that the disparity map output by the first binocular matching network model is less biased than the disparity map output by the second binocular matching network model after training, and
(Chem. 126)

represents that the output gradient constraining the first binocular matching network model matches the output gradient of the trained second binocular matching network model, and
(Chem. 127)

22. The device according to item 21, wherein is the strength factor.
(Item 23)
Furthermore, the expression
(Chem. 128)

,or,
(Chem. 129)

a second determination module configured to determine the reconstruction error using the
(Chem. 130)

represents the number of pixels in the image, and
(Chem. 131)

represents the pixel values of the occlusion map output by the second binocular matching network model after training, and
(Chem. 132)

represents the pixel value of the left image of the real binocular data without a depth label, and
(Chem. 133)

represents the pixel value of the right image of the real binocular data without a depth label, and
(Chem. 134)

represents the pixel value of the synthesized image after sampling the right image, and
(Chem. 135)

represents the pixel value of the synthesized image after sampling the left image, and
(Chem. 136)

represents the pixel values of the disparity map output by the first binocular matching network model of the left image of the real binocular data without depth labels, and
(Chem. 137)

represents the pixel values of the disparity map output by the first binocular matching network model of the right image of the real binocular data without depth labels, and
(Chem. 138)

23. The apparatus of item 22, wherein .epsilon. represents pixel coordinates of a pixel point.
(Item 24)
Furthermore, the expression
(Chem. 139)

,or,
(Chem. 140)

is configured to determine that the disparity map output by the first binocular matching network model is less biased than the disparity map output by the trained second binocular matching network model using a third decision module, wherein said
(Chem. 141)

represents the number of pixels in the image, and
(Chem. 142)

represents the pixel values of the occlusion map output by the second binocular matching network model after training, and
(Chem. 143)

represents the pixel values of the disparity map output by the first binocular matching network model of the left image of the real binocular data without depth labels, and
(Chem. 144)

represents the pixel values of the disparity map output by the first binocular matching network model of the right image of the real binocular data without depth labels, and
(Chem. 145)

represents the pixel values of the disparity map output by the second binocular matching network model after training of the left image, and
(Chem. 146)

represents the pixel values of the disparity map output by the second binocular matching network model after training for the right image, and
(Chem. 147)

represents the pixel coordinates of the pixel point, and
(Chem. 148)

23. A device according to item 22, wherein is the strength factor.
(Item 25)
Furthermore, the expression
(Chem. 149)

,or,
(Chem. 150)

a fourth determination module configured to determine that the output gradient of the first binocular matching network model matches the output gradient of the second binocular matching network model using
(Chem. 151)

represents the number of pixels in the image, and
(Chem. 152)

represents the gradient of the disparity map output by the first binocular matching network model of the left image of real binocular data without depth labels, and
(Chem. 153)

represents the gradient of the disparity map output by the first binocular matching network model of the right image of real binocular data without depth labels, and
(Chem. 154)

represents the gradient of the disparity map output by the second binocular matching network model after training of the left image, said
(Chem. 155)

represents the gradient of the disparity map output by the second binocular matching network model after training on the right image, said
(Chem. 156)

23. The apparatus of item 22, wherein .epsilon. represents pixel coordinates of a pixel point.
(Item 26)
The depth-labeled real binocular data includes a left image and a right image, further obtaining a left image or a right image of the depth-labeled real binocular data as a training sample; 20. Apparatus according to item 19, comprising a third training module configured to train a monocular depth estimation network model based on left or right images of the depth-labeled real binocular data.
(Item 27)
The unlabeled real binocular data includes a left image and a right image, while further inputting the unlabeled real binocular data into the first binocular matching neural network model to obtain a corresponding disparity obtaining a map, based on the corresponding disparity map, the lens baseline length of the camera capturing the real binocular data without the depth label, and the lens focal length of the camera capturing the real binocular data without the depth label, Determining a corresponding depth map of the disparity map, and taking the left image or the right image of the unlabeled real binocular data as sample data, a monocular depth estimation network model based on the corresponding depth map of the disparity map 26. Apparatus according to any one of items 20 to 25, comprising a third training module configured to teach the method to train the monocular depth estimation network model.
(Item 28)
The analysis result of the image to be processed includes the disparity map output by the monocular depth estimation network model, and further, the disparity map output by the monocular depth estimation network model, the monocular depth estimation network model: determining a corresponding depth map of the disparity map based on a lens baseline length of a camera capturing an input image and a lens focal length of a camera capturing an image input to the monocular depth estimation network model; and a first output module configured to output a corresponding depth map of said disparity map.
(Item 29)
15. A monocular depth estimation device comprising a processor and a memory storing a computer program operable in the processor, wherein the processor executes the program in the monocular depth estimation method according to any one of items 1 to 14. A monocular depth estimation device that realizes steps.
(Item 30)
A computer readable storage medium storing a computer program, which when executed by a processor implements the steps in the monocular depth estimation method according to any one of items 1 to 14. medium.

1 is an implementation flowchart 1 of a monocular depth estimation method according to an embodiment of the present application; FIG. 4 is a schematic diagram of depth estimation of a single image according to an embodiment of the present application; Fig. 2 is a training schematic diagram of a second binocular matching neural network model of an embodiment of the present application; FIG. 4 is a training schematic diagram of a monocular depth estimation network model of an embodiment of the present application; FIG. 4 is a schematic diagram of a loss function-related image according to an embodiment of the present application; 2 is an implementation flowchart 2 of a monocular depth estimation method according to an embodiment of the present application; FIG. 4 is a schematic diagram of the effect of the loss function of the example of the present application; FIG. 4 is a schematic diagram of a result of visualization depth estimation according to an embodiment of the present application; 1 is a configuration schematic diagram of a monocular depth estimation device according to an embodiment of the present application; FIG. FIG. 4 is a hardware entity schematic diagram of a monocular depth estimation device according to an embodiment of the present application;

In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the following describes the specific technical solutions of the application in more detail in conjunction with the drawings in the embodiments of the present application. The following examples are intended to illustrate the application and are not intended to limit the scope of the application.

In the description that follows, suffixes to denote elements such as "module", "component" or "unit" are used only to aid in the description of the application and do not themselves have a specific meaning. Thus, "modules", "components" or "units" may be used interchangeably.

In general, using a depth neural network to predict the depth map of a single image, only one image can perform 3D modeling of the image's corresponding scene to obtain the depth of each pixel point. The monocular depth estimation method provided by the embodiments of the present application is obtained by training with a neural network, the training data is derived from the parallax map data output by binocular matching, and does not require expensive depth acquisition equipment such as laser radar. do not do. The binocular matching algorithm that provides the training data is also implemented by a neural network, and the network can achieve a good effect only by pre-training with a large number of virtual binocular image pairs rendered by the rendering engine, and can also be applied to real data. Based on this, further fine-tuning training can be performed to achieve a better effect.

The technical solutions of the present application are further described below in conjunction with drawings and embodiments.

Embodiments of the present application provide a monocular depth estimation method for use in a computing device, and the functions implemented by the method may be implemented by a processor in a server by calling program code, which of course can be stored on a computer storage medium. The server thus includes at least a processor and a storage medium. FIG. 1A is an implementation flowchart 1 of a monocular depth estimation method according to an embodiment of the present application, as shown in FIG. 1A, the method includes:

In step S101, an image to be processed is acquired.

Here, the image to be processed may be acquired by the mobile terminal, and the image to be processed may include an image of an arbitrary scene. In general, mobile terminals are used in the implementation process, such as mobile phones, personal digital assistants (PDAs), navigators, digital phones, video phones, smart watches, smart bracelets, wearable devices, tablets, etc. , may be any type of device capable of processing information. In the implementation process, the server may be a computing device with information processing capabilities, such as mobile terminals such as mobile phones, tablets and laptops, fixed terminals such as personal computers and server clusters.

In step S102, the image to be processed is input to a monocular depth estimation network model obtained by supervised training using a parallax map output by the first binocular matching neural network model, and the analysis result of the image to be processed get

In the embodiments of the present application, the monocular depth estimation network model is mainly obtained by the following three steps. The first step pre-trains a binocular matching neural network using synthetic binocular data rendered by the rendering engine. The second step uses real scene data to train the binocular matching neural network obtained in the first step by fine-tuning. The third step uses the binocular matching neural network obtained in the second step to teach a monocular depth estimation network, thereby training and obtaining a monocular depth estimation network. In the prior art, monocular depth estimation is typically trained using large amounts of labeled real data, or unsupervised methods are used to train monocular depth estimation networks. However, the large amount of labeled real data is expensive to acquire, and if the monocular depth estimation network is trained by the unsupervised method as it is, the depth estimation of the occluded region cannot be processed, and the obtained effect is poor. In contrast, in the present application, the sample data of the monocular depth estimation network model is derived from the disparity map output by the first binocular matching neural network model, that is, the present application performs monocular depth prediction using binocular disparity. Therefore, the method in the present application does not require a large amount of labeled data and can obtain good training effect.

In step S103, the analysis result of the image to be processed is output. Here, the analysis result of the image to be processed refers to the corresponding depth map of the image to be processed. After obtaining an image to be processed, the image to be processed is input to a trained monocular depth estimation network model, the monocular depth estimation network model generally being a depth map of the object to be processed, rather than a depth map. To output the corresponding disparity map of the image, the disparity map output by the monocular depth estimation network model, the lens baseline length of the camera capturing the image to be processed, and the lens focal length of the camera capturing the image to be processed. , the corresponding depth map of the image to be processed should be determined.

FIG. 1B is a schematic diagram of depth estimation of a single image according to an embodiment of the present application. As shown in FIG. is the corresponding depth map of image 11 of .

In practical application, the ratio of the product of the lens base length and the lens focal length and the corresponding disparity map of the output image to be processed is determined as the corresponding depth map of the image to be processed. good too.

Based on the above method embodiments, the embodiments of the present application further provide a monocular depth estimation method, which includes the following.

In step S111, a depth-labeled synthesized binocular image including a synthesized left image and a synthesized right image is obtained as synthesized sample data.

In some embodiments, the method further comprises steps S11 of constructing a virtual 3D scene by a rendering engine; mapping S12 of the 3D scene as binocular images by two virtual cameras; constructing the virtual 3D scene; Step S13 of acquiring depth data of the synthesized binocular image based on the position when constructing the virtual 3D scene, the direction when constructing the virtual 3D scene, and the lens focal length of the virtual camera; and the binocular image based on the depth data a step S14 of labeling images to obtain said combined binocular image.

In step S112, train a second binocular matching neural network model based on the obtained synthetic sample data.

Here, in actual application, the step S112 may be realized by the following steps. Step S1121, training a second binocular matching neural network model based on the combined binocular image to obtain a trained second binocular matching neural network model whose output is a disparity map and an occlusion map. Here, the parallax map expresses the parallax distance in units of pixels between each pixel point in the left image and the corresponding pixel point in the right image, and the occlusion map expresses the above-mentioned distance of each pixel point in the left image. Represents whether the corresponding pixel point in the right image is occluded by an object.

FIG. 1C is a training schematic diagram of the second binocular matching neural network model of the embodiment of the present application, as shown in FIG. 12 is the right image of the combined binocular image,

is the pixel value of all pixel points included in the left image 11 numbered 11,

is the pixel value of all pixel points contained in the right image 12 numbered 12, the image 13 numbered 13 is the occlusion map output after the second binocular matching neural network model is trained, and the number is The image 14 of 14 is the disparity map output after the second binocular matching neural network model is trained, and the image 15 of number 15 is the second binocular matching neural network model.

In step S113, the parameters of the trained second binocular matching neural network model are adjusted according to the acquired real sample data to obtain the first binocular matching neural network model.

Here, the step S113 can be implemented in two forms. Among them, the first implementation mode is implemented by the following steps. Step S1131a, perform supervised training of the trained second binocular matching neural network model based on the obtained depth-labeled real binocular data, thereby weighting the trained second binocular matching neural network model to obtain the first binocular matching neural network model. Here, what is acquired is the depth-labeled real binocular data. In this way, using the depth-labeled real binocular data as it is, the second binocular matching neural network after training in step S112 supervised training, thereby adjusting the weight of the second binocular matching neural network model after training, further improving the effect of the second binocular matching neural network model after training, and the first binocular matching neural network model A network model can be obtained. In this part, the binocular disparity network needs to fit the real data. Using depth-labeled real binocular data, the binocular disparity network may be directly fine-tuned and trained by supervised training to adjust the weights of the network. The second implementation mode is implemented by the following steps. Step S1131b, perform unsupervised training of the trained second binocular matching neural network model based on the acquired real binocular data without depth labels, thereby weighting the trained second binocular matching neural network model to obtain the first binocular matching neural network model. Embodiments of the present application also use real binocular data without depth labels to perform unsupervised training of the post-trained second binocular matching neural network model, whereby the post-trained second binocular matching neural network model is The model weights may be adjusted to obtain a first binocular matching neural network model. Unsupervised training here refers to training with only binocular data without depth data labels, and the process may be realized by an unsupervised fine-tuning method.

In step S114, a monocular depth estimation network model is taught by the disparity map output by the first binocular matching neural network model, thereby training the monocular depth estimation network model.

Here, step S114 may be implemented in two forms. Among them, the first implementation mode is implemented by the following steps. Step S1141a, of the depth-labeled real binocular data including left and right images, obtain a left image or a right image as a training sample. Step S1142a, training a monocular depth estimation network model based on the left image or the right image of the depth-labeled real binocular data. Here, if a depth neural network is used to predict the depth map of a single image, only one image can perform 3D modeling of the corresponding scene of the image to obtain the depth of each pixel point. Therefore, a monocular depth estimation network model may be trained based on the left image or right image of the depth-labeled real binocular data, wherein the depth-labeled real binocular data is used in step S1131a. This is real binocular data with depth labels. The second implementation mode is implemented by the following steps. Step S1141b, inputting the unlabeled real binocular data including left and right images into the first binocular matching neural network model to obtain a corresponding disparity map. Step S1142b, based on the corresponding disparity map, the lens baseline length of the camera capturing the real binocular data without the depth label and the lens focal length of the camera capturing the real binocular data without the depth label, the disparity map Determine the corresponding depth map of . Step S1143b, taking the left image or the right image of the real binocular data without depth label as sample data, and teaching a monocular depth estimation network model based on the corresponding depth map of the disparity map, whereby the monocular depth Train an estimation network model. Here, if a depth neural network is used to predict the depth map of a single image, only one image can perform 3D modeling of the corresponding scene of the image to obtain the depth of each pixel point. Therefore, the left image or right image of the real binocular data without depth labels used in step S1131b is used as sample data, and the left image or right image of the real binocular data without depth labels used in step S1141b is used as sample data. The right image is also taken as sample data, and a monocular depth estimation network model is taught based on the corresponding depth map of the disparity map output in step S1141b, thereby training the monocular depth estimation network model, and the monocular depth after training An estimated network model may be obtained.

FIG. 1D is a training schematic diagram of the monocular depth estimation network model of the embodiment of the present application. As shown in FIG. 1D, FIG. to obtain the disparity map 13 with the corresponding number 13, wherein the real binocular data without depth label includes the left image 11 with the number 11 and the right image 12 with the number 12, and the number 15 is the first binocular matching neural network model. Diagram (b) in FIG. 1D shows a monocular depth estimation network based on the depth map corresponding to the parallax map 13 with the number 13, using the left image or the right image of the real binocular data without the depth label as sample data. teaching a model and thereby training the monocular depth estimation network model, wherein the output by the monocular depth estimation network model of the sample data is a disparity map 14 numbered 14 and an image numbered 16; 16 is a monocular depth estimation network model.

In step S115, an image to be processed is acquired.

Now, once we have the trained monocular depth estimation network model, we can use this monocular depth estimation network model. That is, this monocular depth estimation network model can be used to obtain the corresponding depth map of the image to be processed.

In step S116, the image to be processed is input to the monocular depth estimation network model obtained by supervised training using the parallax map output by the first binocular matching neural network model, and the analysis result of the image to be processed get

In step S117, an analysis result of the image to be processed including the parallax map output by the monocular depth estimation network model is output.

In step S118, the disparity map output by the monocular depth estimation network model, the lens baseline length of the camera that captures the image input to the monocular depth estimation network model, and the image input to the monocular depth estimation network model are captured. Determining a corresponding depth map of the parallax map based on the lens focal length of the corresponding camera.

In step S119, a depth map corresponding to the disparity map is output.

Based on the above method embodiments, the embodiments of the present application further provide a monocular depth estimation method, which includes the following.

In step S121, a depth-labeled synthesized binocular image including a synthesized left image and a synthesized right image is acquired as synthesized sample data.

In step S122, train a second binocular matching neural network model based on the obtained synthetic sample data.

Here, using the synthetic data to train the second binocular matching neural network model can exhibit higher generalization ability.

In step S123, formula (1)

is used to determine the loss function. where

represents the loss function provided by the embodiments of the present application, and

represents the reconstruction error, and

represents that the disparity map output by the first binocular matching network model is less biased than the disparity map output by the second binocular matching network model after training, and

represents that the output gradient constraining the first binocular matching network model matches the output gradient of the trained second binocular matching network model, and

represents the strength factor. here,

is a regular term.

In some embodiments, equation (1) in step S123 may be further subdivided by equations in the following steps. Thus, the method further includes: In step S1231, formula (2)

or formula (3)

is used to determine the reconstruction error. where

represents the number of pixels in the image, and

represents the pixel values of the occlusion map output by the second binocular matching network model after training, and

represents the pixel value of the left image of the real binocular data without a depth label, and

represents the pixel value of the right image of the real binocular data without a depth label, and

represents the pixel values of the reconstructed left image, that is, the synthesized image after sampling the right image, and

represents the pixel values of the reconstructed right image, that is, the synthesized image after sampling the left image, and

represents the pixel values of the disparity map output by the first binocular matching network model of the left image of the real binocular data without depth labels, and

represents the pixel values of the disparity map output by the first binocular matching network model of the right image of the real binocular data without depth labels, and

represents the pixel coordinates of the pixel point, and

represents the output of the second binocular matching network model after training, and

represents the right image or the associated data of the right image, and

represents the left image or the associated data of the left image, and

represents the RGB (Red, Green and Blue) values of an image pixel point. In step S1232, expression (4)

or formula (5)

is used to determine that the disparity map output by the first binocular matching network model is less biased than the disparity map output by the trained second binocular matching network model. where

represents the number of pixels in the image, and

represents the pixel values of the occlusion map output by the second binocular matching network model after training, and

represents the pixel values of the disparity map output by the second binocular matching network after training of the left image of the sample data, and

represents the pixel values of the disparity map output by the second binocular matching network after training of the right image of the sample data, said

represents the pixel values of the disparity map output by the first binocular matching network of the left image of the real binocular data without depth labels, and

represents the pixel values of the disparity map output by the first binocular matching network of the right image of the real binocular data without depth labels, and

represents the pixel coordinates of the pixel point, and

represents the output of the second binocular matching network model after training, and

represents the right image or the associated data of the right image, and

represents the left image or the associated data of the left image, and

represents the strength factor. In step S1233, expression (6)

or formula (7)

is used to determine that the output gradient of the first binocular matching network model matches the output gradient of the second binocular matching network model. where

represents the number of pixels in the image, and

represents the gradient of the disparity map output by the first binocular matching network of the left image of real binocular data without depth labels, and

represents the gradient of the disparity map output by the first binocular matching network for the right image of real binocular data without depth labels, and

represents the gradient of the disparity map output by the second binocular matching network after training of the left image of the sample data, said

represents the gradient of the disparity map output by the second binocular matching network after training on the right image of the sample data, said

represents the output of the second binocular matching network model after training, and

represents the right image or the associated data of the right image, and

represents the left image or the associated data of the left image.

In step S124, a loss function (Loss) is used to perform unsupervised training of a post-trained second binocular matching neural network model based on the real binocular data without depth labels, whereby the post-trained second binocular matching neural network model is Adjust the weights of the binocular matching neural network model to obtain the first binocular matching neural network model.

Here, the loss function (Loss) regularizes the training by fine-tuning according to the output of the second binocular matching neural network after training in step S122, and the prediction widely present in unsupervised fine-tuning in the prior art is It avoids the problem of ambiguity and enhances the effect of the first binocular matching network obtained by fine-tuning, thereby indirectly improving the effect of the monocular depth network obtained by teaching the first binocular matching network. improve to FIG. 1E is a schematic diagram of a loss function-related image according to an embodiment of the present application. As shown in FIG. 1E, FIG. ) is the right image of real binocular data without depth label, and FIG. Fig. 1E is a disparity map output after inputting to a binocular matching neural network model; FIG. 1E is a reconstructed image of the left image combined with the parallax map, and FIG. 1E shows the pixels in the left image represented by FIG. Fig. 1E is a reconstruction error map of the image obtained by finding the difference between the corresponding pixels in the left image of the left image, that is, the reconstruction error map of the left image. FIG. 11 is an occlusion map output after inputting a real binocular image without depth label into a second trained binocular matching neural network model; FIG. Here, all the red frames 11 in FIG. A frame 12 represents an erroneous or occluded portion in the reconstruction error map. Now, when implementing the unsupervised fine-tuning binocular disparity network training in step S124, the left image needs to be reconstructed using the right image, but the region where the occlusion is present is accurately reconstructed. Since it is not configurable, an occlusion map is used to remove the mistraining signal in this portion to improve the effectiveness of unsupervised fine-tuning training.

In step S125, the monocular depth estimation network model is taught by the disparity map output by the first binocular matching neural network model, thereby training the monocular depth estimation network model.

Here, the sample image of the monocular depth estimation network model may be the left image of the real binocular data without the depth label, or the right image of the real binocular data without the depth label. good too. Among them, when the left image is the sample image, equations (1), (2), (4) and (6) are used to determine the loss function, and when the right image is the sample image, the equation (1), Eq. (3), Eq. (5) and Eq. (7) are used to determine the loss function.

In an embodiment of the present application, the step of teaching the monocular depth estimation network model with a disparity map output by the first binocular matching neural network model, thereby training the monocular depth estimation network model, comprises: Teaching the monocular depth estimation network model by the corresponding depth map of the disparity map output by the monocular matching neural network model, that is, providing teaching information, thereby training the monocular depth estimation network model. .

In step S126, an image to be processed is acquired.

In step S127, the image to be processed is input to the monocular depth estimation network model obtained by supervised training using the parallax map output by the first binocular matching neural network model, and the analysis result of the image to be processed get

In step S128, an analysis result of the image to be processed including the parallax map output by the monocular depth estimation network model is output.

In step S129, the parallax map output by the monocular depth estimation network model, the lens baseline length of the camera that captures the image input to the monocular depth estimation network model, and the image input to the monocular depth estimation network model are captured. Determining a corresponding depth map of the parallax map based on the lens focal length of the corresponding camera.

In step S130, output a corresponding depth map of the disparity map.

In an embodiment of the present application, if the image to be processed is an image of a cityscape, the trained monocular depth estimation network model can be used to predict the depth of the image of the cityscape.

Based on the above method embodiments, an embodiment of the present application further provides a monocular depth estimation method, FIG. 2A is an implementation flowchart 2 of a monocular depth estimation method according to an embodiment of the present application, as shown in FIG. Methods include:

In step S201, the synthetic data rendered by the rendering engine is used to train a binocular matching network to obtain the binocular image disparity map.

where the input of the binocular matching network is a pair of binocular images (including left and right images), and the output of the binocular matching network is a disparity map, an occlusion map, i.e., binocular The matching network uses binocular images as input and outputs a disparity map and an occlusion map. Among them, the parallax map is used to express the parallax distance in pixels between each pixel point in the left image and the corresponding pixel point in the right image, and the occlusion map is used to express the correspondence of each pixel in the left image in the right image. It is used to express whether the pixel point to be covered is occluded by another object. Due to changes in viewing angle, some areas in the left image are occluded by other objects in the right image, so the occlusion map is used to level whether pixels in the left image are occluded in the right image. In this part, the binocular matching network is trained using synthetic data generated by a computer rendering engine, firstly constructing some virtual 3D scenes by the rendering engine, and then by two virtual cameras. We map it as an eye image, thereby obtaining synthetic data, and at the same time accurate depth data and data such as camera focal length are also obtained from the rendering engine, so the binocular matching network can be supervised training directly with these labeled data. It can be carried out.

In step S202, the loss function is used to fine-tune the binocular matching network obtained in step S201 based on the real binocular image data by an unsupervised fine-tuning method.

In this part, even if we use real binocular data without depth labels to train the binocular disparity network unsupervised, the binocular disparity network needs to fit the real data. Unsupervised training here refers to training with only binocular data in the absence of depth data labels. The embodiments of the present application provide a new unsupervised fine-tuning method, namely unsupervised fine-tuning using the loss function in the above embodiments. The main purpose of the loss function provided by the embodiments of the present application is to fine-tune the binocular disparity network based on the real binocular data without reducing the pre-training effect, and the fine-tuning process includes step S201 The preliminary outputs of the pre-trained binocular disparity network obtained in , are used for guidance and regularization. FIG. 2B is a schematic diagram of the effect of the loss function of the embodiment of the present application, as shown in FIG. The image 22 numbered 22 is the disparity map obtained when using the loss function provided by the embodiments of the present application. Whereas the loss function of the prior art does not consider the occluded regions alone and also optimizes the image reconstruction error of the occluded regions to zero, which causes the prediction disparity error of the occluded regions and also blurs the edges of the disparity map. Therefore, the loss function in the present application uses an occlusion map to remove this portion of the mistraining signal to improve the training effect of unsupervised fine-tuning.

In step S203, the binocular matching network obtained in step S202 is used to teach monocular depth estimation based on real data, finally obtaining a monocular depth estimation network. where the input of said monocular depth estimation network is a single monocular image and the output of said monocular depth estimation network is a depth map. In step S202, a fine-tuned binocular disparity network is obtained based on the actual data. For each pair of binocular images, the binocular disparity network predicts and obtains a disparity map. With b and lens focal length f, the corresponding depth map of the disparity map can be calculated and obtained, namely equation (8)

can be obtained by computing the corresponding depth map d of the disparity map by In order to train a monocular depth network to predict and obtain a depth map, the left image of a pair of binocular images is taken as input for the monocular depth network, and then the resulting depth map is computed by the binocular disparity network. It may be output and taught to train a monocular depth network to obtain final results. In practical application, it can be trained by the monocular depth estimation method in the embodiments of the present application to obtain a depth estimation module for driverless driving, thereby performing three-dimensional scene reconstruction or obstacle detection. And the unsupervised fine-tuning method provided by the embodiments of the present application improves the performance of the binocular disparity network.

In the prior art, supervised monocular depth estimation methods can only obtain a fairly limited number of accurate labeled data, and they are very difficult to obtain. The performance of unsupervised methods based on reconstruction error is usually limited by the ambiguity of pixel matching. To solve these problems, the embodiments of the present application provide a new monocular depth estimation method to break through the limitations of the prior art supervised and unsupervised depth estimation methods. The method in the present embodiment uses a binocular matching network to train on cross-modal synthetic data and thereby teaches a monocular depth estimation network. Since the binocular matching network obtains disparity based on the pixel matching relationship between left and right images, rather than extracting from semantic features, the binocular matching network can effectively generalize from synthetic data to real data. . The method of the embodiment of the present application mainly includes three steps. First, we use synthetic data to train a binocular matching network to predict occlusion and disparity maps from binocular images. Second, we selectively tune the trained binocular matching network, supervised or unsupervised, using available real data. Third, train a monocular depth estimation network under the guidance of a binocular matching network fine-tuned and trained with real data obtained in the second step. In this way, by indirectly using the binocular matching network, it is possible to more effectively use the synthetic data in monocular depth estimation and improve the performance.

In the first step, synthetic data are utilized to train a binocular matching network, which includes: Currently, graphics rendering engines can generate a number of composite images that include depth information. However, training monocular depth estimation networks by directly merging these synthetic image data with real data usually results in poor performance, because monocular depth estimation is very sensitive to the semantic information input into the scene. The huge modality difference between synthetic and real data renders assisted training using synthetic data utterly useless. However, binocular matching networks have stronger generalization ability, and binocular matching networks trained using synthetic data can output good disparity maps even based on real data. Therefore, embodiments of the present application connect synthetic data and real data via binocular matching network training to improve the performance of monocular depth training. First, a large amount of synthetic binocular data is used to pre-train the binocular matching network. Unlike conventional structures, the binocular matching network in the example further estimates a multi-scale occlusion map under the disparity map. Here, the occlusion map indicates whether, in the correct image, the corresponding pixel point in the right image of the pixel in the left image is occluded by another object. In the next step, the occlusion map is used in an unsupervised fine-tuning method, thereby avoiding misestimation. Among them, using the left-right parallax consistency check method, equation (9)

may be used to obtain an occlusion map with correct labels from a correctly labeled disparity map. where subscript

is in the image

Represents row value, subscript

is in the image

Represents the column value.

represents the disparity map of the left and right images, and

is the parallax map of the left image reconstructed from the right image, and the left parallax map and the parallax map of the left image reconstructed from the right image coincide with each other in the unoccluded area. Assume that the consistency check threshold is 1. The occlusion map is set to 0 in occluded areas and 1 to non-occluded areas. Therefore, this embodiment uses the formula (10)

is used to compute the loss of training the binocular matching network on synthetic data (Loss), and at this stage the loss function

has two parts: the disparity map estimation error

and occlusion map estimation error

consists of Disparity and occlusion prediction also occur in the multiscale hidden layer of the binocular disparity network, and the loss weight of the multiscale prediction remains unchanged.

used for

represents the corresponding disparity map estimation error for each layer, and

represents the corresponding occlusion map estimation error for each layer, and

represents strata. To train the disparity map, we adopt the L1 loss function to avoid the influence of outliers and improve the robustness of the training process. To train the occlusion map, equation (11)

is the occlusion map estimation error

and train an occlusion map with binary cross-entropy loss as the classification task. here,

is the total number of pixels in the image, and

represents the occlusion map with correct labels, and

represents the occlusion map output by the trained binocular matching network.

In the second step, a supervised or unsupervised fine-tuning method is used to train the post-trained binocular matching network obtained in the first step based on real data, which includes: Embodiments of the present application fine-tune the trained binocular matching network in two ways. Among them, in the supervised fine-tuning method, the multi-scale L1 regression loss function

, that is, the disparity map estimation error

is employed to improve the error of previous pixel matching prediction, for which equation (12)

See The results show that even with a small number of surveillance data, say 100 images, the binocular matching network can match the synthetic modal data to the real modal data. For the unsupervised fine-tuning method, for unsupervised network tuning, the prior art unsupervised fine-tuning method gives a blurred disparity map and poor performance, as shown in image 21 in FIG. 2B. This is due to the limitations of unsupervised loss and the ambiguity of pixel matching containing only RGB values. Thus, embodiments of the present application introduce an additional holomorphic term to improve performance due to the constraint. With real data, we obtain the corresponding occlusion and disparity maps from the unfine-tuned post-training binocular matching network, and transform them into

label with . These two data are used to normalize the training process. Furthermore, the unsupervised fine-tuning loss function provided by the embodiments of the present application, namely the loss function

can be obtained by referring to the description in the previous example.

The third step trains a monocular depth estimation network, which includes: So far, we have cross-modally trained a binocular matching network with large amounts of synthetic data and fine-tuned using real data. To train the final monocular depth estimation network, embodiments of the present application employ the disparity map predicted by the trained binocular matching network to provide training data. Loss of monocular depth estimation

is the formula (13)

is obtained from multiple parts shown in . here,

is the sum of pixel points, and

represents the disparity map output by the monocular depth estimation network, and

represents the disparity map output by a trained binocular matching network or by a fine-tuned trained binocular matching network. It should be noted that Equations (9) to (13) all describe using the left image of the real data as a training sample by the monocular depth estimation network. For experiments, the training data is not cropped and scaled because the monocular depth estimation network is sensitive to changes in viewing angle. Both the input for the monocular depth estimation network and the disparity map for teaching the monocular depth estimation network are obtained from the trained binocular matching network. FIG. 2C is a schematic diagram of the results of visualization depth estimation of an embodiment of the present application, showing the corresponding depth maps of three different cityscape images acquired using the monocular depth estimation method in the prior art and an embodiment of the present application, of which: , the first line is the input of the monocular depth estimation network, i.e., three different cityscape images, the second line is the depth data obtained by interpolating to the sparse laser radar depth map by the nearest neighbor method, and the third line is The fifth line from is the corresponding depth map of three input images respectively obtained by three different monocular depth estimation methods in the prior art. Our results are shown in the last three lines, which directly utilize the binocular matching network trained using synthetic data obtained in the first step in our example to teach a monocular depth estimation network. The corresponding depth maps of the three input images of the monocular depth network obtained by Three inputs of the monocular depth network obtained by fine-tuning the trained binocular matching network according to the loss function, and using the disparity map output by the fine-tuned network as training data for the monocular depth estimation network. Supervised fine-tuning of the corresponding depth maps of the images, namely image 24 with number 24, image 25 with number 25, image 26 with number 26, and the post-training binocular matching network, yielding the fine-tuned network The corresponding depth maps of the three input images of the monocular depth network obtained by taking the disparity map output by as training data for the monocular depth estimation network, i.e. image 27 with number 27 and image 28 with number 28 , the image 29 with the number 29 . As can be seen from image 21 numbered 21 to image 29 numbered 29, the model obtained by the monocular depth estimation method in the embodiments of the present application can capture finer scene structure.

An embodiment of the present application provides a monocular depth estimation device, and FIG. 3 is a structural schematic diagram of the monocular depth estimation device of an embodiment of the present application, as shown in FIG. and inputting said image to be processed into a monocular depth estimation network model obtained by supervised training with a disparity map output by a first binocular matching neural network model, said It includes an execution module 302 configured to obtain an analysis result of an image to be processed, and an output module 303 configured to output an analysis result of the image to be processed.

In some embodiments, the apparatus further teaches the monocular depth estimation network model with a disparity map output by the first binocular matching neural network model, thereby training the monocular depth estimation network model. a third training module configured to

In some embodiments, the apparatus further comprises a first training module configured to train a second binocular matching neural network model based on the obtained synthetic sample data; a second training module configured to adjust parameters of the second binocular matching neural network model after training to obtain the first binocular matching neural network model.

In some embodiments, the apparatus is further configured to acquire a depth-labeled synthesized binocular image including a synthesized left image and a synthesized right image as the synthesized sample data. Contains an acquisition module.

In some embodiments, the first training module trains a second binocular matching neural network model based on the synthesized binocular images, and trains a second binocular matching neural network model whose output is a disparity map and an occlusion map. a first training unit configured to obtain a binocular matching neural network model, wherein the disparity map is pixel-by-pixel between each pixel point in the left image and a corresponding pixel point in the right image; The occlusion map expresses whether the corresponding pixel point in the right image of each pixel point in the left image is occluded by an object.

In some embodiments, the apparatus further comprises a construction module configured to construct a virtual 3D scene via a rendering engine; and a construction module configured to map the 3D scene as binocular images via two virtual cameras. a mapping module configured to obtain depth data of the synthesized binocular image based on a position when constructing the virtual 3D scene, a direction when constructing the virtual 3D scene and a lens focal length of the virtual camera; and a third acquisition module configured to label the binocular images based on the depth data to obtain the combined binocular image.

In some embodiments, the second training module performs supervised training of a post-trained second binocular matching neural network model based on the acquired depth-labeled real binocular data, thereby supervising the post-training a second training unit configured to adjust the weights of the second binocular matching neural network model of to obtain the first binocular matching neural network model.

In some embodiments, the second training unit in the second training module further performs unsupervised training of a post-trained second binocular matching neural network model based on the acquired unlabeled real binocular data. and thereby adjust the weights of the second binocular matching neural network model after training to obtain the first binocular matching neural network model.

In some embodiments, a second training unit in the second training module uses a loss function to train a second binocular matching neural network model based on the real binocular data without depth labels. a second training component configured to perform unsupervised training, thereby adjusting weights of the second binocular matching neural network model after training to obtain a first binocular matching neural network model;

In some embodiments, the device further comprises formula (14)

a first determination module configured to determine said loss function utilizing

represents the loss function, and

represents the reconstruction error, and

represents that the disparity map output by the first binocular matching network model is less biased than the disparity map output by the second binocular matching network model after training, and

represents that the output gradient constraining the first binocular matching network model matches the output gradient of the trained second binocular matching network model, and

represents the strength factor.

In some embodiments, the device further comprises formula (15)

or formula (16)

a second determination module configured to determine the reconstruction error using the

represents the number of pixels in the image, and

represents the pixel values of the occlusion map output by the second binocular matching network model after training, and

represents the pixel value of the left image of the real binocular data without a depth label, and

represents the pixel value of the right image of the real binocular data without a depth label, and

represents the pixel value of the synthesized image after sampling the right image, and

represents the pixel value of the synthesized image after sampling the left image, and

represents the pixel values of the disparity map output by the first binocular matching network model of the left image of the real binocular data without depth labels, and

represents the pixel value of the disparity map output by the first binocular matching network model of the right image of the real binocular data without depth labels,

represents the pixel coordinates of a pixel point.

In some embodiments, the device further comprises formula (17)

or formula (18)

is configured to determine that the disparity map output by the first binocular matching network model is less biased than the disparity map output by the trained second binocular matching network model using a third decision module, wherein said

represents the pixel values of the disparity map output by the second binocular matching network model after training of the left image of the sample data, and

represents the pixel values of the disparity map output by the second binocular matching network model after training of the right image of the sample data, and

represents the strength factor.

In some embodiments, the device further comprises formula (19)

or formula (20)

a fourth determination module configured to determine that the output gradient of the first binocular matching network model matches the output gradient of the second binocular matching network model using

represents the gradient of the disparity map output by the first binocular matching network model of the left image of real binocular data without depth labels, and

represents the gradient of the disparity map output by the first binocular matching network model of the right image of real binocular data without depth labels, and

represents the gradient of the disparity map output by the second binocular matching network model after training on the left image of the sample data, said

represents the gradient of the disparity map output by the second binocular matching network model after training on the right image of the sample data.

In some embodiments, the depth-labeled real binocular data includes a left image and a right image, whereas the third training module comprises a left image of the depth-labeled real binocular data. or a first acquisition unit configured to acquire a right image as a training sample; and training a monocular depth estimation network model based on the left image or the right image of the depth-labeled real binocular data. and a configured first training unit.

In some embodiments, the unlabeled real binocular data includes a left image and a right image, whereas the third training module further converts the unlabeled real binocular data to the first a second acquisition unit configured to input a binocular matching neural network model to obtain a corresponding disparity map; and a first determination unit configured to determine a corresponding depth map of the disparity map based on a lens focal length of a camera that captures the real binocular data without depth labels; taking the left image or the right image of the eye data as sample data and teaching a monocular depth estimation network model based on the corresponding depth map of the disparity map, thereby training the monocular depth estimation network model. and a second training unit.

In some embodiments, the analysis result of the image to be processed includes a disparity map output by the monocular depth estimation network model, whereas the apparatus further outputs a disparity map output by the monocular depth estimation network model. Correspondence of the disparity map based on the disparity map, the lens baseline length of the camera that captures the image input to the monocular depth estimation network model, and the lens focal length of the camera that captures the image input to the monocular depth estimation network model. and a first output module configured to output a corresponding depth map of the disparity map.

It should be noted here that the above apparatus embodiments are similar in their description to the above method embodiments and have similar beneficial effects to the method embodiments. For technical details not disclosed in the apparatus embodiments of the present application, please refer to the description for the method embodiments of the present application. In embodiments of the present application, the monocular depth estimation method described above is implemented in the form of software functional modules and may be stored on a computer readable storage medium when sold or used as an independent product. Based on this observation, the technical solutions in the embodiments of the present application can be implemented in the form of software products, which are substantially or contribute to the prior art, and the computer software products are stored in a storage medium. and comprises instructions that cause a computer device to perform all or part of the methods described in the embodiments herein. The storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard disk, a ROM (Read Only Memory), a magnetic disk or an optical disk. Thus, embodiments of the present application are not limited to any specific combination of hardware and software. In contrast, an embodiment of the present application is a monocular depth estimation device comprising a processor and a memory having a computer program operable on the processor, the processor implementing steps in a monocular depth estimation method when executing the program. To provide a monocular depth estimation device that In contrast, an embodiment of the present application provides a computer readable storage medium having a computer program stored thereon, the computer program implementing steps in a monocular depth estimation method when executed by a processor. do. It should be pointed out here that the above storage medium and apparatus embodiments are similar in their description to the above method embodiments and have similar beneficial effects to the method embodiments. . For technical details not disclosed in the storage medium and device embodiments of the present application, please refer to the description of the method embodiments of the present application.

It should be noted that FIG. 4 is a schematic diagram of the hardware entity of the monocular depth estimation device of the embodiment of the present application. As shown in FIG. 402 and processor 403, of which memory 401 is configured to store instructions and applications executable by processor 403, and data to be processed or processed by processor 403 and each module in monocular depth estimation device 400. can be cached, which can be implemented by FLASH® (flash memory) or RAM (Random Access Memory). Communication bus 402 may allow monocular depth estimation device 400 to communicate with other terminals or servers over a network, and may also provide connections and communications between processor 403 and memory 401 . Processor 403 generally controls the overall operation of monocular depth estimation device 400 .

It should be noted that, as used herein, the terms “comprising,” “consisting of,” or any other variation are intended to include non-exclusive inclusion, whereby a process, method, article or apparatus comprising a series of elements is intended to include not only those elements, but also other elements not specified or inherent in such processes, methods, articles or devices. Unless specifically stated otherwise, an element limited by the following phrase "comprising a" does not exclude the presence of other identical elements in the process, method, article or apparatus containing that element.

From the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented in the form of a combination of software and the necessary common hardware platform. may be used, although the former is often the preferred embodiment. Based on this observation, the technical solution of the present application can be substantially implemented in the form of software or the part that contributes to the prior art, and the computer software product comprises a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes a plurality of instructions that cause a terminal device (which may be a mobile phone, computer, server, air conditioner, network device, etc.) to perform the method described in each embodiment of the present application.

This application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (apparatus) and computer program products according to embodiments of the application. It is to be understood that each flow and/or block in the flowchart and/or block diagrams, and combinations of flows and/or blocks in the flowchart and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a common computer, special purpose computer, embedded processor or other programmable data processing apparatus to produce a machine, thereby rendering the computer or other programmable data processing apparatus The instructions executed by the processor create the means to perform the functions specified in one or more flows of the flowcharts and/or one or more blocks of the block diagrams. These computer program instructions may be stored in a computer readable memory capable of directing a computer or other programmable data processing apparatus to operate in a particular manner, whereby the instructions stored in the computer readable memory may be stored in the computer readable memory. creates an article of manufacture that includes instruction means that implement the functions specified in one or more of the flows in the flowcharts and/or one or more blocks in the block diagrams.

These computer program instructions may be loaded into a computer or other programmable data processing apparatus to cause the computer or other programmable data processing apparatus to perform a series of operational steps to produce a computer-implemented process. , whereby instructions executed in a computer or other programmable data processing apparatus provide steps for implementing the functions specified in one or more flows of the flowcharts and/or one or more blocks of the block diagrams. .

The above is only a preferred embodiment of the present application, and is not intended to limit the patent scope of the present application. Any direct or indirect diversion thereof to is likewise covered by the patent protection scope of the present application.

Claims (16)

A monocular depth estimation method performed by a computing device, the monocular depth estimation method comprising:
obtaining an image to be processed;
inputting the image to be processed into a trained monocular depth estimation network model to obtain an analysis result of the image to be processed, wherein the monocular depth estimation network model is processed by a first binocular matching neural network model: a step obtained by supervised training with the output disparity map;
a step of outputting an analysis result of the image to be processed;
including
The training process of the first binocular matching neural network model includes:
training a second binocular matching neural network model based on the obtained synthetic sample data to obtain a second binocular matching neural network model after training;
adjusting parameters of the trained second binocular matching neural network model according to the acquired real sample data to obtain a first binocular matching neural network model;
including
The monocular depth estimation method further comprises acquiring a depth-labeled synthesized binocular image including a synthesized left image and a synthesized right image as the synthesized sample data,
Obtaining the depth-labeled synthesized binocular image includes constructing a virtual 3D scene by a rendering engine, mapping the 3D scene as a binocular image by two virtual cameras, and obtaining depth data of the synthesized binocular image based on the position when constructing the scene, the direction when constructing the virtual 3D scene, and the lens focal length of the virtual camera; and based on the depth data, labeling binocular images and obtaining said combined binocular image . training a second binocular matching neural network model based on the obtained synthetic sample data,
training a second binocular matching neural network model based on the combined binocular image to obtain a trained second binocular matching neural network model whose output is a disparity map and an occlusion map ;
The parallax map expresses the parallax distance in units of pixels between each pixel point in the left image and the corresponding pixel point in the right image, and the occlusion map expresses each pixel point in the right image in the left image. 2. The monocular depth estimation method of claim 1 , which expresses whether the corresponding pixel point is occluded by an object. adjusting parameters of the second binocular matching neural network model after training based on the acquired real sample data to obtain a first binocular matching neural network model,
supervised training a second post-trained binocular matching neural network model based on the acquired depth-labeled real binocular data, thereby adjusting the weights of the post-trained second binocular matching neural network model. , obtaining a first binocular matching neural network model. adjusting parameters of the second binocular matching neural network model after training based on the acquired real sample data to obtain a first binocular matching neural network model ,
unsupervised training of a second post-trained binocular matching neural network model based on the acquired real binocular data without depth labels, thereby adjusting the weights of the post-trained second binocular matching neural network model. , further comprising obtaining a first binocular matching neural network model. unsupervised training of a post-trained second binocular matching neural network model based on the acquired real binocular data without depth labels, thereby adjusting the weights of the post-trained second binocular matching neural network model. and obtaining a first binocular matching neural network model,
unsupervised training of a post-trained second binocular matching neural network model based on said unlabeled real binocular data using a loss function, thereby said post-trained second binocular matching neural network model 5. The monocular depth estimation method of claim 4 , comprising adjusting the weights of , to obtain a first binocular matching neural network model. The monocular depth estimation method is based on the formula

further comprising determining the loss function using
where

represents the loss function, and

represents the reconstruction error, and

represents that the disparity map output by the first binocular matching network model is less biased than the disparity map output by the second binocular matching network model after training, and

represents that the output gradient constraining the first binocular matching network model matches the output gradient of the trained second binocular matching network model, and

6. The monocular depth estimation method according to claim 5 , wherein ? represents an intensity factor. The monocular depth estimation method is based on the formula

,or,

further comprising determining the reconstruction error using
where

represents the number of pixels in the image, and

represents the pixel values of the occlusion map output by the second binocular matching network model after training, and

represents the pixel value of the left image of the real binocular data without a depth label, and

represents the pixel value of the right image of the real binocular data without a depth label, and

represents the pixel value of the synthesized image after sampling the right image, and

represents the pixel value of the synthesized image after sampling the left image, and

represents the pixel values of the disparity map output by the first binocular matching network model of the left image of the real binocular data without depth labels, and

represents the pixel values of the disparity map output by the first binocular matching network model of the right image of the real binocular data without depth labels, and

7. A monocular depth estimation method according to claim 6 , wherein represents the pixel coordinates of a pixel point. The monocular depth estimation method is based on the formula

,or,

determining that the disparity map output by the first binocular matching network model is less biased than the disparity map output by the trained second binocular matching network model using ,
where

represents the number of pixels in the image, and

represents the pixel values of the occlusion map output by the second binocular matching network model after training, and

represents the pixel values of the disparity map output by the first binocular matching network model of the left image of the real binocular data without depth labels, and

represents the pixel values of the disparity map output by the first binocular matching network model of the right image of the real binocular data without depth labels, and

represents the pixel values of the disparity map output by the second binocular matching network model after training of the left image, and

represents the pixel values of the disparity map output by the second binocular matching network model after training for the right image, and

represents the pixel coordinates of the pixel point, and

7. The method of monocular depth estimation according to claim 6 , wherein ? represents an intensity factor. The monocular depth estimation method is based on the formula

,or,

determining that the output gradient of the first binocular matching network model matches the output gradient of the second binocular matching network model using
where

represents the number of pixels in the image, and

represents the gradient of the disparity map output by the first binocular matching network model of the left image of real binocular data without depth labels, and

represents the gradient of the disparity map output by the first binocular matching network model of the right image of real binocular data without depth labels, and

represents the gradient of the disparity map output by the second binocular matching network model after training of the left image, said

represents the gradient of the disparity map output by the second binocular matching network model after training on the right image, said

7. A monocular depth estimation method according to claim 6 , wherein represents the pixel coordinates of a pixel point. The depth-labeled real binocular data includes left and right images, for which the training process of the monocular depth estimation network model comprises:
obtaining a left image or a right image of the depth-labeled real binocular data as a training sample;
training a monocular depth estimation network model based on left or right images of the depth-labeled real binocular data ;
A monocular depth estimation method according to claim 3 , comprising: Wherein the unlabeled real binocular data includes left and right images, the training process of the monocular depth estimation network model comprises:
inputting the real binocular data without depth labels into the first binocular matching neural network model to obtain a corresponding disparity map;
Based on the lens base length of the camera that captures the corresponding parallax map and the real binocular data without the depth label and the lens focal length of the camera that captures the real binocular data without the depth label, the corresponding determining a depth map;
taking the left image or the right image of the real binocular data without depth labels as sample data, and teaching a monocular depth estimation network model based on the corresponding depth map of the disparity map, whereby the monocular depth estimation network model and
The monocular depth estimation method according to any one of claims 4 to 9 , comprising The analysis result of the image to be processed includes a disparity map output by the monocular depth estimation network model, and further comprising:
A disparity map output by the monocular depth estimation network model, a lens baseline length of a camera that captures an image that is input to the monocular depth estimation network model, and a lens of a camera that captures an image that is input to the monocular depth estimation network model determining a corresponding depth map of the disparity map based on the focal length;
outputting a corresponding depth map of the disparity map ;
The monocular depth estimation method according to claim 10 or 11 , comprising: A monocular depth estimation device, the monocular depth estimation device comprising:
an acquisition module configured to acquire an image to be processed;
an execution module configured to input the image to be processed into a trained monocular depth estimation network model to obtain an analysis result of the image to be processed, wherein the monocular depth estimation network model is configured to: an execution module obtained by supervised training with a disparity map output by an eye-matching neural network model;
an output module configured to output an analysis result of the image to be processed ;
including
The monocular depth estimator includes a first training module configured to train a second binocular matching neural network model based on the acquired synthetic sample data, and a trained second model based on the acquired real sample data. a second training module configured to adjust parameters of the binocular matching neural network model to obtain a first binocular matching neural network model;
The monocular depth estimation apparatus further includes a first acquisition module configured to acquire a depth-labeled synthesized binocular image including a synthesized left image and a synthesized right image as the synthesized sample data. ,
The monocular depth estimation device comprises: a building module configured to build a virtual 3D scene by a rendering engine; a mapping module configured to map the 3D scene as binocular images by two virtual cameras; a second acquisition configured to acquire depth data of the synthesized binocular image based on a position and orientation when constructing the virtual 3D scene and a lens focal length of the virtual camera; and a third acquisition module configured to label the binocular image based on the depth data to obtain the combined binocular image . a memory storing a computer program executable by a processor ;
A processor that executes the monocular depth estimation method according to any one of claims 1 to 12 by executing the computer program
monocular depth estimation equipment including ; A computer -readable storage medium storing a computer program, which, when executed by a processor, implements the monocular depth estimation method according to any one of claims 1 to 12 , computer readable storage medium; A computer program configured to cause a computer to perform the monocular depth estimation method according to any one of claims 1-12 .

Images