KR20210138788A - Scenario depth and camera motion prediction method and device, device, medium and program - Google Patents

Abstract

The present invention relates to a scenario depth and camera motion prediction method and apparatus, an electronic device, a medium and a program, the method comprising the steps of: acquiring a target image frame at time t; determining, by the scenario depth prediction network, a prediction depth map corresponding to the target image frame by performing scenario depth prediction on the target image frame using the first hidden state information at time t-1 - the first 1 The hidden state information includes feature information related to the scenario depth, and the scenario depth prediction network is obtained by auxiliary training based on the camera motion prediction network.

Description

Scenario depth and camera motion prediction method and device, device, medium and program

This application is filed on the basis of a Chinese patent application with an application number of 202010348872.2 and an filing date of April 28, 2020, and claims the priority of the Chinese patent application, all contents of the Chinese patent application are incorporated herein by reference. are cited

The present invention relates to the field of computer technology, and to a method and apparatus for predicting scenario depth and camera motion, an electronic device, a computer readable storage medium, and a computer program, but is not limited thereto.

Predicting scenario depth and camera motion using images acquired using a monocular image acquisition device (e.g., monocular camera) as input has been one of the active and important research directions in the field of computer vision over the past 20 years, It has wide application in many fields such as augmented reality, unmanned driving and mobile robot positioning and navigation.

An embodiment of the present invention provides a description method for a scenario depth and camera motion prediction method and apparatus, an electronic device, a medium, and a program.

A scenario depth prediction method provided in an embodiment of the present invention includes: obtaining a target image frame at time t; determining, by the scenario depth prediction network, a prediction depth map corresponding to the target image frame by performing scenario depth prediction on the target image frame using the first hidden state information at time t-1, , where the first hidden state information includes feature information related to the scenario depth, and the scenario depth prediction network is obtained by auxiliary training based on the camera motion prediction network.

In some embodiments of the present invention, through the scenario depth prediction network performing scenario depth prediction on the target image frame using the first hidden state information at time t-1, The determining of the predicted depth map may include performing feature extraction on the target image frame to determine a first feature map corresponding to the target image frame, wherein the first feature map is a feature map related to the scenario depth. - ; determining the first hidden state information at time t according to the first feature map and the first hidden state information at time t-1; and determining the predicted depth map according to the first hidden state information at time t.

In some embodiments of the present invention, the first hidden state information at time t-1 includes the first hidden state information under a different scale at time t-1; The step of determining a first feature map corresponding to the target image frame by performing feature extraction on the target image frame includes performing down-sampling of a plurality of scales on the target image frame to obtain the target image frame. determining the first feature map under a corresponding different scale; The determining of the first hidden state information at time t according to the first feature map and the first hidden state information at time t-1 includes, for an arbitrary scale, the first feature map under the scale and determining, according to the first hidden state information under the scale at time t-1, the first hidden state information under the scale at time t; The determining of the predicted depth map according to the first hidden state information at the time t includes: performing feature fusion to the first hidden state information under different scales at the time t to determine the predicted depth map includes

In some embodiments of the present invention, the method further comprises: obtaining a sequence of sample image frames corresponding to a time t, wherein the sequence of sample image frames includes a frame of a first sample image at a time t and the frame of the first sample image contains adjacent sample image frames of - ; determining, by the camera motion prediction network, a sample prediction camera motion corresponding to the sample image frame sequence by performing camera pose prediction on the sample image frame sequence using the second hidden state information at time t-1 - the second hidden state information includes characteristic information related to camera movement; A sample prediction depth map corresponding to the first sample image frame through which the scenario depth prediction network to be trained performs scenario depth prediction on the first sample image frame using the first hidden state information at time t-1 determining, wherein the first hidden state information includes feature information related to a scenario depth; constructing a loss function according to the sample prediction depth map and the sample prediction camera motion; and performing training on the to-be-trained scenario depth prediction network according to the loss function to obtain the scenario depth prediction network.

In some embodiments of the present invention, according to the sample prediction depth map and the sample prediction camera motion, building a loss function comprises: according to the sample prediction camera motion, the first sample image in the sample image frame sequence determining a reprojection error term for an adjacent sample image frame of a frame with respect to the first sample image frame; determining a penalty function term according to the distribution continuity of the sample prediction depth map; and constructing the loss function according to the reprojection error term and the penalty function term.

An embodiment of the present invention also provides a camera motion prediction method, comprising the steps of: obtaining a sequence of image frames corresponding to time t, wherein the sequence of image frames includes a target image frame at time t and an adjacent image frame of the target image frame Ham - ; and determining, by a camera motion prediction network, a predicted camera motion corresponding to the image frame sequence by performing camera pose prediction on the image frame sequence using the second hidden state information at time t-1. The second hidden state information includes feature information related to camera motion, and the camera motion prediction network is obtained by auxiliary training based on the scenario depth prediction network.

In some embodiments of the present invention, the camera motion prediction network performs camera pose prediction on the image frame sequence using the second hidden state information at time t-1, so that The determining of the predicted camera motion includes: performing feature extraction on the image frame sequence to determine a second feature map corresponding to the image frame sequence, the second feature map being a feature map related to the camera motion - ; determining the second hidden state information at time t according to the second image feature and the second hidden state information at time t-1; and determining the predicted camera movement according to the second hidden state information at time t.

In some embodiments of the present invention, the predicted camera movement comprises a relative pose between adjacent image frames in the sequence of image frames.

In some embodiments of the present invention, the method further comprises: obtaining a sequence of sample image frames corresponding to time t, wherein the sequence of sample image frames includes a first sample image frame at time t and a sequence of the first sample image frame at time t. Contains adjacent sample image frames - ; The scenario depth prediction network determines a sample prediction depth map corresponding to the first sample image frame by performing scenario depth prediction on the first sample image frame using the first hidden state information at time t-1. doing, wherein the first hidden state information includes feature information related to the scenario depth; A camera motion prediction network to be trained determines a sample prediction camera motion corresponding to the sample image frame sequence by performing camera pose prediction on the sample image frame sequence using the second hidden state information at time t-1. doing, wherein the second hidden state information includes characteristic information related to camera movement; constructing a loss function according to the sample prediction depth map and the sample prediction camera motion; and performing training on the to-be-trained camera motion prediction network according to the loss function to obtain the camera motion prediction network.

In some embodiments of the present invention, according to the sample prediction depth map and the sample prediction camera motion, building a loss function comprises: according to the sample prediction camera motion, the first sample image in the sample image frame sequence determining a reprojection error term for an adjacent sample image frame of a frame with respect to the first sample image frame; determining a penalty function term according to the distribution continuity of the sample prediction depth map; and constructing the loss function according to the reprojection error term and the penalty function term.

An embodiment of the present invention also provides a scenario depth prediction apparatus, comprising: a first acquiring module, configured to acquire a target image frame at time t; A first scenario, wherein the scenario depth prediction network is configured to determine a predicted depth map corresponding to the target image frame by performing scenario depth prediction on the target image frame using the first hidden state information at time t-1 and a depth prediction module, wherein the first hidden state information includes feature information related to a scenario depth, and the scenario depth prediction network is obtained by auxiliary training based on a camera motion prediction network.

In some embodiments of the present invention, the first scenario depth prediction module is configured to perform feature extraction on the target image frame to determine a first feature map corresponding to the target image frame. - the first feature map is a feature map related to the scenario depth; a second determining submodule, configured to determine the first hidden state information at time t according to the first feature map and the first hidden state information at time t-1; , a third determining submodule, configured to determine the predicted depth map according to the first hidden state information at time t.

In some embodiments of the present invention, the first hidden state information at time t-1 includes the first hidden state information under a different scale at time t-1; the first determining submodule is specifically configured to perform down-sampling of a plurality of scales on the target image frame to determine the first feature map under different scales corresponding to the target image frame; The second determining submodule is specifically, for any scale, according to the first feature map under the scale and the first hidden state information under the scale at time t-1, the second determination submodule under the scale at time t 1 configured to determine the hidden state information; The third determining submodule is specifically configured to perform feature fusion on the first hidden state information under a different scale of time t to determine the prediction depth map.

In some embodiments of the present invention, the device further comprises a first training module, wherein the first training module comprises:

obtain a sequence of sample image frames corresponding to time t, wherein the sequence of sample image frames includes a first sample image frame at time t and an adjacent sample image frame of the first sample image frame;

A camera motion prediction network determines a sample predicted camera motion corresponding to the sample image frame sequence by performing camera pose prediction on the sample image frame sequence using the second hidden state information at time t-1; the second hidden state information includes characteristic information related to camera movement;

A sample prediction depth map corresponding to the first sample image frame through which the scenario depth prediction network to be trained performs scenario depth prediction on the first sample image frame using the first hidden state information at time t-1 determine, wherein the first hidden state information includes feature information related to the scenario depth;

construct a loss function according to the sample prediction depth map and the sample prediction camera motion;

and perform training on the to-be-trained scenario depth prediction network according to the loss function to obtain the scenario depth prediction network.

In some embodiments of the present invention, the first training module is specifically configured to: according to the sample prediction camera movement, the first sample image frame and an adjacent sample image frame in the sample image frame sequence are the first sample image frames determine a reprojection error term for determine a penalty function term according to the distribution continuity of the sample prediction depth map; and build the loss function according to the reprojection error term and the penalty function term.

An embodiment of the present invention also provides an apparatus for predicting camera motion, wherein the apparatus for predicting camera motion is a second obtaining module, configured to obtain a sequence of image frames corresponding to time t, wherein the sequence of image frames includes a target image frame at time t and including an image frame adjacent to the target image frame; a first camera, configured to determine a predicted camera motion corresponding to the image frame sequence, through the camera motion prediction network performing camera pose prediction on the image frame sequence using the second hidden state information at time t-1 a motion prediction module, wherein the second hidden state information includes feature information related to camera motion, and the camera motion prediction network is obtained by auxiliary training based on a scenario depth prediction network.

In some embodiments of the present invention, the first camera motion prediction module is configured to perform feature extraction on the image frame sequence to determine a second feature map corresponding to the image frame sequence. - the second feature map is a feature map related to camera movement; a seventh determining submodule, configured to determine the second hidden state information at time t according to the second image feature and the second hidden state information at time t-1; and an eighth determining submodule configured to determine the predicted camera motion according to the second hidden state information at time t.

In some embodiments of the present invention, the predicted camera movement comprises a relative pose between adjacent image frames in the sequence of image frames.

In some embodiments of the present invention, the device further comprises a second training module, wherein the second training module comprises:

obtain a sequence of sample image frames corresponding to time t, wherein the sequence of sample image frames includes a first sample image frame at time t and an adjacent sample image frame of the first sample image frame;

The scenario depth prediction network determines a sample prediction depth map corresponding to the first sample image frame by performing scenario depth prediction on the first sample image frame using the first hidden state information at time t-1. and wherein the first hidden state information includes feature information related to the scenario depth;

A camera motion prediction network to be trained determines a sample prediction camera motion corresponding to the sample image frame sequence by performing camera pose prediction on the sample image frame sequence using the second hidden state information at time t-1. and, the second hidden state information includes characteristic information related to camera movement;

construct a loss function according to the sample prediction depth map and the sample prediction camera motion;

and perform training on the to-be-trained camera motion prediction network according to the loss function to obtain the camera motion prediction network.

In some embodiments of the present invention, the second training module is specifically configured to: according to the sample prediction camera movement, an adjacent sample image frame of the first sample image frame in the sample image frame sequence is the first sample image frame determine a reprojection error term for determine a penalty function term according to the sample prediction depth map and distribution continuity; and build the loss function according to the reprojection error term and the penalty function term.

An embodiment of the present invention also provides an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; Here, the processor is configured to execute the method by calling the instruction stored in the memory.

An embodiment of the present invention also provides a computer readable medium having computer program instructions stored thereon, wherein the computer program instructions implement the method when executed by a processor.

An embodiment of the present invention also provides a computer program comprising computer readable code, wherein when the computer readable code is run in an electronic device, a processor in the electronic device is executed to implement the any one method do.

In an embodiment of the present invention, a target image frame corresponding to time t is obtained, and since the scenario depth between adjacent times has a relation in time series, the first hidden state information related to the scenario depth at time t-1 is used. Accordingly, by performing scenario depth prediction on the target image frame through the scenario depth prediction network, it is possible to obtain a prediction depth map with higher prediction accuracy corresponding to the target image frame.

In the embodiment of the present invention, an image frame sequence including a target image frame at time t and an adjacent image frame of the target image frame corresponding to time t is obtained, and since the camera poses between adjacent times have a related relationship in the time series, , , by performing camera pose prediction on the image frame sequence through the camera motion prediction network using the second hidden state information related to the camera motion at time t-1, it is possible to obtain a predicted camera motion with high prediction accuracy.

It should be understood that the above general description and the following detailed description are illustrative and interpretative only, and are not intended to limit the present invention. Other features and aspects of the present invention will become more apparent as the exemplary embodiments are described in detail below in accordance with the accompanying drawings.

The drawings below constitute the entire specification as a part of this specification, and these drawings are for explaining the technical solution of the present invention together with the specification by illustrating an embodiment suitable for the present invention.
1 is a flowchart of a scenario depth prediction method in an embodiment of the present invention;
2 is a block diagram of a scenario depth prediction network in an embodiment of the present invention;
3 is a block diagram of unmonitored network training in an embodiment of the present invention;
4 is a flowchart of a camera motion prediction method according to an embodiment of the present invention;
5 is an exemplary structural diagram of a scenario depth prediction apparatus according to an embodiment of the present invention;
6 is a structural diagram of an apparatus for predicting camera motion according to an embodiment of the present invention;
7 is an exemplary structural diagram of an electronic device according to an embodiment of the present invention;
8 is an exemplary structural diagram of an electronic device according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Various exemplary embodiments, features and aspects of the present invention will be described in detail below with reference to the drawings. In the drawings, the same reference numbers indicate elements having the same or similar functions. While various aspects of the embodiments are illustrated in drawings, the drawings are not necessarily drawn to scale unless specifically stated otherwise.

The specific word “exemplary” herein means “used as an example, embodiment, or descriptive”. Any embodiment described herein as “exemplary” is not necessarily to be construed as superior or superior to other examples.

The term “and/or” in the text is only for describing the relation of the related object, and means that three relations may exist, for example, “A and/or D” means that A exists alone. or A and D are present simultaneously, or D is present alone. and, for example, at least one of A, C and D may represent any one or a plurality of elements selected from the set formed by A, C and D.

Further, in order to better illustrate the present invention, numerous details are provided in the specific embodiments below. It should be understood by those skilled in the art that the present invention may be practiced without some specific details. In some instances, methods, means, components, and circuits well known in the art have not been described in detail in order to emphasize the subject matter of the present invention.

1 is a flowchart of a scenario depth prediction method according to an embodiment of the present invention. As shown in FIG. 1 , the scenario depth prediction method may be executed by a terminal device or other processing device, where the terminal device is a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, or a wireless device. It may be a telephone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle device, a wearable device, and the like. The other processing device may be a server or a cloud server or the like. In some embodiments, the scenario depth prediction method may be implemented through a method in which a processor calls a computer readable instruction stored in a memory. 1 , the method may include the following steps.

In step S11, a target image frame at time t is acquired.

In step S12, the scenario depth prediction network determines a prediction depth map corresponding to the target image frame by performing scenario depth prediction on the target image frame using the first hidden state information at time t-1, 1 The hidden state information includes feature information related to the scenario depth, and the scenario depth prediction network is obtained by auxiliary training based on the camera motion prediction network.

In an embodiment of the present invention, acquiring the target image frame at time t is obtained by using the first hidden state information related to the scenario depth at time t-1 because the scenario depth between adjacent times has a relation in time series. , by performing scenario depth prediction on the target image frame through the scenario depth prediction network, it is possible to obtain a prediction depth map with higher prediction accuracy corresponding to the target image frame.

In some embodiments, the scenario depth prediction network determines a prediction depth map corresponding to the target image frame by performing scenario depth prediction on the target image frame using the first hidden state information at time t-1. The step includes: performing feature extraction on the target image frame to determine a first feature map corresponding to the target image frame, the first feature map being a feature map related to the scenario depth; determining first hidden state information at time t according to the first feature map and the first hidden state information at time t-1; and determining a prediction depth map according to the first hidden state information at time t.

Since the scenario depths between adjacent times have a related relationship in the time series, the scenario depth prediction network provides a first feature map associated with the scenario depth corresponding to the target image frame at the current time (eg, time t), and the previous time ( For example, at t-1 time), the first hidden state information related to the current time and the scenario depth may be determined using the first hidden state information related to the scenario depth, so that the current time and the scenario depth related to the current time and the scenario depth may be determined. By performing scenario depth prediction on the target image frame based on the first hidden state information, a prediction depth map having higher prediction accuracy corresponding to the target image frame at the current time may be obtained.

For example, when predicting the predicted depth map corresponding to each image frame in the image frame sequence (including the image frame from the first time to the t-th time) using the scenario depth prediction network, in the initialization step of the scenario depth prediction network , set a preset initial value of the first hidden state information related to the scenario depth. Based on the preset initial value of the first hidden state information and the first feature map related to the scenario depth corresponding to the image frame at the first time, by determining the first hidden state of the first time, 1 perform scenario depth prediction on the image frame at the first time based on the hidden state to obtain a predicted depth map corresponding to the image frame at the first time; Based on the first hidden state at the first time and the first feature map associated with the scenario depth corresponding to the image frame at the second time, by determining the first hidden state at the second time, the first concealment of the second time performing scenario depth prediction on the image frame at the second time based on the state to obtain a predicted depth map corresponding to the image frame at the second time; Based on the first hidden state at the second time and the first feature map associated with the scenario depth corresponding to the image frame at the third time, by determining the first hidden state at the third time, the first concealment of the third time performing scenario depth prediction on the image frame at the third time based on the state to obtain a predicted depth map corresponding to the image frame at the third time; By sequential analogy in this way, a predicted depth map corresponding to each image frame is obtained in the final image frame sequence (including the image frame from the first time to the t-th time).

In some embodiments, the first hidden state information at time t-1 includes first hidden state information under a different scale at time t-1; The step of performing feature extraction on the target image frame to determine the first feature map corresponding to the target image frame may include performing a plurality of scale down-sampling on the target image frame under different scales corresponding to the target image frame. determining a first feature map; The determining of the first hidden state information at time t according to the first feature map and the first hidden state information at time t-1 may include, in the case of an arbitrary scale, the first feature map under the scale and time t-1 determining, according to the first hidden state information under the scale of , the first hidden state information under the scale at time t; Determining the predicted depth map according to the first hidden state information at time t may include performing feature fusion on the first hidden state information under different scales at time t to determine the predicted depth map have.

,제2 스케일 하의 제1 특징 맵

과 제3 스케일 하의 제1 특징 맵

이다. 여기서, 다양한 스케일 ConvGRU은 다양한 스케일 제1 특징 맵의 스케일과 대응되며, 즉, 다양한 스케일 ConvGRU은, 제1 스케일 하의 ConvGRU⁰,제2 스케일 하의 ConvGRU¹및 제3 스케일 하의 ConvGRU²을 포함한다.In order to better determine the prediction depth map corresponding to the target image frame at time t, the scenario depth prediction network can adopt various scale feature fusion mechanisms. 2 shows a block diagram of a scenario depth prediction network according to an embodiment of the present invention, and as shown in FIG. 2 , the scenario depth prediction network includes a depth encoder 202, various scale convolutional gate recursive units (Convolutional Gated). Recurrent Unit, ConvGRU) and a depth decoder 205 are included. input the target image frame 201 at time t into the depth encoder 202 to perform down-sampling of various scales to obtain a first feature map 203 under different scales corresponding to the target image frame; That is, the first feature map under the first scale

, the first feature map under the second scale

and a first feature map under a third scale

am. Here, the various-scale ConvGRU corresponds to the scale of the various-scale first feature map, that is, the various-scale ConvGRU includes ConvGRU ^{0 under the first scale, ConvGRU 1} under the second scale, ^{and ConvGRU 2} under the third scale.

을 ConvGRU⁰에 입력하고, 제1 특징 맵

을 ConvGRU¹에 입력하며, 제1 특징 맵

을 ConvGRU²에 입력한다. ConvGRU⁰은 제1 특징 맵

과 ConvGRU⁰에 저장된 t-1 시각의 제1 스케일 하의 제1 은닉 상태 정보

에 대해 특징 융합을 수행하여, t 시각의 제1 스케일 하의 제1 은닉 상태

을 얻고, ConvGRU⁰은 t 시각의 제1 스케일 하의 제1 은닉 상태

에 대해 저장을 수행하고, 및 t 시각의 제1 스케일 하의 제1 은닉 상태

를 깊이 디코더로 출력하며; ConvGRU¹은 제1 특징 맵

과 ConvGRU¹에 저장된 t-1 시각의 제2 스케일 하의 제1 은닉 상태 정보

에 대해 특징 융합을 수행하여, t 시각의 제2 스케일 하의 제1 은닉 상태

를 얻고, ConvGRU¹은 t 시각의 제2 스케일 하의 제1 은닉 상태

에 대해 저장을 수행하고, 및 t 시각의 제2 스케일 하의 제1 은닉 상태

를 깊이 디코더로 출력하며; ConvGRU²은 제1 특징 맵

과 ConvGRU²에 저장된 t-1 시각의 제3 스케일 하의 제1 은닉 상태 정보

에 대해 특징 융합을 수행하여, t 시각의 제3 스케일 하의 제1 은닉 상태

를 얻고, ConvGRU²은 t 시각의 제3 스케일 하의 제1 은닉 상태

에 저장을 수행하며, 및 t 시각의 제3 스케일 하의 제1 은닉 상태

를 깊이 디코더로 출력한다. 도 2에서, 다양한 스케일의 은닉 상태(204)는 t 시각의 제1 스케일 하의 제1 은닉 상태

,제2 스케일 하의 제1 은닉 상태

및 제3 스케일 하의 제1 은닉 상태

를 포함한다.Taking FIG. 2 as an example, the first feature map

is input to ConvGRU ⁰ , and the first feature map

is input into ConvGRU ¹ , the first feature map

is entered into ConvGRU ^2. ConvGRU ⁰ is the first feature map

and the first hidden state information under the first scale of time t-1 stored in ^{ConvGRU 0}

Perform feature fusion on , the first hidden state under the first scale at time t

, and ConvGRU ⁰ is the first hidden state under the first scale at time t.

perform storage on , and a first hidden state under a first scale at time t

to the depth decoder; ConvGRU ¹ is the first feature map

and the first hidden state information under the second scale of time t-1 stored in ^{ConvGRU 1}

Perform feature fusion on , so that the first hidden state under the second scale at time t is

, and ConvGRU ¹ is the first hidden state under the second scale at time t.

perform storage on , and a first hidden state under a second scale at time t

to the depth decoder; ConvGRU ² is the first feature map

and the first hidden state information under the third scale of time t-1 stored in ^{ConvGRU 2}

Feature fusion is performed on the first hidden state under the third scale at time t.

, and ConvGRU ² is the first hidden state under the third scale at time t

perform storage on , and the first hidden state under the third scale at time t

is output to the depth decoder. In Fig. 2, the various scales of the hidden state 204 are the first hidden states under the first scale at time t.

, the first hidden state under the second scale

and a first hidden state under a third scale

includes

,제2 스케일 하의 제1 은닉 상태

및 제3 스케일 하의 제1 은닉 상태

의 스케일을 타깃 이미지 프레임(201)의 스케일(타깃 이미지 프레임의 스케일을 타깃 스케일로 약칭한다.)과 동일하게 회복하여, t 시각의 타깃 스케일 하의 세 개 제1 은닉 상태를 얻는다. 제1 은닉 상태 정보가 시나리오 깊이와 관련되는 특징 정보를 포함하므로, 시나리오 깊이 예측 네트워크에서도 특징 맵의 형식으로 존재하며, 따라서, t 시각의 타깃 스케일 하의 세 개 제1 은닉 상태에 대해 특징 맵 융합을 수행함으로써, t 시각의 타깃 이미지 프레임에 대응되는 예측 깊이 맵

을 얻는다.The depth decoder 205 each has a first hidden state under a first scale at time t.

, the first hidden state under the second scale

and a first hidden state under a third scale

The scale of is restored equal to the scale of the target image frame 201 (the scale of the target image frame is abbreviated as the target scale) to obtain three first hidden states under the target scale at time t. Since the first hidden state information includes feature information related to the scenario depth, it also exists in the form of a feature map in the scenario depth prediction network, and thus, feature map fusion is performed for the three first hidden states under the target scale at time t. By performing the prediction depth map corresponding to the target image frame at time t

get

In some embodiments, the scenario depth prediction method further comprises: obtaining a sample image frame sequence corresponding to a time t, wherein the sample image frame sequence is adjacent to the frame of the first sample image at time t and the frame of the first sample image Contains sample image frames - ; determining, by the camera motion prediction network, a sample prediction camera motion corresponding to the sample image frame sequence by performing camera pose prediction on the sample image frame sequence using the second hidden state information at time t-1; 2 The hidden state information includes characteristic information related to camera movement; The scenario depth prediction network to be trained determines a sample prediction depth map corresponding to the first sample image frame by performing scenario depth prediction on the first sample image frame using the first hidden state information at time t-1. a step of - wherein the first hidden state information includes feature information related to the scenario depth; constructing a loss function according to the sample prediction depth map and the sample prediction camera motion; and performing training on the scenario depth prediction network to be trained according to the loss function to obtain a scenario depth prediction network.

In an embodiment of the present invention, the scenario depth prediction network is obtained by auxiliary training based on the camera motion prediction network, or the scenario depth prediction network and the camera motion prediction network are obtained through joint training. Using the relationship between the scenario depth and the camera pose between adjacent times in the time series, a mechanism of sliding window data fusion is introduced, and hidden state information related to the scenario depth and camera movement at the target time (time t) in the sliding window sequence. By extracting and remembering , non-monitoring network training is performed on the scenario depth prediction network and/or the camera motion prediction network.

In an embodiment of the present invention, a training set may be generated in advance, and the training set includes a sequence of sample image frames obtained by successively adopted in a time series, thereby training the scenario depth prediction network to be trained based on the training set. carry out 3 shows a block diagram of non-monitoring network training in an embodiment of the present invention. As shown in FIG. 3 , the target time is time t, and the sample image frame sequence 301 corresponding to the target time (that is, the sample image frame sequence included in the sliding window corresponding to the target time) is at time t. One sample image frame I _t , includes an adjacent sample image frame I _{t-1 at time t-1} and an adjacent sample image frame I _t+1 at time t+1. The number of adjacent sample image frames of the first sample image frame in the sample image frame sequence may be determined according to an actual situation, and is not limited in the present invention.

을 얻음으로써, 제1 특징 맵

을 ConvGRU에 입력하여, 제1 특징 맵

과 ConvGRU에 저장된 t-1 시각의 제1 은닉 상태 정보

에 대해 특징 융합을 수행하여, t 시각의 제1 은닉 상태

를 얻고, ConvGRU가 t 시각의 제1 은닉 상태

에 대해 저장을 수행하고, t 시각의 제1 은닉 상태

를 깊이 디코더(205)에 출력함으로써, t 시각의 제1 샘플 이미지 프레임에 대응되는 샘플 예측 깊이 맵

을 얻는다.The single-scale feature fusion mechanism adopted by the to-be-trained scenario depth prediction network shown in Fig. 3 is a single-scale feature fusion mechanism. In the network training process, the scenario depth prediction network to be trained may adopt the single-scale feature fusion mechanism shown in FIG. 3, and may also use various scale feature fusion mechanisms shown in FIG. 2, which is not limited in the present invention. . As shown in FIG. 3 , the scenario depth prediction network to be trained includes a depth encoder 202 , a ConvGRU and a depth decoder 205 . A first feature map corresponding to the first _{sample image frame I t} by inputting the first sample image frame I _t at time t to the depth encoder 202 and performing feature extraction

By obtaining , the first feature map

is input into ConvGRU, the first feature map

and the first hidden state information at time t-1 stored in ConvGRU

By performing feature fusion on , the first hidden state at time t

, and ConvGRU is the first hidden state at time t

store on , and the first hidden state at time t

by outputting to the depth decoder 205, the sample prediction depth map corresponding to the first sample image frame at time t

get

을 얻음으로써, 제2 특징 맵

을 ConvGRU에 입력하여, 제2 특징 맵

과 ConvGRU에 저장된 t-1 시각의 제2 은닉 상태 정보

에 대해 특징 융합을 수행하여, t 시각의 제2 은닉 상태

를 얻고, ConvGRU는 t 시각의 제2 은닉 상태

에 대해 저장을 수행하고, t 시각의 제2 은닉 상태

를 포즈 디코더로 출력함으로써, t 시각의 샘플 이미지 프레임 시퀀스에 대응되는 샘플 예측 카메라 움직임[

,

]을 얻는다.Taking FIG. 3 as an example, as shown in FIG. 3 , a pause encoder 302 , a ConvGRU and a pause decoder 303 are included in the camera motion prediction network. A second feature map corresponding to the sample image frame sequence by inputting the sample image frame sequence [I _t ,I _t-1 ,I _{t+1 ] corresponding to time t to the pause encoder 302 and performing feature extraction}

By obtaining , the second feature map

is input into ConvGRU, and the second feature map

and the second hidden state information at time t-1 stored in ConvGRU

By performing feature fusion on , the second hidden state at time t

, and ConvGRU is the second hidden state at time t

store on , and the second hidden state at time t

By outputting to the pose decoder, the sample prediction camera motion [

,

] is obtained.

과 샘플 예측 카메라 움직임[

,

]에 따라, 손실 함수

를 구축할 수 있다. 구체적으로, 샘플 예측 카메라 움직임[

,

]에 따라, 샘플 이미지 프레임 시퀀스에서의 인접 샘플 이미지 프레임I_t-1및 I_t+1이 제1 샘플 이미지 프레임I_t에 대한 재투영 오차항

을 결정하고; 샘플 예측 깊이 맵

의 분포 연속성에 따라, 벌칙 함수항

을 결정한다. 하기 공식(1)을 통해 손실 함수

를 구축하며;Taking Fig. 3 as an example, the sample prediction depth map

and sample prediction camera motion[

,

] according to the loss function

can be built Specifically, sample prediction camera motion [

,

], so that adjacent sample image frames I _t-1 and I _t+1 in the sample image frame sequence are reprojection error terms for the first sample image frame I _{t .}

to determine; Sample prediction depth map

According to the distribution continuity of

to decide The loss function through the following formula (1)

to build;

(1).

(One).

는 가중 파라미터이며, 실제 상황에 따라

의 값의 크기를 결정할 수 있으며, 본 발명은 이에 더이상 구체적으로 한정하지 않는다.here,

is a weighting parameter, according to the actual situation

The magnitude of the value of can be determined, and the present invention is not specifically limited thereto.

의 분포 연속성에 따라, 벌칙 함수항

를 결정하는 구체적 과정은, 제1 샘플 이미지 프레임I_t에서 각 픽셀 포인트의 경사도 값을 결정하고, 각 픽셀 포인트의 경사도 값은 제1 샘플 이미지 프레임I_t의 분포 연속성(평활성이라고 할 수 있음)을 반영할 수 있으며, 따라서, 각 픽셀 포인트의 경사도 값에 따라 제1 샘플 이미지 프레임I_t에서의 경계 영역(경사도 값은 임계값의 픽셀 포인트 구성의 영역보다 크거나 같음)과 비경계 영역(경사도 값은 임계값의 픽셀 포인트 구성의 영역보다 작음)을 결정할 수 있으므로써, 제1 샘플 이미지 프레임I_t에 대응되는 샘플 예측 깊이 맵

에서의 경계 영역과 비경계 영역을 결정할 수 있고; 샘플 예측 깊이 맵

에서 각 픽셀 포인트의 경사도 값을 결정하며, 샘플 예측 깊이 맵

에서 비 경계 영역의 분포 연속성 및 경계 영역의 분포 불연속성을 보장하기 위해, 샘플 예측 깊이 맵

에서 비경계 영역에서의 각 픽셀 포인트에 대해, 경사도 값과 정비례하는 벌칙 인자를 설정하고; 샘플 예측 깊이 맵

에서 경계 영역에서의 각 픽셀 포인트에 대해, 경사도 값과 반비례하는 벌칙 인자를 설정함으로써; 샘플 예측 깊이 맵

에서 각 픽셀 포인트의 벌칙 인자에 기반하여, 벌칙 함수항

을 구축한다.In some embodiments, the sample prediction depth map

According to the distribution continuity of

Specifically the process is, the slope value of the first sample image frame, each pixel point, each pixel point determines the slope value, and in I _t is (which may be described as smoothness) distribution continuity of the first sample image frame I _t to determine the Thus, according to the gradient value of each pixel point, the boundary region (the gradient value is greater than or equal to the region of the pixel point configuration of the threshold) and the non-boundary region (the gradient value) in _{the first sample image frameI t .} is smaller than the area of the pixel point configuration of the threshold value), so that the sample prediction depth map corresponding to _{the first sample image frameI t .}

can determine a boundary region and a non-boundary region in ; Sample prediction depth map

Determine the slope value of each pixel point in the sample prediction depth map

To ensure the distribution continuity of the non-boundary region and the distribution discontinuity of the boundary region in the sample prediction depth map

for each pixel point in the non-boundary region in , set a penalty factor that is directly proportional to the gradient value; Sample prediction depth map

by setting a penalty factor inversely proportional to the gradient value for each pixel point in the boundary region in ; Sample prediction depth map

Based on the penalty factor of each pixel point in

to build

Since the sample prediction depth map and the sample prediction camera motion are obtained using the time series relation between the scenario depth and the camera motion between adjacent views, the reprojection error term determined according to the predicted camera motion obtained by the camera motion prediction network, and the scenario depth prediction By synthetically using the loss function constructed with the penalty function term determined according to the prediction depth map obtained with the network, the training is performed on the scenario depth prediction network to be trained, and the scenario depth prediction network obtained by training improves the prediction accuracy of the scenario depth prediction. can be improved

In some embodiments, the camera motion prediction network in FIG. 3 may be a camera motion prediction network to be trained, and according to the loss function, training is performed on the to-be-trained camera motion network to obtain a scenario depth prediction network to be trained and By implementing federated training on the camera motion network to be trained, we obtain a trained scenario depth prediction network and a camera motion prediction network.

Since the predicted depth map and the predicted camera motion are obtained using the time series relation between the scenario depth and the camera motion between adjacent views, the reprojection error term determined according to the predicted camera motion obtained by the camera motion prediction network, and the scenario depth prediction network are By comprehensively using the loss function constructed with the penalty function term determined according to the obtained prediction depth map, joint training is performed on the scenario depth prediction network and the camera motion prediction network, and the obtained scenario depth prediction network and the camera motion prediction network are to improve the prediction accuracy of scenario depth prediction and camera motion prediction.

In some embodiments, the depth encoder and the pause encoder may reuse the ResNet18 structure, reuse the ResNet54 structure, and may reuse other structures, although the present invention is not specifically limited thereto. The depth decoder and the pause decoder may adopt the Unet network structure, and may also adopt other decoder network structures, but the present invention is not specifically limited thereto.

In some embodiments, the ConvGRU includes a convolution operation, and the activation function in the ConvGRU is an ELU activation function.

For example, it can be improved through the convolution gate recursive unit ConvGRU, which can perform data processing only on one-dimensional data, replace the linear operation in ConvGRU with the convolution operation, and change the tanh activation function in ConvGRU to the ELU By substituting with the activation function, ConvGRU capable of performing data processing on two-dimensional image data is obtained.

Using that scenario depth and/or camera movement have a relation in time series, image frame sequences corresponding to different times are cyclically convolved according to time series through ConvGRU, and the first hidden state corresponding to different times and / or obtain a second hidden state.

In order to implement the mechanism of sliding window data fusion, in addition to adopting the ConvGRU, Convolutional Long Short-Term Memory (ConvLSTM) can be adopted, and also a structure that can implement other sliding window data fusion. may be adopted, and the present invention is not specifically limited thereto.

4 is a flowchart according to a method for predicting camera motion according to an embodiment of the present invention. As shown in FIG. 4 , the method for predicting camera motion may be executed by a terminal device or other processing device, where the terminal device is a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, or a wireless device. It may be a telephone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle device, a wearable device, and the like. The other processing device may be a server or a cloud server or the like. In some possible implementation manners, the camera motion prediction method may be implemented through a method in which a processor calls a computer readable instruction stored in a memory. As shown in Figure 4, the method also

obtaining an image frame sequence corresponding to time t S41, wherein the image frame sequence includes a target image frame at time t and an adjacent image frame of the target image frame;

Step S42 - Second concealment of the camera motion prediction network by performing camera pose prediction on the image frame sequence using the second hidden state information at time t-1 The state information includes feature information related to camera motion, and the camera motion prediction network is obtained by auxiliary training based on the scenario depth prediction network.

In an embodiment of the present invention, the step of obtaining an image frame sequence including a target image frame at time t and an adjacent image frame of the target image frame includes: since camera motion between adjacent times has a related relationship in time series, t- 1 Using the second hidden state information related to the camera motion, the camera pose prediction is performed on the image frame sequence through the camera motion prediction network, and the predicted camera motion corresponding to the image frame sequence has higher prediction accuracy. can be obtained

In some embodiments, the camera motion prediction network determines the predicted camera motion corresponding to the image frame sequence by performing camera pose prediction on the image frame sequence using the second hidden state information at time t-1. The step includes: performing feature extraction on the image frame sequence to determine a second feature map corresponding to the image frame sequence, the second feature map being a feature map related to camera motion; determining second hidden state information at time t according to the second image feature and second hidden state information at time t-1; and determining the predicted camera movement according to the second hidden state information at time t.

The camera motion between adjacent times has a relation in time series, so that the camera motion prediction network has a second feature map related to the scenario depth corresponding to the sequence of image frames at time t, and a second feature map related to camera motion at time t-1. 2 By using the hidden state information, second hidden state information related to the camera motion at time t can be determined, so that the image frame sequence at time t is based on the second hidden state information related to the camera motion at time t. By performing the camera motion prediction for the time t, it is possible to obtain a prediction depth map with higher prediction accuracy corresponding to the image frame sequence at time t.

In some embodiments, predictive camera movement may include relative poses between adjacent image frames in a sequence of image frames. Here, the relative pose is a six-dimensional parameter, and includes three-dimensional rotation information and three-dimensional translation information.

,

]에는 인접 이미지 프레임I_t-1에서 타깃 이미지 프레임I_t사이의 상대 포즈

,및 타깃 이미지 프레임I_t에서 인접 이미지 프레임I_t+1사이의 상대 포즈

가 포함된다.For example, predictive camera movement[

,

] contains the relative poses between adjacent image framesI _t-1 and target image framesI _t

, and the relative poses between the target image frameI _t and adjacent image frames I _t+1

is included

을 얻음으로써, 제2 특징 맵

을 ConvGRU에 입력하여, 제2 특징 맵

과 ConvGRU에 저장된 t-1 시각의 제2 은닉 상태 정보

에 대해 특징 융합을 수행하여, t 시각의 제2 은닉 상태

를 얻고, ConvGRU는 t 시각의 제2 은닉 상태

에 대해 저장을 수행하고, 및 t 시각의 제2 은닉 상태

를 포즈 디코더에 출력함으로써, t 시각의 이미지 프레임 시퀀스에 대응되는 예측 카메라 움직임[

,

]을 얻는다.Taking FIG. 3 as an example, as shown in FIG. 3 , the camera motion prediction network includes a pause encoder, a ConvGRU, and a pause decoder. The image frame sequence [I _t ,I _t-1 ,I _t+1 ] corresponding to time t is input to the pause encoder 302 and feature extraction is performed to perform feature extraction, and thus a second feature map corresponding to the image frame sequence

By obtaining , the second feature map

is input into ConvGRU, and the second feature map

and the second hidden state information at time t-1 stored in ConvGRU

By performing feature fusion on , the second hidden state at time t

, and ConvGRU is the second hidden state at time t

perform storage on , and the second hidden state at time t

By outputting to the pose decoder, the predicted camera motion corresponding to the sequence of image frames at time t[

,

] is obtained.

For example, when predicting the predicted camera motion corresponding to the image frame sequence using the camera motion prediction network, in the initialization step of the camera motion prediction network, a preset initial value of the second hidden state information related to the camera motion is set do. By determining the second hidden state of the first time based on the preset initial value of the second hidden state information and the second feature map related to the camera motion corresponding to the image frame sequence at the first time, performing camera motion prediction on the image frame sequence at the first time based on the second hidden state to obtain a predicted camera motion corresponding to the image frame sequence at the first time; determining a second hidden state at a second time based on a second hidden state at the first time and a second feature map associated with camera movement corresponding to the sequence of image frames at the second time; performing camera motion prediction on the image frame sequence at the second time based on the hidden state to obtain a predicted camera motion corresponding to the image frame sequence at the second time; determining a second hidden state at the third time, based on the second hidden state at the second time and a second feature map associated with camera motion corresponding to the sequence of image frames at the third time, by determining the second hidden state at the third time performing camera motion prediction on the image frame sequence at the third time based on the hidden state to obtain a predicted camera motion corresponding to the image frame sequence at the third time; By sequential analogy in this way, the predicted camera motion corresponding to the sequence of image frames at the final different time is obtained.

In some embodiments, the method for predicting camera motion further comprises: obtaining a sequence of sample image frames corresponding to time t, the sample image frame sequence being a first sample image frame at time t and adjacent samples of the first sample image frame at time t Contains image frames - ; Determining, by the scenario depth prediction network, a prediction depth map corresponding to the first sample image frame by performing scenario depth prediction on the target image frame using the first hidden state information at time t-1 - first The hidden state information includes feature information related to the scenario depth; determining, by the camera motion prediction network to be trained, a sample prediction camera motion corresponding to the sample image frame sequence, by performing camera pose prediction on the sample image frame sequence using the second hidden state information at time t-1 - the second hidden state information includes characteristic information related to camera movement; building a loss function according to the sample prediction depth map and the sample prediction camera motion; and obtaining a camera motion prediction network by performing training on the to-be-trained camera motion prediction network according to the loss function.

In some embodiments, according to the sample-predicted depth map and the sample-predicted camera movement, building the loss function comprises: according to the sample-predicted camera movement, adjacent sample image frames of the first sample image frame in the sample image frame sequence. 1 determining a reprojection error term for a sample image frame; determining a penalty function term according to the distribution continuity of the sample prediction depth map; and constructing a loss function according to the reprojection error term and the penalty function term.

In an embodiment of the present invention, the camera motion prediction network is obtained by auxiliary training based on the scenario depth prediction network, or the scenario depth prediction network and the camera motion prediction network are obtained through joint training. In some embodiments, training may be performed on a camera motion prediction network to be trained based on FIG. 3 , and in the training process, the camera motion prediction network in FIG. 3 is a camera motion prediction network to be trained, FIG. The scenario depth prediction network in the to-be-trained scenario depth prediction network (by jointly training the to-be-trained scenario depth prediction network and the to-be-trained camera motion prediction network) may also be a trained scenario depth prediction network (the to-be-trained camera motion prediction network). may be performed alone), the specific training process and FIG. 3 are the same, and the embodiment of the present invention will not be described any further.

The predicted depth map and the predicted camera motion are obtained using the time series relation between the scenario depth and the camera motion between adjacent views, and the reprojection error term determined according to the predicted camera motion obtained by the camera motion prediction network, and the scenario depth prediction By comprehensively using the loss function constructed with the penalty function term determined according to the prediction depth map obtained by the network, joint training is performed on the scenario depth prediction network and the camera motion prediction network to predict the scenario depth prediction network and the camera motion obtained by training. It allows the network to improve the prediction accuracy of scenario depth prediction and camera motion prediction.

In an embodiment of the present invention, the scenario depth prediction network and the camera motion prediction network obtained by training through the network training method shown in FIG. 3 may perform environment depth prediction and three-dimensional scenario construction. For example, by applying the scenario depth prediction network to the navigation scenarios of indoor and outdoor mobile robots such as vacuum cleaners and lawnmowers, and acquiring RGB images through RGB (Red Green Blue) cameras, the scenario depth prediction network is used to By determining the predicted depth map corresponding to the RGB image, and determining the camera movement of the RGB camera using the camera prediction network, distance measurement and 3D scenario construction for obstacles are implemented, and obstacle avoidance and navigation tasks are completed.

It is to be understood that each of the method embodiments mentioned in the present invention can be combined with each other to form a combined embodiment, as long as the principle logic is not violated, and due to the limitation of the width, the description is no longer repeated in the present invention. I never do that. A person skilled in the art will know that in the above method of the specific embodiment, the recording order of each step does not imply a strict execution order, and the specific execution order of each step is determined by its function and possible internal logic.

In addition, the present invention further provides a scenario depth/camera motion prediction device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any one scenario depth/camera motion prediction method provided by the present invention. and corresponding technical solutions and descriptions and corresponding descriptions of reference method parts, which are not further repeated herein.

5 is a block diagram of a scenario depth prediction apparatus according to an embodiment of the present invention. As shown in FIG. 5 , the scenario depth prediction apparatus 50 includes:

a first acquiring module 51, configured to acquire a target image frame at time t; and

A first scenario depth prediction module 52, wherein the network performs scenario depth prediction on the target image frame through using the first hidden state information at time t-1 to determine a predicted depth map corresponding to the target image frame - The first hidden state information includes feature information related to the scenario depth, and the scenario depth prediction network is obtained by auxiliary training based on the camera motion prediction network - ; includes

In some embodiments, the first scenario depth prediction module 52 comprises:

a first determining submodule, configured to perform feature extraction on the target image frame to determine a first feature map corresponding to the target image frame, wherein the first feature map is a feature map related to the scenario depth;

a second determining submodule, configured to determine, according to the first feature map and the first hidden state information at time t-1, the first hidden state information at time t; and

and a third determining submodule, configured to determine, according to the first hidden state information at time t, a prediction depth map.

In some embodiments, the first hidden state information at time t-1 includes first hidden state information under a different scale at time t-1;

the first determining submodule is specifically configured to perform a plurality of scale downsampling on the target image frame to determine a first feature map under a different scale corresponding to the target image frame;

The second determining submodule is specifically, for any scale, according to the first feature map under the scale and the first hidden state information under the scale at time t-1, the first hidden state information under the scale at time t is configured to determine;

The third determining submodule is specifically configured to perform feature fusion on the first hidden state information under a different scale of time t, to determine the prediction depth map.

In some embodiments, the scenario depth prediction apparatus 50 also includes a first training module, the first training module comprising:

obtain a sequence of sample image frames corresponding to time t, wherein the sequence of sample image frames includes a first sample image frame at time t and an adjacent sample image frame of the first sample image frame;

A camera motion prediction network determines a sample predicted camera motion corresponding to the sample image frame sequence by performing camera pose prediction on the sample image frame sequence using the second hidden state information at time t-1; the second hidden state information includes characteristic information related to camera movement;

A sample prediction depth map corresponding to the first sample image frame through which the scenario depth prediction network to be trained performs scenario depth prediction on the first sample image frame using the first hidden state information at time t-1 determine, wherein the first hidden state information includes feature information related to the scenario depth;

construct a loss function according to the sample prediction depth map and the sample prediction camera motion;

and perform training on the to-be-trained scenario depth prediction network according to the loss function to obtain the scenario depth prediction network.

In some embodiments, the first training module is specifically configured to: according to the sample prediction camera movement, an adjacent sample image frame of the first sample image frame in the sample image frame sequence is re-projected onto the first sample image frame determine an error term; determine a penalty function term according to the sample prediction depth map and distribution continuity; and construct the loss function according to the reprojection error term and the penalty function term.

6 is a block diagram of an apparatus for predicting camera motion according to an embodiment of the present invention. As shown in Figure 6, the camera motion prediction device 60,

a second acquiring module 61, configured to acquire an image frame sequence corresponding to time t, wherein the image frame sequence includes a target image frame at time t and an adjacent image frame of the target image frame; and

a first camera motion prediction module 62, wherein the network performs camera pose prediction on the image frame sequence through using the second hidden state information at time t-1 to determine a predicted camera motion corresponding to the image frame sequence - The second hidden state information includes feature information related to camera motion, and the camera motion prediction network is obtained by auxiliary training based on the scenario depth prediction network.

In some embodiments, the first camera motion prediction module 62 includes:

a sixth determining submodule, configured to perform feature extraction on the image frame sequence to determine a second feature map corresponding to the image frame sequence, wherein the second feature map is a feature map related to camera motion;

a seventh determining submodule, configured to determine, according to the second image feature and the second hidden state information at time t-1, the second hidden state information at time t; and

an eighth determining submodule, configured to determine, according to the second hidden state information at time T, the predicted camera movement.

In some embodiments, the predictive camera movement includes relative poses between adjacent image frames in a sequence of image frames.

In some embodiments, the camera motion prediction device 60 also includes a second training module, the second training module comprising:

obtain a sequence of sample image frames corresponding to time t, wherein the sequence of sample image frames includes a first sample image frame at time t and an adjacent sample image frame of the first sample image frame;

The scenario depth prediction network determines a sample prediction depth map corresponding to the first sample image frame by performing scenario depth prediction on the first sample image frame using the first hidden state information at time t-1. and, the first hidden state information includes feature information related to the scenario depth;

A camera motion prediction network to be trained determines a sample prediction camera motion corresponding to the sample image frame sequence by performing camera pose prediction on the sample image frame sequence using the second hidden state information at time t-1. and, the second hidden state information includes characteristic information related to camera movement;

construct a loss function according to the sample prediction depth map and the sample prediction camera motion; and

and perform training on the to-be-trained camera motion prediction network according to the loss function to obtain the camera motion prediction network.

In some embodiments, the second training module is specifically configured to: according to the sample prediction camera movement, an adjacent sample image frame of the first sample image frame in the sample image frame sequence is re-projected onto the first sample image frame determine an error term; determine a penalty function term according to the sample prediction depth map and distribution continuity; and construct the loss function according to the reprojection error term and the penalty function term.

In some embodiments, a function possessed by an apparatus provided in an embodiment of the present invention or a module included may be used to execute the method described in the above-described method embodiment, and the implementation is referred to the description of the above-described method embodiment. and, for the sake of brevity, are not further repeated here.

Embodiments of the present invention also provide a computer readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the method when executed by a processor. A computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

An embodiment of the present invention also provides an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; Here, the processor is configured to execute the arbitrary scenario depth prediction method or any camera motion prediction method by calling the instruction stored in the memory.

An embodiment of the present invention also provides a computer program product comprising computer readable code, wherein when the computer readable code is run on the device, the processor in the device can determine the scenario depth and/or the depth provided by any of the preceding embodiments. Alternatively, a command for implementing a camera motion prediction method is executed.

Embodiments of the present invention also provide another computer program product for storing computer readable instructions, such that when the instructions are executed, the computer executes the scenario depth and/or camera motion prediction method provided in any one of the above embodiments. do.

The electronic device may be provided as a terminal, server, or other type of device.

7 is a block diagram of an electronic device 800 according to an embodiment of the present invention. As shown in FIG. 7 , the electronic device 800 may be a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, or a terminal such as a personal portable terminal.

Referring to FIG. 7 , the electronic device 800 includes a first processing component 802 , a first memory 804 , a first power component 806 , a multimedia component 808 , an audio component 810 , and a first input one or more of an Input Output (I/O) interface 812 , a sensor component 814 , and a communication component 816 .

The first processing component 802 generally controls the overall operation of the electronic device 800 , such as operations related to displays, phone calls, data communications, camera operations, and recording operations. The first processing component 802 may include one or a plurality of processors 820 for executing instructions for implementing all or part of the method steps. Further, the first processing component 802 may include one or a plurality of modules for facilitating interaction between the first processing component 802 and other components. For example, the first processing component 802 can include a multimedia component 808 and a multimedia module for facilitating an interaction between the first processing component 802 .

The first memory 804 is configured to store various types of data to support the operation of the electronic device 800 . Examples of such data include instructions of any application program or method running on the electronic device 800 , contact data, phonebook data, messages, photos, videos, and the like. The first memory 804 includes: Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), and Electrically Erasable Programmable Read Only Memory (EEPROM); Any type of volatile, such as Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic Disk or Optical Disk Alternatively, it may be implemented by a non-volatile storage device or a combination thereof.

The first power component 806 supplies power to various components of the electronic device 800 . The first power component 806 may include a power management system, one or more power sources, and other components related to generating, managing, and distributing power for the electronic device 800 .

The multimedia component 808 includes a screen that provides one output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (Touch Pad, TP). When the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or a plurality of touch sensors for sensing touch, swipe and gestures on the touch panel. The touch sensor may not only detect a boundary of a touch or slide operation, but also detect a duration and pressure associated with the touch or slide operation. In some implementations, the multimedia component 808 includes at least one of one front camera and one rear camera. When the electronic device 800 is in an operation mode such as a photographing mode or a video mode, at least one of the front camera and the rear camera may receive external multimedia data. Each front and rear camera may be a fixed optical lens system or may have a focal length and optical zoom function.

The audio component 810 is configured to output and/or input an audio signal. For example, the audio component 810 includes a single microphone (MICrophone, MIC), and when the electronic device 800 is in an operation mode such as a call mode, a recording mode, and a voice recognition mode, the microphone is an external audio signal is configured to receive The received audio signal may be further stored in the first memory 804 or transmitted by the communication component 816 . In some embodiments, the audio component 810 further includes one speaker for outputting an audio signal.

The first input/output interface 812 provides an interface between the first processing member 802 and an external interface module, which may be a keyboard, a click wheel, a button, or the like. Such buttons may include, but are not limited to, a home button, a volume button, a start button, and a lock button.

The sensor component 814 includes one or a plurality of sensors to provide status assessment of various aspects for the electronic device 800 . For example, the sensor component 814 may detect an on/off state of the electronic device 800 , a relative positioning of the component, for example, the component is a monitor and a keypad of the electronic device 800 , and a sensor The component 814 is the electronic device 800 or a change in the position of one component in the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the direction or acceleration/deceleration of the electronic device 800, and the electronic device ( 800) can be detected. The sensor component 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor component 814 may further include a photosensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor, for use in an imaging application. In some embodiments, the sensor component 814 may further include an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate communication between the electronic device 800 and another device in a wired or wireless manner. The electronic device 800 may access the wireless Internet based on a communication standard such as WiFi, 2G or 3G, or a combination thereof. In one exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication member 816 further includes a Near Field Communication (NFC) module for promoting short-range communication. For example, the NFC module is a radio frequency identifier (Radio Frequency Identification, RFID) technology, infrared communication standard (Infrared Data Association, IrDA) technology, Ultra WideBand (UWB) technology, Bluetooth (BlueTooth, BT) technology and other It can be implemented based on technology.

In an exemplary embodiment, the electronic device 800 includes one or a plurality of ApplicationSpecific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processors (DSPs), to execute the method. Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic devices for predicting depth of any of the above scenarios. Alternatively, the camera motion prediction method may be implemented.

In an exemplary embodiment, there is also provided a non-volatile computer readable storage medium, such as a first memory 804 , comprising computer program instructions, the computer program instructions being the processor of the electronic device 800 to complete the method. 820 may be executed.

8 is a block diagram of an electronic device according to an embodiment of the present invention. As shown in FIG. 8 , the electronic device 900 may be provided as one server. Referring to FIG. 8 , the electronic device 900 includes a second processing component 922 , and also includes memory resources represented by one or more processors and a second memory 932 , and includes a second processing component ( 922), for example to store instructions that can be executed by an application program. The application program stored in the second memory 932 may include one or more modules corresponding to each instruction set. Further, the second processing component 922 is configured to execute the instructions to execute any of the scenario depth prediction methods or any of the camera motion prediction methods.

The electronic device 900 also includes one second power component 926 configured to perform power management of the electronic device 900 , and one wired or wireless network interface 950 configured to connect the electronic device 900 to a network. , and one second input/output (I/O) interface 958 . The electronic device 900 may operate an operating system based on what is stored in the second memory 932 , such as, for example, Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

In an exemplary embodiment, there is also provided a non-volatile computer-readable storage medium, such as a second memory 932 , comprising computer program instructions, the computer program instructions being used in the electronic device 900 to complete the method. 2 may be executed by processing component 922 .

The invention may be at least one of a system, a method and a computer program product. The computer program product may include a computer readable storage medium comprising computer readable program instructions for causing a processor to implement each aspect of the present invention.

A computer-readable storage medium may be a tangible device capable of holding and storing instructions executed by an instruction execution device. The computer-readable storage medium may be, for example, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any of the aforementioned electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, and a semiconductor storage device. may be an appropriate combination of , but is not limited thereto. More specific examples of computer readable storage media (non-limiting list) include portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), and erasable and programmable Erasable Programmable Read Only Memory (EPROM or Flash Memory), Static Random Access Memory (SRAM), Portable Compact Disk Read-Only Memory (CD-ROM), A mechanical coding device such as a Digital Versatile Disk (DVD), a memory stick, a floppy disk, a device storing instructions, a hole card or a convex structure in a groove, and any suitable combination of the foregoing. shall not be interpreted as instantaneous signals themselves, such as radio or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through fiber optic cables), or electrical signals transmitted over wires .

The computer readable program instructions described herein may be downloaded from a computer readable storage medium to each computing/processing device, or stored in an external computer or external storage via a network such as at least one of the Internet, a local area network, a wide area network, and a wireless network. It can be downloaded to your device. Networks may include copper transport cables, fiber optic transport, wireless transport, routers, firewalls, switches, gateway computers, and edge servers. A network adapter card or network interface of each computing/processing device receives computer readable program instructions from the network and transmits the computer readable program instructions for storage in a computer readable storage medium in each computing/processing device.

Computer program instructions for carrying out the operations of the present invention may include assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or one or more programming instructions. It may be source code or object code written in any combination of languages, including object-oriented programming languages such as Smalltalk, C++, and the like, and conventional programming languages such as "C" languages or similar programming languages. The computer readable program instructions may be fully or partially executed on a user's computer, executed as a standalone software package, and partially executed on the user's computer, partially on a remote computer, or completely on a remote computer or server. In situations involving remote computers, the remote computer is connected to your computer through any type of network, including a local area network (LAN) or wide area network (WAN), or to an external computer (e.g. For example, you can connect via the Internet using your Internet Service Provider). In some embodiments, an electronic circuit such as a programmable logic circuit, a Field Programmable Gate Array (FPGA) or a Programmable Logic Array (PLA) uses state information from computer readable program instructions. to be personalized, the electronic circuit being capable of executing computer readable program instructions, thereby implementing various aspects of the present invention.

The present invention is described with reference to at least one of a flowchart and a block diagram according to a method, an apparatus (system), and a computer program product of an embodiment of the present invention. It should be understood that each block in the flowchart and/or block diagram and each combination of blocks in the flowchart and/or block diagram may all be implemented by computer readable program instructions.

These computer readable program instructions are provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing device, thereby creating a machine, which, when executed by the processor of the computer or other programmable data processing device, provides flow diagrams and blocks. An apparatus for implementing the functions/operations specified in one or a plurality of blocks in the figure is created. These computer readable program instructions may be stored in a computer readable storage medium, which may cause a computer, a programmable data processing apparatus, and other devices to operate in a specific way, such that the computer readable medium having the instructions stored thereon may include: An article of manufacture is included, wherein the article of manufacture includes instructions for implementing functions/operations specified in one or a plurality of blocks in at least one of a flowchart and a block diagram.

The computer readable program instructions may also be loaded into a computer, other programmable data processing device, or other device such that a series of operational steps is performed on the computer, other programmable data processing device, or other device to create a computer-implemented process. By causing the instructions to be executed in a computer, other programmable data processing device, or other device, the instructions to be executed in the computer, other programmable data processing device, or other device implement the functions/operations specified in one or a plurality of blocks in at least one of the flowchart and block diagram.

The flow diagrams and block diagrams in the drawings illustrate implementable system architectures, functions, and operations of systems, methods, and computer program products according to a plurality of embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or portion of an instruction, wherein the module, program segment or portion of the instruction contains executable instructions for implementing one or a plurality of specified logical functions. include In some alternative implementations, the functions indicated in the blocks may occur in a different order than indicated in the figures. For example, two consecutive blocks may actually be executed in parallel, sometimes in reverse order depending on the function involved, which is determined by the function involved. It should also be noted that each block in at least one of the block diagrams and flowcharts, and combinations of blocks in at least one of the block diagrams and flowcharts, may be implemented by a dedicated hardware-based system of a designated function or operation, or It may be implemented by a combination of dedicated hardware and computer instructions.

The computer program product may be specifically implemented through hardware, software, or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium, and in another optional embodiment, the computer program product is specifically implemented as a software product such as a Software Development Kit (SDK) or the like. is implemented

Each embodiment of the present invention has been described above, and the description is illustrative, non-exhaustive, and is not limited to each disclosed embodiment. Many modifications and changes will be apparent to those skilled in the art without departing from the scope and spirit of each described embodiment. The choice of terminology used herein is intended to best interpret the principle of each embodiment, practical application, or improvement over market technology, or to enable those skilled in the art to understand each embodiment disclosed herein.

industrial practicality

An embodiment of the present invention provides a method and apparatus, an electronic device, a medium and a program for predicting scenario depth and camera motion, the method comprising: acquiring a target image frame at time t; determining, by the scenario depth prediction network, a prediction depth map corresponding to the target image frame by performing scenario depth prediction on the target image frame using the first hidden state information at time t-1 - the first 1 The hidden state information includes feature information related to the scenario depth, and the scenario depth prediction network is obtained by auxiliary training based on the camera motion prediction network. An embodiment of the present invention can obtain a prediction depth map with high prediction accuracy corresponding to a target image frame.

Claims (23)

A scenario depth prediction method comprising:
obtaining a target image frame at time t; and
determining, by the scenario depth prediction network, a prediction depth map corresponding to the target image frame by performing scenario depth prediction on the target image frame using the first hidden state information at time t-1 - the first 1 The hidden state information includes feature information related to the scenario depth, and the scenario depth prediction network is obtained by auxiliary training based on the camera motion prediction network. According to claim 1,
determining, by the scenario depth prediction network, a prediction depth map corresponding to the target image frame, by performing scenario depth prediction on the target image frame using the first hidden state information at time t-1,
performing feature extraction on the target image frame to determine a first feature map corresponding to the target image frame, wherein the first feature map is a feature map related to a scenario depth;
determining the first hidden state information at time t according to the first feature map and the first hidden state information at time t-1; and
and determining the prediction depth map according to the first hidden state information at time t. 3. The method of claim 2,
the first hidden state information at the time t-1 includes the first hidden state information under a different scale at the time t-1;
Determining a first feature map corresponding to the target image frame by performing feature extraction on the target image frame,
performing a plurality of scale down-sampling on the target image frame to determine the first feature map under a different scale corresponding to the target image frame;
The step of determining the first hidden state information at time t according to the first feature map and the first hidden state information at time t-1 includes:
for any scale, determining the first hidden state information under the scale at time t according to the first feature map under the scale and the first hidden state information under the scale at time t-1 do;
Determining the predicted depth map according to the first hidden state information at the time t includes:
and determining the predicted depth map by performing feature fusion on the first hidden state information under a different scale at time t. According to any one of claims 1 to 3,
The method also comprises:
obtaining a sequence of sample image frames corresponding to time t, wherein the sequence of sample image frames includes a first sample image frame at time t and an adjacent sample image frame of the first sample image frame;
determining, by the camera motion prediction network, a sample prediction camera motion corresponding to the sample image frame sequence by performing camera pose prediction on the sample image frame sequence using the second hidden state information at time t-1 - the second hidden state information includes characteristic information related to camera movement;
A sample prediction depth map corresponding to the first sample image frame through which the scenario depth prediction network to be trained performs scenario depth prediction on the first sample image frame using the first hidden state information at time t-1 determining, wherein the first hidden state information includes feature information related to a scenario depth;
constructing a loss function according to the sample prediction depth map and the sample prediction camera motion; and
and performing training on the to-be-trained scenario depth prediction network according to the loss function to obtain the scenario depth prediction network. 5. The method of claim 4,
According to the sample prediction depth map and the sample prediction camera movement, building a loss function comprises:
determining, according to the sample prediction camera movement, a re-projection error term of an adjacent sample image frame of the first sample image frame in the sample image frame sequence with respect to the first sample image frame;
determining a penalty function term according to the distribution continuity of the sample prediction depth map; and
and constructing the loss function according to the reprojection error term and the penalty function term. A method for predicting camera motion, comprising:
obtaining a sequence of image frames corresponding to time t, wherein the sequence of image frames includes a target image frame at time t and an adjacent image frame of the target image frame; and
determining, by a camera motion prediction network, a predicted camera motion corresponding to the image frame sequence, by performing camera pose prediction on the image frame sequence using the second hidden state information at time t-1; 2 The hidden state information includes feature information related to camera motion, and the camera motion prediction network is obtained by auxiliary training based on a scenario depth prediction network. 7. The method of claim 6,
The step of determining, by the camera motion prediction network, the predicted camera motion corresponding to the image frame sequence, by performing camera pose prediction on the image frame sequence using the second hidden state information at time t-1,
performing feature extraction on the image frame sequence to determine a second feature map corresponding to the image frame sequence, wherein the second feature map is a feature map related to camera motion;
determining the second hidden state information at time t according to the second feature map and the second hidden state information at time t-1; and
and determining the predicted camera motion according to the second hidden state information at time t. 8. The method of claim 6 or 7,
wherein the predicted camera motion includes a relative pose between adjacent image frames in the sequence of image frames. 9. The method according to any one of claims 6 to 8,
The method also comprises:
obtaining a sequence of sample image frames corresponding to time t, wherein the sequence of sample image frames includes a first sample image frame at time t and an adjacent sample image frame of the first sample image frame;
The scenario depth prediction network determines a sample prediction depth map corresponding to the first sample image frame by performing scenario depth prediction on the first sample image frame using the first hidden state information at time t-1. doing, wherein the first hidden state information includes feature information related to the scenario depth;
A camera motion prediction network to be trained determines a sample prediction camera motion corresponding to the sample image frame sequence by performing camera pose prediction on the sample image frame sequence using the second hidden state information at time t-1. doing, wherein the second hidden state information includes characteristic information related to camera movement;
constructing a loss function according to the sample prediction depth map and the sample prediction camera motion; and
and performing training on the to-be-trained camera motion prediction network according to the loss function to obtain the camera motion prediction network. 10. The method of claim 9,
According to the sample prediction depth map and the sample prediction camera movement, building a loss function comprises:
determining, according to the sample prediction camera movement, a re-projection error term of an adjacent sample image frame of the first sample image frame in the sample image frame sequence with respect to the first sample image frame;
determining a penalty function term according to the distribution continuity of the sample prediction depth map; and
and constructing the loss function according to the reprojection error term and the penalty function term. A scenario depth prediction device, comprising:
a first acquiring module, configured to acquire a target image frame at time t; and
A first scenario, wherein the scenario depth prediction network is configured to determine a predicted depth map corresponding to the target image frame by performing scenario depth prediction on the target image frame using the first hidden state information at time t-1 Scenario comprising a depth prediction module, wherein the first hidden state information includes feature information related to the scenario depth, and the scenario depth prediction network is obtained by auxiliary training based on a camera motion prediction network. Depth prediction device. 12. The method of claim 11,
The first scenario depth prediction module includes:
a first determining submodule, configured to perform feature extraction on the target image frame to determine a first feature map corresponding to the target image frame, wherein the first feature map is a feature map related to the scenario depth;
a second determining submodule, configured to determine, according to the first feature map and the first hidden state information at time t-1, the first hidden state information at time t; and
and a third determining submodule, configured to determine a prediction depth map according to the first hidden state information at time t. 13. The method of claim 12,
the first hidden state information at time t-1 includes first hidden state information under a different scale at time t-1;
the first determining submodule is specifically configured to perform a plurality of scale downsampling on the target image frame to determine a first feature map under a different scale corresponding to the target image frame;
The second determining submodule is specifically, for any scale, according to the first feature map under the scale and the first hidden state information under the scale at time t-1, the first hidden state information under the scale at time t is configured to determine;
The third determining submodule is specifically configured to perform feature fusion on the first hidden state information under a different scale at time t to determine a prediction depth map. 14. The method according to any one of claims 11 to 13,
The device also includes a first training module, the first training module comprising:
obtain a sequence of sample image frames corresponding to time t, wherein the sequence of sample image frames includes a first sample image frame at time t and an adjacent sample image frame of the first sample image frame;
A camera motion prediction network determines a sample prediction camera motion corresponding to the sample image frame sequence by performing camera pose prediction on the sample image frame sequence using the second hidden state information at time t-1, the second hidden state information includes characteristic information related to camera movement;
A sample prediction depth map corresponding to the first sample image frame through which the scenario depth prediction network to be trained performs scenario depth prediction on the first sample image frame using the first hidden state information at time t-1 determine, wherein the first hidden state information includes feature information related to the scenario depth;
construct a loss function according to the sample prediction depth map and the sample prediction camera motion; and
and perform training on the to-be-trained scenario depth prediction network according to the loss function to obtain the scenario depth prediction network. 15. The method of claim 14,
The first training module, specifically,
according to the sample prediction camera movement, an adjacent sample image frame of the first sample image frame in the sample image frame sequence determines a reprojection error term for the first sample image frame; determine a penalty function term according to the distribution continuity of the sample prediction depth map; and constructing the loss function according to the reprojection error term and the penalty function term. A camera motion prediction device comprising:
a second acquiring module, configured to acquire a sequence of image frames corresponding to time t, wherein the sequence of image frames includes a target image frame at time t and an adjacent image frame of the target image frame; and
a first camera, configured to determine a predicted camera motion corresponding to the image frame sequence, through the camera motion prediction network performing camera pose prediction on the image frame sequence using the second hidden state information at time t-1 A camera comprising a motion prediction module, wherein the second hidden state information includes feature information related to camera motion, and the camera motion prediction network is obtained by auxiliary training based on a scenario depth prediction network. motion prediction device. 17. The method of claim 16,
The first camera motion prediction module,
a sixth determining submodule, configured to perform feature extraction on the image frame sequence to determine a second feature map corresponding to the image frame sequence, wherein the second feature map is a feature map related to camera motion;
a seventh determining submodule, configured to determine, according to the second image feature and the second hidden state information at time t-1, the second hidden state information at time t; and
and an eighth determining submodule, configured to determine a predicted camera motion according to the second hidden state information at time t. 18. The method of claim 16 or 17,
wherein the predicted camera motion includes a relative pose between adjacent image frames in an image frame sequence. 19. The method according to any one of claims 16 to 18,
The device also includes a second training module,
The second training module,
obtain a sequence of sample image frames corresponding to time t, wherein the sequence of sample image frames includes a first sample image frame at time t and an adjacent sample image frame of the first sample image frame;
The scenario depth prediction network determines a sample prediction depth map corresponding to the first sample image frame by performing scenario depth prediction on the first sample image frame using the first hidden state information at time t-1. and, the first hidden state information includes feature information related to the scenario depth;
A camera motion prediction network to be trained determines a sample prediction camera motion corresponding to the sample image frame sequence by performing camera pose prediction on the sample image frame sequence using the second hidden state information at time t-1. and, the second hidden state information includes characteristic information related to camera movement;
construct a loss function according to the sample prediction depth map and the sample prediction camera motion; and
and perform training on the to-be-trained camera motion prediction network according to the loss function to obtain the camera motion prediction network. 20. The method of claim 19,
The second training module,
according to the sample prediction camera movement, an adjacent sample image frame of the first sample image frame in the sample image frame sequence determines a reprojection error term for the first sample image frame; determine a penalty function term according to the distribution continuity of the sample prediction depth map; and construct the loss function according to the reprojection error term and the penalty function term. As an electronic device,
processor; and
a memory configured to store processor-executable instructions;
The electronic device, characterized in that the processor is configured to execute the method according to any one of claims 1 to 10 by calling an instruction stored in the memory. A computer-readable storage medium comprising:
A computer readable storage medium having stored thereon computer program instructions, which when executed by a processor implements the method according to any one of claims 1 to 10. A computer program comprising:
A computer comprising computer readable code, wherein when the computer readable code is run in an electronic device, a processor in the electronic device implements the method according to any one of claims 1 to 10. program.

Images