JP2022519194A - Depth estimation - Google Patents

Abstract

An image processing system that estimates the depth of the scene is provided. The image processing system comprises a fusion engine, which receives a first depth estimate from the geometric reconstruction engine and a second depth estimate from the neural network architecture. The fusion engine is configured to stochastically fuse the first depth estimation and the second depth estimation to output the fusion depth estimation of the scene. The fusion engine receives the uncertainty measurement of the first depth estimation from the geometric reconstruction engine and the uncertainty measurement of the second depth estimation from the neural network architecture, and makes the uncertainty measurement. It is configured to be used to stochastically fuse a first depth estimate with a second depth estimate.

Description

The present invention relates to estimating the depth of a scene. The present invention relates, but not exclusively, to the depth estimation used by a robotic device to navigate and / or interact within its environment.

In the fields of computer vision and robotics, it is often necessary to construct representations of three-dimensional (3D) space. By constructing a representation of 3D space, it is possible to map a real world environment to a virtual or digital domain, which can be used and manipulated by electronic devices. For example, in an augmented reality application, a user can use a handheld device to interact with a virtual object that corresponds to an entity in the surrounding environment, or a mobile robot device can locate and map simultaneously, and thus that environment. A representation of the 3D space may be needed to enable navigation. In many applications, intelligent systems may need to have a representation of the environment so that digital information sources can be tied to physical objects. This enables an advanced human-machine interface in which the physical environment surrounding people is the interface. Similarly, such representations can also enable advanced machine-world interfaces, for example, allowing robotic devices to interact with and manipulate physical objects in a real-world environment.

There are several techniques available for constructing representations of 3D space. For example, structural restoration from motion, as well as location and mapping concurrency (SLAM) are two such techniques. SLAM techniques usually involve estimating the depth of the 3D scene to be mapped. Depth estimation can be done using a depth camera. However, depth cameras are usually limited in range, consume relatively high power, and may not function properly in outdoor environments such as bright sunlight. In other cases, depth estimation can be done, for example, based on spatial images, without the use of depth cameras.

K.K. described in the minutes of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) in 2017. Tateno et al. The paper "CNN-SLAM: Real-time sense monocular SLAM with learned depth prediction" describes the fusion of depth maps obtained by a convolutional neural network (CNN) with depth measurements taken directly from a monocular SLAM. To restore blurry depth boundaries, a CNN predicted depth map is used as an initial estimate of the reconstruction and is continuously refined by a direct SLAM scheme that relies on small pixel-by-pixel baseline stereo matching. However, this approach does not maintain overall consistency.

Given existing techniques, effective and efficient methods of depth estimation are desired, for example to improve mapping in 3D space.

An image processing system for estimating the depth of a scene according to the first aspect of the present invention is provided. The image processing system comprises a fusion engine, which receives a first depth estimate from the geometric reconstruction engine and a second depth estimate from the neural network architecture, with the first depth estimate. Probabilistically fusing with the second depth estimation to output the fusion depth estimation of the scene, the fusion engine has the uncertainty measurement of the first depth estimation from the geometric reconstruction engine and the neural network architecture. The uncertainty measurement of the second depth estimation from is configured to receive, and the fusion engine uses the uncertainty measurement to probabilistically perform the first depth estimation and the second depth estimation. It is configured to be fused with.

In some embodiments, the fusion engine receives surface orientation estimation and surface orientation estimation uncertainty measurements from the neural network architecture, and uses surface orientation estimation and surface orientation estimation uncertainty measurements to first. It is configured to stochastically fuse the depth estimation and the second estimation of.

In some embodiments, the surface orientation estimation includes one or more of a depth gradient estimation in a first direction, a depth gradient estimation in a direction orthogonal to the first direction, and a surface normal estimation.

In some embodiments, the fusion engine is configured to specify the scale estimation when the first depth estimation and the second estimation are stochastically fused.

In some embodiments, the scene is captured in the first frame of the video data, a second depth estimate for the first frame of the video data is received, and the first depth estimate is for the video data. A plurality of first depth estimates for the first frame are included, and at least one of the plurality of first depth estimates includes a second frame of video data that is different from the first frame of the video data. Generated using, the fusion engine is configured to process a second depth estimate and one of a plurality of depth estimates for each iteration to iteratively output the fusion depth estimate for the scene.

In some embodiments, the first depth estimation, the second depth estimation, and the fusion depth estimation each include a depth map for a plurality of pixels.

In some embodiments, the first depth estimation is a medium density depth estimation, and the second depth estimation and the fusion depth estimation each include a high density depth estimation.

In some embodiments, the system comprises a monocular camera that captures frames of video data, a tracking system that identifies the orientation of the monocular camera while observing the scene, and a geometric reconstruction engine. .. In such an embodiment, the geometric reconstruction engine uses the orientation from the tracking system and the frame of the video data to generate a depth estimate for at least a subset of the pixels from the frame of the video data. The geometric reconstruction engine is configured to minimize metering errors and generate depth estimates.

In some embodiments, the system comprises a neural network architecture, the neural network architecture comprising one or more neural networks, configured to receive and make predictions of the pixel values of a frame of video data. In the prediction, to generate a second depth estimate, a depth estimate for each of the first sets of image parts, at least one surface orientation estimate for each of the second sets of image parts, and each depth. One or more uncertainty measurements associated with the estimation and one or more uncertainty measurements associated with each surface orientation estimation are predicted.

A method of estimating the depth of a scene according to a second aspect of the present invention is provided. The method is to use the geometric reconstruction of the scene to generate a first depth estimate of the scene, which outputs an uncertainty measurement of the first depth estimation. The generation is configured to generate a second depth estimate of the scene using the neural network architecture, which is an uncertainty measurement of the second depth estimate. The fusion depth estimation of the scene is made by probabilistically fusing the first depth estimation and the second depth estimation using the generation and the uncertainty measurement, which is configured to output. Including to generate.

In some embodiments, the method comprises acquiring image data representing two or more views of the scene from a camera prior to generating a first depth estimate. In such an embodiment, generating the first depth estimation is by obtaining the attitude estimation of the camera and at least minimizing the photometric error, which is a function of the attitude estimation and the image data. Includes generating depth estimates for.

In some embodiments, the method comprises acquiring image data representing one or more views of the scene from the camera prior to generating the first depth estimation. In such an embodiment, generating a second depth estimate is a neural network architecture that receives image data and uses a neural network architecture to generate a second depth estimate. Predicting depth estimates for each set of image parts, predicting at least one surface orientation estimate for each set of image parts using a neural network architecture, and using a neural network architecture, Includes predicting a set of uncertainty measurements for each depth estimation and each surface orientation estimation. The surface orientation estimation may include one or more of a depth gradient estimation in a first direction, a depth gradient estimation in a direction orthogonal to the first direction, and a surface normal estimation.

In some embodiments, the method comprises acquiring image data representing two or more views of the scene from a camera prior to generating a first depth estimate, the image data comprising a plurality of pixels. In such an embodiment, generating a first depth estimate is to obtain a camera attitude estimate and generate a medium density depth estimate that includes a depth estimate for some of the pixels in the image data. And, including. In these embodiments, generating a second depth estimate involves generating a high density depth estimate for the pixels in the image data, probabilistically performing a first depth estimate and a second depth estimate. Fusing involves outputting a high density depth estimate for the pixels in the image data.

In some embodiments, the method is iteratively repeated, and for subsequent iterations, the method comprises determining whether to generate a second depth estimate, a first depth estimate and a second. Probabilistic fusion with the depth estimation of

In some embodiments, the method is applied to a frame of video data, and probabilistic fusion of a first depth estimate and a second depth estimate is the first for a given frame of video data. It involves optimizing a cost function that includes a first cost term associated with the depth estimation of and a second cost term associated with the second depth estimation. In such an embodiment, the first cost term includes a function of the fusion depth estimate, the first depth estimate, and the uncertainty value of the first depth estimate, and the second cost term is. , A function of a fusion depth estimate, a second depth estimate, and a second depth estimation uncertainty value is included, and the cost function is optimized to identify the fusion depth estimate. Optimizing the cost function can include identifying the scale factor for the fusion depth estimation, which indicates the scale of the fusion depth estimation for the scene. In some embodiments, the method involves using a neural network architecture to generate at least one surface orientation estimate for the scene, where the neural network architecture is uncertain about each of the at least one surface orientation estimates. Configured to output measurements, the cost function includes a third cost term associated with at least one surface orientation estimate, the third cost term being a fusion depth estimate and a surface orientation estimate. Includes a function with an uncertainty value for each of the at least one surface orientation estimates.

In the particular set of examples according to the second aspect, the geometric reconstruction of the scene is configured to generate a first depth probability volume of the scene, where the first depth probability volume is the first depth estimation. Of the first plurality of depth estimates, including a first plurality of depth estimates including, and a first plurality of uncertainty measurements associated with each depth estimate of the first plurality of depth estimates. The uncertainty measurement associated with a given depth estimation of is the probability that a given area of the scene will be at the depth represented by the given depth estimation of the first plurality of depth estimations. The neural network architecture is configured to output a second depth probability volume of the scene, where the second depth probability volume is a second plurality of depth estimates, including a second depth estimate, and a second plurality. A second plurality of uncertainty measurements associated with each depth estimation of the depth estimation, and an uncertainty measurement associated with a given depth estimation of the second plurality of depth estimations. , Represents the probability that a given area of the scene is at the depth represented by the given depth estimation of the second plurality of depth estimations.

In some embodiments of a particular set of examples, generating a second depth estimate of the scene uses a neural network architecture to process the image data that represents the image of the scene, and the second. A second plurality of depth estimates, including generating a depth probability volume, comprises a plurality of depth estimation sets, each associated with a different portion of the image of the scene.

In some embodiments of a particular set of examples, the second plurality of depth estimates include depth estimates with predefined values. There can be non-uniform spacing between the predefined values. A predefined value may include multiple logarithmic depth values within a predefined depth range.

In some embodiments of the particular set of examples, generating the first depth probability volume of the scene is the first frame of video data representing the first observation of the scene and the second of the scene. It is to process with the second frame of the video data representing the observation of to generate a set of photometric errors for each of the plurality of parts of the first frame, each of which has a plurality of first photometric errors. Includes that generation associated with each depth estimation with different depth estimation and scaling the photometric error to convert the photometric error to their respective probability values.

In some embodiments of a particular set of examples, the probabilistic fusion of a first depth estimate and a second depth estimate using uncertainty measurements is a first plurality of failures. It involves combining a certainty measurement with a second plurality of uncertainty measurements to generate a fusion probability volume. In these embodiments, generating the fusion depth estimation of the scene may include obtaining the fusion depth estimation of the scene from the fusion probability volume. These examples may include using a fusion probability volume to obtain a depth probability function and using a depth probability function to obtain a fusion depth estimation. In these embodiments, obtaining a fusion depth estimate may include optimizing a cost function, which is the first cost term obtained using the fusion probability volume and the depth value. Includes a second cost term, including a local geometric constraint on. In such cases, the method is to receive surface orientation estimates and occlusion boundary estimates from further neural network architectures, and to use surface orientation estimates and occlusion boundary estimates to generate a second cost term. , Can be further included. In these embodiments, the fusion depth probability volume can be the first fusion depth probability volume associated with the first frame of video data representing the first observation of the scene, and the method is a first fusion depth. The second occupancy volume of the scene is warped based on the conversion of the stochastic volume to the first occupancy probability volume and the attitude data representing the posture of the camera while observing the scene. Obtaining the second occupancy probability volume associated with the second frame of the video data representing the observation of, and the second occupancy probability volume, the second fusion depth probability volume associated with the second frame. Can include converting to.

An image processing system for estimating the depth of a scene is provided according to a third aspect of the present invention, wherein the image processing system has a first depth probability volume from a geometric reconstruction engine and a second from a neural network architecture. Using a fusion engine and a fusion depth probability volume that receives the depth probability volume of and outputs the fusion depth probability volume of the scene by fusing the first depth probability volume and the second depth probability volume. It is equipped with a depth estimation engine that estimates the depth of the scene.

A method of estimating the depth of a scene is provided according to a fourth aspect of the invention, in which the geometric reconstruction of the scene is used to generate a first depth probability volume of the scene and a neural. Using a network architecture to generate a second depth-probability volume for a scene and a fusion of a first depth-probability volume and a second depth-probability volume to generate a fusion depth-probability volume for a scene. And to generate a fusion depth estimate for the scene using the fusion depth probability volume.

A computing system according to a fifth aspect of the present invention is provided, in which the computing system includes a monocular capture device that provides a frame of video, a position determination and mapping simultaneous execution system that provides orientation data of the monocular capture device, and a fifth. It comprises a system of the first or third aspect, a medium density multi-view stereo component that receives frames of orientation data and video and implements a geometric reconstruction engine, and an electronic circuit that implements a neural network architecture. ..

A robotic device according to a sixth aspect of the present invention is provided, wherein the robotic device is a computing system according to the fifth aspect and one or more actuators that allow the robotic device to interact with the surrounding three-dimensional environment. An interaction engine having at least a portion of the surrounding 3D environment shown in the scene, the one or more actuators and at least one processor controlling the one or more actuators, for fusion depth estimation. It comprises the interaction engine, which is used to interact with the surrounding 3D environment.

A non-temporary computer-readable storage medium comprising computer-executable instructions according to a seventh aspect of the present invention is provided, and when the computer-executable instructions are executed by the processor, the computing device is subjected to the method described above. Have one run.

Further functionality will be apparent from the following description of embodiments of the invention given as merely examples with reference to the accompanying drawings.

It is a schematic diagram which shows the Example of the 3D (3D) space. FIG. 6 is a schematic diagram showing the available degrees of freedom of an exemplary object in 3D space. FIG. 6 is a schematic diagram showing video data generated by an exemplary capture device. It is the schematic of the image processing system by an Example. It is a schematic diagram of an image processing system according to a further embodiment. It is a schematic diagram of an image processing system according to still another embodiment. It is a schematic diagram which shows the uncertainty measurement of the surface orientation estimation and the surface orientation estimation by an Example. It is a schematic diagram of an image processing system according to still another embodiment. It is a schematic diagram which shows the component of the computing system by an embodiment. It is a schematic diagram which shows the component of the robot device by an Example. It is a schematic diagram which shows the Example of the various functions described with reference to FIGS. 1-7. It is a flow diagram which shows the exemplary method of estimating the depth of a scene. It is a flow diagram which shows the further exemplary method of estimating the depth of a scene. FIG. 5 is a schematic diagram illustrating an embodiment of a processor and a non-temporary computer-readable storage medium including computer-executable instructions. It is a schematic diagram which shows the fusion of the 1st depth estimation and the 2nd depth estimation of a scene by a further embodiment. FIG. 3 is a schematic diagram of a system for acquiring a second depth probability volume according to an embodiment. FIG. 3 is a schematic diagram illustrating an example of uncertainty measurement associated with each depth estimate obtained using the system of FIG. FIG. 3 is a schematic diagram of a system for acquiring a first depth probability volume according to an embodiment. It is a flow diagram which shows the exemplary method of getting the fusion depth estimation of a scene. FIG. 6 is a schematic diagram of a system for obtaining fusion depth estimates by using the method of FIG. It is a flow diagram which shows the exemplary method of acquiring the 2nd fusion depth probability volume. It is a flow diagram which shows the exemplary method of estimating the depth of a scene by a further embodiment. FIG. 3 is a schematic diagram of an image processing system for estimating the depth of a scene according to a further embodiment.

Some embodiments described herein make it possible to estimate the depth of the scene. Such embodiments include the generation of a first depth estimate of the scene using the geometric reconstruction of the scene. The first depth estimation can be generated, for example, by processing an image of the scene. The image can be, for example, a two-dimensional (2D) color image, eg, an RGB (red, green, blue) image. The first depth estimation can be generated based on geometric constraints. For example, the color of a pixel in an image that represents a given part of the scene can be assumed to be independent of the position of the camera used to capture the image. This can be used to generate a first depth estimate, as further described with reference to the figure. The geometric reconstruction is also configured to output a first depth estimation uncertainty measurement that indicates, for example, the accuracy of the first depth estimation. For example, if the first depth estimation is more constrained and can be estimated accurately, the uncertainty measurement can be lower than in other cases where the first depth estimation is less constrained.

A second depth estimate of the scene is generated using the neural network architecture. The neural network architecture is also configured to output a second depth estimation uncertainty measurement. For example, a scene image can be processed using a neural network architecture such as a convolutional neural network (CNN) trained to predict both depth estimation and associated uncertainties from the input image. Uncertainty measurements may indicate the reliability of the associated second depth estimation. For example, a second depth estimate of an image region containing objects that were not present in the training data used to train the neural network architecture can be relatively uncertain, and thus relatively obtained from the neural network architecture. Can be associated with high uncertainty measurements. Conversely, a second depth estimate of the image area containing the objects present in the training data may be associated with a lower uncertainty measurement.

The first depth estimate and the second depth estimate are stochastically fused using uncertainty measurements to generate a fused depth estimate for the scene. By combining the first depth estimation and the second depth estimation in this way, the accuracy of the fusion depth estimation can be improved. For example, the first depth estimation (based on geometric constraints) may compare one part of the scene to another part of the scene to provide a reliable relative depth estimation for that part of the scene. In this way, the first depth estimation may be able to place or locate that part of the scene at a suitable position in the real world environment, as compared to, for example, other parts of the scene. However, the first depth estimation has the accuracy of capturing the depth gradient within that portion of the scene, for example due to the uneven texture of the surface within that portion of the scene, such as changes in depth within that portion of the scene. Can be low. In contrast, a second depth estimate (obtained from a neural network architecture) can accurately capture the depth gradient in the scene, but position a given part of the scene compared to the rest of the scene. The accuracy to identify can be low. However, by stochastically fusing the first depth estimation and the second depth estimation using uncertainty measurement, the individual effects of the first depth estimation and the second depth estimation are synergistic. Therefore, the accuracy of fusion depth estimation is improved. For example, to ensure the overall consistency of the fusion depth estimation, the uncertainty measurement may constrain the fusion of the first depth estimation and the second depth estimation. In addition, blurry artifacts in the estimated depth of the scene can be reduced compared to other methods.

1A and 1B schematically show an embodiment of a 3D space and a capture of image data associated with that space. Next, FIG. 1C shows a capture device configured to generate image data when displaying space. These examples are presented to better illustrate some of the features described herein and should not be considered limiting, but some for ease of explanation. Functions have been omitted and simplified.

FIG. 1A shows Example 100 of the 3D space 110. The 3D space 110 can be at least part of an internal physical space and / or an external physical space, such as a room or geographical location. The 3D space 110 of the present embodiment 100 includes several physical objects 115 arranged in the 3D space. These objects 115 may include, among other things, one or more of people, electronic devices, furniture, animals, building parts, and equipment. The 3D space 110 of FIG. 1A is shown with a bottom surface, but this does not have to be the case in all embodiments, for example the environment may be in the air or in extraterrestrial space.

Example 100 also includes various exemplary capture devices 120-A, 120-B, 120-C (collectively referred to as reference number 120) that can be used to capture video data associated with 3D space 110. Is shown. A capture device such as the capture device 120-A of FIG. 1A may include a camera configured to record the data generated by observing the 3D space 110 in digital or analog format. For example, the capture device 120-A can be a monocular capture device such as a monocular camera. Monocular cameras usually capture images of the scene from one position at a time and may have a single lens or lens system. In contrast, a stereo camera generally includes at least two lenses, each with a separate image sensor. The monocular capture device that can be used as the capture device 120-A can be a monocular multidirectional camera device arranged to capture an image of 3D space 110 from a plurality of angular positions. When in use, multiple images can be captured one after another. In some cases, multiple angular positions occupy a large field of view. In certain cases, the capture device 120-A may include an omnidirectional camera, eg, a device configured to capture a substantially 360 degree field of view. In this example, the omnidirectional camera may include a device with a panoramic annular lens, for example the lens may be attached in connection with a charge-coupled array.

The capture device 120-A may be mobile to capture multiple images in 3D space from multiple different locations. For example, the capture device 120-A may be configured to capture different frames corresponding to different observation portions of the 3D space 110. The capture device 120-A may be movable relative to a stationary platform and may include an actuator that changes the position and / or orientation of the camera with respect to, for example, the 3D space 110. In another example, the capture device 120-A can be a handheld device operated and moved by a human user. In one case, the capture device 120-A may include a still image device such as a camera configured to capture a series of images, in another case the capture device 120-A may videoframe the series of images. It may be equipped with a video device that captures video data, including in the format of. For example, the capture device 120-A may be a monocular camera or monocular capture device that captures or captures frames of video data.

FIG. 1A also shows a plurality of capture devices 120-B, 120-C connected to a robot device 130 configured to move within the 3D space 110. The robot device 135 may include an autonomous aerial movable device and / or an autonomous ground movable device. In the 100th embodiment, the robot device 130 comprises an actuator 135, which allows the device to navigate the 3D space 110. These actuators 135 include exemplary wheels, and in other cases, these actuators 135 may include railroad tracks, drilling mechanisms, rotors, and the like. On such a device, one or more capture devices 120-B, 120-C may be mounted statically or movably. In some cases, the robot device can be statically mounted within the 3D space 110, but some of the devices, such as arms or other actuators, move in space and interact with objects in space. Can be configured as Each capture device 120-B, 120-C may capture different types of image data, video data, and / or may include a stereo image source. In one example, at least one of the capture devices 120-B, 120-C is configured to capture photometric data, such as a color or grayscale image. In one example, one or more of the capture devices 120-B, 120-C may be mobile independently of the robot device 130. In one example, one or more of the capture devices 120-B, 120-C may be mounted on a rotating mechanism that rotates, for example, in an angular arc and / or rotates at 360 degrees, and / or a scene. Consists of optics adapted to capture a panorama (eg, a complete panorama of up to 360 degrees). It will be appreciated that in some cases a capture device similar to or identical to the capture device 120-A can be used as one or both of the capture devices 120-B, 120-C of FIG. 1A.

FIG. 1B shows Example 140 of the degrees of freedom available to the capture device 120 and / or the robot device 130. In the case of a capture device such as 120-A, the device orientation 150 may be in line with the axis of the lens or other imaging device. As an example of rotating around one of the three axes, the normal axis 155 is shown in the figure. Similarly, in the case of the robot device 130, the alignment direction 145 of the robot device 130 may be defined. It may indicate the orientation and / or direction of travel of the robot device. The bobbin axis 155 is also shown. Only a single normal axis is shown for the capture device 120 or robot device 130, but these devices are around one or more of the axes schematically shown as 140, as described below. Can be rotated.

More generally, the orientation and position of the capture device can be defined in three dimensions with respect to six degrees of freedom (6DOF), and the position is within each dimension of the three dimensions, eg [x, y, z]. It can be defined by coordinates and the orientation can be defined by an angle vector representing rotation around each of the three axes, eg [θ _x , θ _y , θ _z ]. Positions and orientations can be considered, for example, in three dimensions with respect to the origin defined in the 3D coordinate system. For example, the [x, y, z] coordinates may represent a transformation from the origin to a particular position in the 3D coordinate system, and the angle vector [θ _x , θ _y , θ _z ] may represent a rotation in the 3D coordinate system. Can be defined. A transformation with 6DOF can be defined as a matrix, so the transformation is applied by multiplication by the matrix. In some embodiments, the capture device can be defined with respect to a limited set of 6 degrees of freedom, for example in the case of a capture device on a ground vehicle, the y-dimension can be constant. In some embodiments, such as the robot device 130, the orientation and position of the capture device connected to another device can be defined relative to the orientation and position of that other device, eg, the orientation and position of the robot device 130. Can be defined relative to position.

In the embodiments described herein, the orientation and position of the capture device can be defined as the orientation of the capture device, as described, for example, in the 6DOF transformation matrix. Similarly, the orientation and position of an object representation can be defined as the orientation of the object representation, as described, for example, in the 6DOF transformation matrix. For example, when video data or a series of still images are recorded, the attitude of the capture device may change over time so that the capture device may take a different attitude at time t + 1 than time t. In the case of a handheld mobile computing device with a capture device, the attitude of the handheld device can change as it is moved within the 3D space 110 by the user.

FIG. 1C schematically shows an example of a capture device configuration. In Example 160 of FIG. 1C, the capture device 165 is configured to generate image data 170. In FIG. 1C, the image data 170 includes a plurality of frames 175. Each plume 175 may be associated with a particular time t (ie, F _t ) within the period in which an image in 3D space, such as 110 in FIG. 1, is captured. Frame 175 usually consists of a 2D representation of the measurement data. For example, frame 175 may include a 2D array or matrix of pixel values recorded at time t. In the embodiment of FIG. 1C, all frames 175 in the image data are the same size, but this does not have to be the case in all the embodiments. Pixel values in frame 175 represent measurements of specific parts of 3D space. In FIG. 1C, the image data represents a plurality of views of the scene from the monocular capture device, each of which is captured at different times t. However, in other cases, the image data captured by the capture device (ie, an image capture system or video capture system) represents multiple views of the scene captured at the same time, or at least partially overlapping, with each other. obtain. This can be the case where the capture device is a stereo capture system.

In the embodiment of FIG. 1C, each frame 175 contains photometric data. Photometric data usually represents the photometric characteristics of an image, such as luminance, intensity, and color. In FIG. 1C, each frame 175 contains an intensity value for each pixel of frame 175, which may be stored, for example, at a grayscale level or luminance level of 0-255 per color band or color channel. For example, grayscale level 0 corresponds to the darkest intensity (eg black), for example grayscale level 255 corresponds to the brightest intensity (eg white), and grayscale levels 0-255 are between black and white. Corresponds to strength. In FIG. 1C, the photometric data represents red, green, and blue pixel intensity values at a given resolution. Therefore, each frame 175 represents a color image, and each [x, y] pixel value in the frame contains an RGB vector [R, G, B]. As an embodiment, the resolution of the color data can be 640 x 480 pixels. In other embodiments, other color spaces may be used and / or the photometric data may represent other photometric properties.

The capture device 165 may be configured to store image data 170 in a connected data storage device. In another example, the capture device 165 may transmit the image data 170 to the connected computing device, for example as a data stream or frame by frame. Connected computing devices may be directly connected or indirectly connected, for example via a universal serial bus (USB) connection, eg image data 170 may be transmitted over one or more computer networks. .. In yet another example, the capture device 165 may be configured to transmit image data 170 over one or more computer networks and store it in a networked storage device. The image data 170 may be stored and / or transmitted frame by frame or, for example, on a batch basis where multiple frames can be combined.

The image data 170 may also perform one or more preprocessing operations before being used in the embodiments described below. In one case, pretreatment may be applied so that the two sets of frames have a common size and resolution.

In some cases, the capture device 165 may be configured to generate video data in image data format. However, the video data may similarly represent multiple frames captured at different times. In one case, the video data captured by the capture device 165 may include a compressed video stream or file. In this case, a frame of video data may be reconstructed from a stream or file, for example as the output of a video decoder. Following video stream or file preprocessing, video data may be retrieved from memory location.

It will be appreciated that FIG. 1C is provided as an embodiment and may use a configuration different from that shown in the figure to generate the image data 170 for use in the methods and systems described below. The image data 170 may further include any measurement sensory input configured in two-dimensional format representing a captured or recorded view in 3D space. For example, this may include photometric data, depth data electromagnetic imaging, ultrasonic imaging and radar output, among others. In these cases, only imaging devices associated with a particular data format, such as RGB devices without depth data, may be required.

FIG. 2 shows an exemplary image processing system 200 for estimating the depth of a scene. In the image processing system 200 of FIG. 2, the geometric reconstruction engine produces an uncertainty measurement 235 for the first depth estimation 230 and the first depth estimation 230. The first depth estimation 230 and the uncertainty measurement 235 of the first depth estimation 230 may be collectively referred to as the first depth data 250. The geometric reconstruction engine is configured to obtain a first depth estimate 230, for example by processing at least two images of the scene. As described with reference to FIGS. 1A-1C, at least two images can be captured using any suitable capture device and can be represented as image data such as RGB data. The geometric reconstruction engine may utilize a photometric technique to generate a first depth estimate 230. For example, regardless of the location of the capture device used to capture an image of a given part of the scene, the given part of the scene has the same photometric properties (such as brightness, intensity, and / or color). Should be. The geometric reconstruction engine can utilize this to generate a first depth estimate 230. As an embodiment, the geometric reconstruction engine processes at least two images of the same scene captured from different positions to determine the depth of a given portion of the scene that minimizes photometric error. obtain. For example, when the first depth estimation 230 is closest to the actual depth of a given portion of the scene, the photometric error can be minimized. However, this is merely an example, and in other embodiments, other geometric techniques may be used to generate the first depth estimate 230. In particular, the geometric reconstruction technique described herein with reference to, eg, FIG. 2, uses two images that can be obtained, eg, using a monocular system, but other embodiments, eg, single. In the case of stereo images, one or more images may be used.

In some cases, image data representing two or more views of the scene is obtained from a capture device such as a camera before generating the first depth estimation 230. In such cases, generating the first depth estimation 230 is first by acquiring the attitude estimation of the camera and at least minimizing the measurement error which is a function of the attitude estimation and the image data. To generate a depth estimate of 230, including.

Camera pose estimation usually indicates the position and orientation of the camera while capturing the image represented by the image data. If the image data represents, for example, a series of views corresponding to a video frame, the pose estimation may indicate the position and placement of the camera over time through the video frame. For example, image data can be acquired by moving a camera (RGB camera, etc.) to people in the environment (inside the room, etc.). Thus, at least a subset of video frames (hence the subset of images represented by image data) may have a corresponding pose estimate that represents the position and orientation of the camera at the time the frame was recorded. Attitude estimation is not present in every frame of the video (or every image in a series of images), but within the recorded time range of a subset of the video or images acquired by the camera. Can be specified with respect to a subset of the time of.

A variety of different methods can be used to obtain camera attitude estimates. For example, camera orientation can be estimated using a known SLAM system that receives image data and outputs attitude, using camera sensors that indicate position and orientation, and / or using custom attitude tracking methods. And can be estimated. In SLAM systems, for example, camera orientation can be estimated based on the processing of images captured by the camera over time.

The first depth estimation 230 can be obtained by at least minimizing the photometric error, which is a function of the pose estimation and the image data. In some cases, a mapping function is applied to map the pixels of the first image (corresponding to the first view of the scene) to the corresponding positions of the second image (corresponding to the second view of the scene). Then, a remapping version of the first image can be obtained. Such a mapping function depends, for example, on the estimated attitude of the camera while capturing the first image and the pixel depth of the first image. Then, for each pixel of the remapped version of the first image, the photometric properties can be identified (eg, using an intensity function that returns the intensity value of a given pixel). The same intensity function can then be used to identify the corresponding photometric characteristics for each pixel of the first image (acquired by the camera). The photometric characteristics associated with pixels of a given depth (such as pixel intensity values) should be independent of the camera's orientation, so the remapping version of the first image and the photometric characteristics of the first image itself , If the depth is estimated correctly, it should be the same. In this way, the pixel depth of the first image can be iteratively changed to the difference between the photometric error (eg, the photometric characteristics of the first image and the photometric characteristics of the remapped version of the first image). Based on) can be calculated for each iteration. The first depth estimation 230 for a given pixel can be thought of as a depth value that minimizes such photometric errors. In the embodiment, the depth estimation used iteratively during the photometric error minimization process can be along the epipolar lines of the image. If a depth estimate already exists for a given pixel (eg, taken from a previous frame or image that has a pixel corresponding to a given pixel), the depth estimate that is iteratively entered into the metering error calculation , Can be within a given range of the previous depth estimation, eg, within the range of two plus or minus the uncertainty measurements associated with the previous depth estimation. This can improve the efficiency of the generation of the first depth estimation 230 by concentrating on the search for suitable depth values within a more likely range of depth values. In some cases, interpolation may be performed between two adjacent depth values that are close to or contain the depth value associated with the minimum metering error. A preferred method for obtaining the first depth estimate 230 was published in the minutes of the 2013 International Conference on Computer Vision (ICCV). Explained in the paper "Semi-Dense Visual Odometry for a Monocular Camera" by Angel et al. However, other methods may be used instead.

Usually, there is uncertainty associated with the first depth estimation 230. Uncertainty represents, for example, the confidence that the first depth estimate 230 correctly corresponds to the actual depth. For example, uncertainty can depend on metering uncertainty (which can be limited or dependent on the metering resolution of the capture device), and metering uncertainty limits the accuracy with which the first depth estimate 230 can be identified. Can be. Uncertainty is further or instead between the method used to generate the first depth estimation 230 and the adjacent interpolation points if the generation of the first depth estimation 230 involves an interpolation process. It may depend on any inherent uncertainty associated with this method, such as step size. Uncertainty can be considered to correspond to the error associated with the first depth estimation 230. In the embodiment of FIG. 2, the geometric reconstruction engine is configured to output both the first depth estimation 230 and the uncertainty measurement 235 of the first depth estimation 230. The geometric reconstruction engine can be configured to generate an array or matrix containing the mean μ corresponding to the first depth estimate 230 and the variance θ corresponding to the uncertainty measurement 235, which is merely It's just an example. Mean and variance may be provided for the entire image, one or more image portions, one or more of the pixels of the image.

In an embodiment in which the generation of the first depth estimation 230 involves minimizing (or other optimization) the photometric error, the uncertainty measurement 235 associated with the first depth estimation 230 is the first depth estimation 230. Can be obtained by calculating the Jacobian term J based on the difference in photometric error between the two depth values used in the interpolation to obtain and based on the difference between these two depth values. In such a case, the uncertainty θ _geo of the first depth estimation 230 can be considered as follows.
θ _geo = (J ^T J) ^-1
However, this is just an example, and other uncertainty measurements may be used in other embodiments.

In some cases, the first depth estimation 230 may be a first depth map for a plurality of pixels. For example, the first depth estimation 230 may include a pixel-by-pixel depth estimation for each pixel of the input image of the scene. Therefore, the resolution of the input image and the first depth map corresponding to the first depth estimation 230 can be the same. It should be appreciated that preprocessing may be performed on the input image prior to generating the first depth estimation 230, which may include changing the resolution of the input image. For example, the resolution of the input image can be reduced, for example by downsampling the image to reduce the computational requirements for processing the input image. In another example, the first depth estimation 230 may include a single depth value for a plurality of pixels, and the depth value of the input image and the pixel have a one-to-many correspondence. For example, multiple pixels, eg images with similar photometric properties such as similar color or intensity, may be combined and the depth value may be obtained for this pixel combination.

In some cases, the first depth estimation 230 may be a so-called "medium density" depth estimation. In such cases, the first depth estimation 230 may include a depth estimation of a subset of parts of the scene, for example captured by an input image (or a plurality of images). For example, medium density depth estimation may include depth estimation of a portion of a pixel in image data representing two or more views of a scene, for example corresponding to a portion of two or more views of the scene. The portion of the scene from which the first depth estimation 230 has been obtained may correspond to a portion of pixels that meets some image criteria, such as some photometric criteria. For example, the first depth estimation 230 may be obtained for a portion of the image that has been identified as containing a sufficient amount of detail or information. This can be specified, for example, by calculating an image gradient that indicates a change in photometric characteristics (such as brightness or color) over a given area. The image gradient can correspond to a depth gradient that indicates a change in depth over a given area of the scene, or can be used as a substitute for a depth gradient. The image gradient is usually relatively large in an image region that has a large amount of detail, for example, in a relatively small region of the scene where the change in depth is relatively large and corresponds to a feature-rich portion of the scene. In other cases, the first depth estimation 230 may be a so-called "low density" depth estimation. In these cases, the first depth estimation 230 may be obtained for a portion of the image identified as corresponding to a particular image feature. For example, image keypoints can be identified, and image keypoints usually correspond to characteristic positions in the image that can be reliably repositioned from various viewpoints, rotations, scales, and illuminances. In such a case, the first depth estimation 230 can be obtained for the image patch containing the key points without obtaining the depth estimation of other image portions. In yet another example, the first depth estimation 230 may be a so-called "high density" depth estimation in which a depth estimation is obtained for the entire image (or image portion) regardless of the content of the image or image portion.

In some cases, the uncertainty measurement 235 of the first depth estimate 230 may be of the same type or contain the same resolution as the first depth estimate 230. For example, if the first depth estimation 230 includes a pixel-by-pixel depth estimation of the input image, there may also be a corresponding pixel-by-pixel uncertainty measurement. Conversely, if the first depth estimation 230 includes depth estimation for a plurality of pixels in the input image, there may also be a corresponding uncertainty measurement for the plurality of pixels. Similarly, if the first depth estimate 230 is low density, medium density, or high density, then the uncertainty measurement 235 can also be low density, medium density, or high density, respectively. However, in other cases, the type or resolution of the uncertainty measurement 235 may differ from the type or resolution of the first depth estimate 230.

The image processing system 200 of FIG. 2 is also configured to generate a second depth estimation 240 and an uncertainty measurement 245 of the second depth estimation 240, the second depth estimation 240, and the second depth. The uncertainty measurement 245 of the estimation 240 may be collectively referred to as the second depth data 260. The second depth data 260 can be generated using a neural network architecture, which is unsupervised image data or supervised (ie labeled) to predict depth estimation and associated uncertainty measurements. Can be trained with image data. A variety of different neural network architectures can be used. For example, a neural network architecture may include at least one convolutional neural network (CNN), which can be a so-called "deep" neural network with multiple layers.

In some embodiments, image data representing one or more views of the scene may be obtained from a capture device, such as a camera, prior to generating the first depth estimation 230. In such cases, generating a second depth estimate 240 may include receiving image data in a neural network architecture. The image data can be in any suitable format and may represent, for example, multiple 2D images of the scene captured from a plurality of different locations. Next, using a neural network architecture, a depth estimate can be predicted for each set of image portions and a second depth estimate 240 can be generated. The set of image portions may correspond to the entire image (or a plurality of images), or to a subset of an image or a plurality of images.

Similar to the first depth estimation 230, the second depth estimation 240 can be a second depth map for a plurality of pixels, for example, the depth values and pixels of the input image of the scene are one-to-one mappings. .. However, in other cases, the second depth estimation 240 may include a single depth value for a plurality of pixels, and the depth value of the input image and the pixel have a one-to-many correspondence. Further, the second depth estimation 240 may be a low density, medium density, or high density depth estimation. In one example, the two depth estimates have different densities, for example the first depth estimate 230 may be a medium density depth estimate and the second depth estimate 240 may be a high density depth estimate. Further, as described with reference to the first depth estimation 230, the type or resolution of the uncertainty measurement 245 of the second depth estimation 240 is the same as the type or resolution of the second depth estimation 240. It can or can be different.

The image processing system 200 of FIG. 2 includes a fusion engine 270, which includes a first depth estimation 230 from a geometric reconstruction engine and an uncertainty measurement 235 of the first depth estimation 230. It is configured to receive a second depth estimate 240 from the neural network architecture and an uncertainty measurement 245 of the second depth estimate 240. The fusion engine 270 uses the uncertainty measurement 235 of the first depth estimation 230 and the uncertainty measurement 245 of the second depth estimation 240 to use the first depth estimation 230 and the second depth estimation 240. And are stochastically fused to output the fusion depth estimation 280 of the scene. In this way, both the first depth estimation 230 and the second depth estimation 240 contributed to the fusion depth estimation 280, whereby the first depth estimation 230 or the second depth estimation 240 was used alone. Compared with the case, the accuracy of the fusion depth estimation 280 can be improved.

For example, the uncertainty of the first depth estimate 230 (based on geometric constraints) is in areas of low textured scenes, such as walls where the depth is relatively unchanged or the depth changes little by little. In the area, it can be higher. In addition, additional or alternative, the first depth estimation 230 can be relatively uncertain in areas where part of the scene is partially obstructed. In contrast, the second depth estimate 240 (obtained by the neural network architecture) may be less uncertain than the first depth estimate 230 in ambiguous regions (eg, low textured regions), but is higher. In the textured area, the second depth estimation 240 may be less accurate, even though the high texture area was accurately captured by the first depth estimation 230. The use of uncertainty measurement 235 and uncertainty measurement 245 favorably balances, for example, the first depth estimation 230 and the second depth estimation 240, respectively, based on their relative uncertainties. By contributing to the fusion depth estimation 280, the stochastic fusion of the first depth estimation 230 and the second depth estimation 240 is assisted. For example, in areas of the scene where the uncertainty measurement 235 associated with the first depth estimation 230 is higher than the uncertainty measurement 245 associated with the second depth estimation 240, the second depth estimation 240 may be: It can contribute to the fusion depth estimation 280 larger than the first depth estimation 230. In addition, because overall consistency can be maintained, the Fusion Depth Estimate 280 accurately captures the depth of the scene at the overall level, not just the selected local scene area.

In some cases, the first depth estimation 230 is a medium density depth estimation, and the second depth estimation 240 and the fusion depth estimation 280 each include a high density depth estimation. For example, the first depth estimation 230 may be obtained for a portion of the scene that has sufficient texture for which the first depth estimation 230 can be reasonably accurate. In such cases, the first depth estimation 230 cannot be obtained for other parts of the scene where the texture is lacking. However, the second depth estimation 240 may be obtained for the entire scene captured in the image (or image portion). Therefore, in such cases, by fusing the first depth estimation 230 and the second depth estimation 240, the fusion depth estimation 280 can also be obtained for the entire scene captured in the image. In such cases, a portion of the fusion depth estimation 280 can be obtained by fusing both the first depth estimation 230 and the second depth estimation 240 (eg, a portion of the fusion depth estimation 280 is of the scene. Corresponds to areas with many textures). However, different parts of the fusion depth estimation 280 can only be obtained from the second depth estimation 240 (eg, because a portion of the fusion depth estimation 280 corresponds to a smooth portion of the scene, the first depth estimation 230 is reliable. Can be low).

A variety of different methods can be used to stochastically fuse the first depth estimation 230 and the second depth estimation 240. For example, a cost function based on a first depth estimate 230 and a second depth estimate 240, as well as an uncertainty measurement 235 and an uncertainty measurement 245 probabilistically makes a first depth estimate 230 and a second depth estimate 240. Can be optimized to obtain a fusion depth estimate of 280. Cost function optimization may include iteratively calculating the value of the cost function with different input depth estimates, such as obtaining a fusion depth estimate 280 from which the minimum value of the cost function is obtained. The cost function can be referred to as a loss function or an error function instead.

In the embodiment of FIG. 2, the cost function includes a first cost term associated with the first depth estimation 230 and a second cost term associated with the second depth estimation 240. The first cost term is a fusion depth estimate, a first depth estimate (eg, obtained from a first depth estimate 230 obtained from a geometric reconstruction engine), and a first depth estimate 230. It includes a function with an uncertainty value (eg, obtained from the uncertainty measurement 235 of the first depth estimate 230). Similarly, the second cost term is a fusion depth estimate, a second depth estimate (eg, obtained from a second depth estimate 240 obtained from a neural network architecture), and a second depth estimate 240. It includes a function with an uncertainty value (eg, obtained from the uncertainty measurement 245 of the second depth estimate 240). The cost function is optimized to identify the fusion depth estimates that form the fusion depth estimation 280 output by the fusion engine 270. This involves, for example, iteratively modifying the fusion depth estimate to identify the fusion depth estimate that optimizes the cost function. For example, a cost function can be considered optimized if its value meets a predefined criterion, for example if its value is less than or equal to a predefined minimum. In other cases, cost function optimization may include cost function minimization. In this way, both the first depth estimation 230 and the second depth estimation 240, as well as the uncertainty measurement 235 and the uncertainty measurement 245, serve as constraints on the fusion depth estimation 280 to be acquired and fused. The accuracy of the depth estimation 280 is improved.

However, it will be understood that the use of cost functions is just an example. In other embodiments, the first depth estimation and the second depth estimation can be stochastically fused using uncertainty measurements in different ways.

Accordingly, some embodiments herein provide an accurate reconstruction of the depth estimation of the scene, thereby facilitating the interaction between the robotic device and the real-world environment. Specifically, some embodiments herein enable real-time or near-real-time operation (as opposed to other depth estimation techniques) and in a variety of different environments, including outdoor and indoor locations. Designed to provide depth estimation for the scene.

FIG. 3 is a schematic diagram of the image processing system 300 according to a further embodiment. The image processing system 300 of FIG. 3 is similar to the image processing system 200 of FIG. 2 in various respects. The corresponding function of FIG. 3, which is the same as the function of FIG. 2, is given the same reference number, but is incremented by 100.

In FIG. 3, the fusion engine 370 is configured to receive a surface orientation estimation 320 and an uncertainty measurement 325 of the surface orientation estimation 320 in addition to the first depth estimation, the second depth estimation, and the uncertainty measurement. It is composed of. The fusion engine 370 is configured to stochastically fuse the first depth estimation and the second depth estimation using the surface orientation estimation 320 and the uncertainty measurement 325 of the surface orientation estimation 320.

For example, the surface orientation estimation 320 indicates the orientation or tilt of the surface corresponding to the pixels or other image areas of the image of the scene captured by the capture device. For example, surface orientation can be considered to capture the surface orientation angle of pixels or other image areas of the image of the scene captured by the capture device. For example, surface orientation corresponds to a surface normal, which is the axis perpendicular to a given surface. In other cases, the surface orientation estimation 320 may accommodate measurements of surface gradients, such as surface changes. Using the surface orientation of multiple pixels, an index of surface properties corresponding to multiple pixels can be obtained. For example, a relatively smooth and unchanged surface can have a relatively constant surface orientation. Conversely, highly textured surfaces can be associated with a variety of different surface orientations.

The surface orientation estimation 320 and the uncertainty measurement 325 of the surface orientation estimation 320 can be obtained by various different methods. For example, the image of the scene may be processed to identify the uncertainty measurement 325 of the surface orientation estimation 320 and the surface orientation estimation 320 based on changes in photometric properties, such as pixel intensity values of the pixels of the image.

FIG. 4 is a schematic diagram of the image processing system 400 according to still another embodiment. The image processing system 400 of FIG. 4 is similar to the image processing system 300 of FIG. 3 in many respects, but clearly illustrates the geometric reconstruction engine 430 and the neural network architecture 420.

In FIG. 4, the frame 410 of the video data is received. Frame 410 is captured by a capture device such as a camera and includes a view of the scene. It should be appreciated that in other cases, the image processing system 400 of FIG. 4 can be used to process image data representing still images rather than video data representing video.

The frame 410 is processed by the geometric reconstruction engine 430 and the neural network architecture 420. The geometric reconstruction engine 430 and the neural network architecture 420 may be configured to generate a first depth data 450 and a second depth data 460, as described with reference to FIG.

In the embodiment of FIG. 4, the second depth data 460 includes surface orientation estimation and uncertainty measurement of surface orientation estimation. In this embodiment, the surface orientation estimation and the uncertainty measurement of the surface orientation estimation are generated by the neural network architecture 420 in addition to the second depth estimation and the second depth estimation uncertainty measurement.

The neural network architecture 420 of FIG. 4 may include one or more neural networks and is configured to receive frame pixel values of video data such as frame 410 shown in FIG. The neural network architecture 420 predicts a depth estimate for each of the first sets of image parts and at least one surface orientation estimate for each of the second sets of image parts to generate a second depth estimate. Is configured to predict. The first set of image portions may be the same as or different from the second set of image portions. For example, the first set and the second set of image parts may be completely overlapped, partially overlapped, or not overlapped at all. Depth estimation and at least one surface orientation can be obtained at different resolutions. In such cases, the resolution of one or both of the depth estimation and at least one surface orientation can then be modified, for example by interpolation, to obtain the desired resolution. For example, at least one surface orientation can be obtained at a lower resolution than the depth estimate, but can then be upscaled to the same resolution as the depth estimate. The neural network architecture 420 is also configured to predict one or more uncertainty measurements associated with each depth estimation and one or more uncertainty measurements associated with each surface orientation estimation.

The first depth data 450 and the second depth data 460 are stochastically fused using the fusion engine 470 to obtain a fusion depth estimation 480. In the figure, the fusion engine 470 also uses at least one surface orientation to obtain a fusion depth estimate of 480. In an embodiment where the cost function is optimized to obtain a fusion depth estimate 480, the cost function may include a third cost term associated with at least one surface orientation estimate. In such cases, the third cost term is a fusion depth estimate, a surface orientation estimate (eg, obtained from the neural network architecture 420), and an uncertainty value per at least one surface orientation estimate (eg, surface). It may include a function) (obtained from uncertainty measurements for each orientation estimation). For example, the third cost term may include the sum of the cost terms for each surface orientation estimation. The cost function optimization may be as described for FIG. 2, but with the addition of surface orientation information.

By acquiring the fusion depth estimation 480 using the surface orientation information, the accuracy of the fusion depth estimation 480 can be further improved. For example, surface orientation estimation (and its associated uncertainty measurements) can impose constraints between a given pixel and its adjacent pixels. In this way, the overall consistency of the fusion depth estimate 480 can be improved.

FIG. 5 is a schematic view showing the uncertainty measurement 325 of the surface orientation estimation 320 and the surface orientation estimation 320 according to the example 500. In FIG. 5, the surface orientation estimation 320 has a depth gradient estimation 510 in the first direction (direction along the x-axis in this case) and a direction orthogonal to the first direction (in this case where a Cartesian coordinate system exists). Includes a depth gradient estimate of 520 (in the direction along the y-axis).

For example, a depth gradient estimate in a given direction represents an estimate of the change in depth of a scene (eg, a scene captured in an image) in that given direction. Depth gradient estimation can be used to identify rapid or distinctive changes in depth in an image of a scene. For example, the depth gradient can be relatively high in the region of the image corresponding to that portion of the scene that has different depths across a portion of the scene. Conversely, depth gradients can be relatively low in other areas of the image that correspond to another portion of the scene that resides at a relatively constant depth with respect to the camera. By estimating depth gradients in two different directions (such as two directions that are orthogonal to each other, that is, perpendicular to each other), the depth characteristics of the scene captured in the image can be identified more accurately and / or more efficiently. ..

In other embodiments, the surface orientation estimation 320 may include other orientation estimates in addition to or in place of the depth orientation estimates 510 and 520. For example, the surface orientation estimation 320 may include a surface normal estimation.

As shown in FIG. 5, in some cases, there is a corresponding uncertainty measurement for each surface orientation estimation. Therefore, in FIG. 5, the first uncertainty measurement 530 associated with the depth gradient estimation 510 in the first direction and the second uncertainty associated with the depth gradient estimation 520 in the direction orthogonal to the first direction. There is a certainty measurement 540.

Uncertainty measurements for each surface orientation estimate can be generated in a variety of different ways. For example, a neural network architecture (which can be used to generate a second depth estimate that is stochastically fused with a first depth estimate by a fusion engine) has been associated with surface orientation estimates, and each surface orientation estimation. It can be trained to produce the corresponding uncertainty measurement.

In some cases, the second depth estimation and / or surface orientation estimation (s) can be logarithmic estimation. This can facilitate the generation of these estimates by the neural network architecture, as negative values have numerical implications. Moreover, the difference between the two log depths (eg, corresponding to the gradient of the log depth) corresponds to the ratio of the two depths whose scale is invariant. Further, if the log depth gradient is predicted in two orthogonal directions (as in the embodiment of FIG. 5), a probabilistic fusion of the first depth estimation and the second depth estimation (eg, using the log depth gradient). ) Is linear and can be performed without dot product and normalization actions. Therefore, the fusion process can be executed more efficiently than in other cases.

FIG. 6 is a schematic diagram of the image processing system 600 according to still another embodiment. The image processing system 600 includes a monocular capture device 605, which is an embodiment of a capture device or camera that captures an image of a scene. The monocular capture device 605 is configured to capture video data representing a video of the scene. The scene is captured in the first frame of the video, which may be referred to as key frame 610 in the embodiment of FIG. The keyframe 610 corresponds, for example, to a frame of video for which a more complete depth estimate is obtained, and is more feature-rich than, for example, a new part or other part of the scene where the depth was not previously estimated. Corresponds to, or includes, the portion of the scene identified as. For example, the first frame of a video for which depth estimation has not been previously obtained can be considered a key frame. The keyframe can be a keyframe designated by an external system, for example an external SLAM system. In another case, the frame obtained after the monocular capture device 605 travels a distance that exceeds the threshold distance may be a key frame. Other frames captured by the monocular capture device 605 can be considered as reference frame 615.

The frames captured by the monocular capture device 605 in the embodiment of FIG. 6 (whether keyframes 610 or reference frames 615) are processed using the tracking system 625. The tracking system 625 is used to identify the posture of the monocular capture device 605 while observing the scene (eg while capturing a frame). As described with reference to FIG. 2, the tracking system 625 may include a motion sensor, which is configured to move a robotic device connected to or supporting the monocular capture device 605. It can be connected to the actuator or can form part of the actuator. Thus, the tracking system 625 may capture the odometry data and process the odometry data to generate a pose estimation for the monocular capture device 605.

In FIG. 6, the tracking system 625 captures the pose estimation 640 of the monocular capture device 605 while capturing the reference frame 615 and the pose estimation 635 of the monocular capture device 605 while capturing the key frame. Generate. The estimated orientations of the monocular capture device 605 640, 635, and the video data captured by the monocular capture device 605 are used by the geometric reconstruction engine 630 to be used by the geometric reconstruction engine 630 to provide depth from the frame of the video data to at least a subset of the pixels. An estimate is generated. In some cases, the geometric reconstruction engine 630 is configured to minimize metering errors and generate depth estimates. This will be further explained with reference to FIG. The geometric reconstruction engine 630 of FIG. 6 outputs a first depth data 650 including depth estimation and depth estimation uncertainty measurement.

In some cases, the first depth data 650 may be regenerated for each frame captured by the monocular capture device 605, for example the first depth data 650 may be associated with a key frame and further captured and processed. It can be updated iteratively for each additional reference frame that is made. The first depth data 650 can be generated in real time or near real time and can therefore be performed frequently, for example at a rate corresponding to the frame rate of the monocular capture device 605.

For frames identified as corresponding to keyframes 610, the image processing system 600 of FIG. 6 further includes processing of keyframes 610 to generate second depth data 660 using the neural network architecture 620. However, the processing of the reference frame 615 using the neural network architecture 620 may be omitted in some cases. This is schematically shown in FIG. 6 using a dashed line. The dashed line corresponds to a portion of the image processing pipeline that can be selectively executed by the image processing system 600. For example, the generation of the first depth data 650 may be performed on the frame regardless of whether the frame is the reference frame 615 or the key frame 610, while the generation of the second depth data may be performed on the key frame 610. On the other hand, it can be executed selectively. In the embodiment of FIG. 6, the second depth data 660 includes a second depth estimation, a second depth estimation uncertainty measurement, at least one surface orientation estimation, and a surface orientation estimation uncertainty measurement. But this is just an example. The neural network architecture 620 may be similar or identical to the neural network architectures of the other embodiments described herein.

The image processing system 600 of FIG. 6 also includes a fusion engine 670, which has a first depth estimation and a second depth estimation (acquired by the geometric reconstruction engine 630 and the neural network architecture 620, respectively). , Constructed to be statistically fused using the associated uncertainty measurement. In FIG. 6, the fusion engine 670 also uses at least one surface orientation estimation and the respective uncertainty measurements associated with at least one surface orientation estimation to make a first depth estimation and a second depth estimation. And statistically fuse. However, the use of at least one surface orientation estimation and each uncertainty measurement associated with at least one surface orientation estimation may be omitted in other cases.

The fusion engine 670 is configured to generate a fusion depth estimation 680 by statistically fusing the first depth estimation and the second depth estimation. The fusion engine 670 of FIG. 6 is also configured to specify the scale estimation when the first depth estimation and the second depth estimation are stochastically fused. For example, if the fusion engine 670 is configured to optimize the cost function to specify the fusion depth estimate 680, optimizing the cost function may include specifying the scale factor 685 of the fusion depth estimate 680. .. In such cases, the scale factor 685 indicates the scale of the fusion depth estimate 680 for the scene. Therefore, the scale factor can compensate for the scale inaccuracy of the scene provided by the first depth estimation and the second depth estimation. The scale factor can be scalar. For example, the first depth estimation has an arbitrary scale because the first depth estimation is generated based on the attitude provided by the monocular capture device 605 of FIG. However, the generation of scale factors makes it possible to obtain depth estimates at a particular scale.

In the case where the fusion depth estimation 680 is specified by optimizing the cost function, the first cost term of the cost function is the fusion depth estimation value, the first depth estimation value, the uncertainty value of the first depth estimation, And may include a function of scale factor. Optimizing the cost function may include iteratively modifying the scale factor as well as the fusion depth estimation 680 to identify the scale factor and fusion depth estimation 680 that optimize (eg, minimize) the cost function. In such cases, the cost function may also include a second cost term and / or a third cost term, as described with reference to FIGS. 2 and 4. In such cases, the second and / or third cost term may be independent of the scale factor.

As described, the second depth data 660 may be generated by the neural network architecture 620 less frequently than the first depth data 650 is generated by the geometric reconstruction engine 630. For example, for the key frame 610, both the first depth data 650 and the second depth data 660 can be generated. The key frame 610 in which the generation of the second depth data may be omitted may be less than the reference frame 615 in which the generation of the second depth data may be omitted.

As an embodiment, the scene may be captured in the first frame of the video data and a second depth estimate for the first frame of the video data may be received. The second depth estimation can be generated from the neural network architecture 620. Therefore, the first frame of the video data can be considered to be the key frame 615. In this embodiment, a plurality of first depth estimates are obtained. At least one of the plurality of first depth estimates is generated using a second frame of video data that is different from the first frame of video data. For example, a plurality of first depth estimates (generated by the geometric reconstruction engine 630) include a first depth estimate of the first frame (keyframe 610) and a second frame (reference). It may include a first depth estimation (which is frame 615). In this example, the fusion engine 670 is configured to process one of a second depth estimation and a plurality of depth estimates for each iteration and iteratively output the fusion depth estimation 680 of the scene. For example, upon receiving the first frame, the fusion engine 670 has a first depth estimate generated using the first frame and a second depth estimate generated using the first frame. Can be fused. However, upon receiving the second frame, the fusion engine 670 instead uses the first depth estimation generated using the second frame and the previously generated second using the first frame. Can be fused with the depth estimation of. In other words, the second depth estimate cannot be regenerated frame by frame and can instead be reused from the previous frame (such as the previous keyframe 615). In other words, the generation of the fusion depth estimate 680 can be iteratively repeated. The method may include determining whether to generate a second depth estimate for subsequent iterations. As described above, such a determination is based on the content of the scene captured in the image, for example, whether the content has changed significantly compared to the image before the scene (eg, a monocular capture device). (By moving 605), or based on whether the content is feature-rich or not, etc. In response to the determination that no second depth estimation is generated (eg for reference frame 615), these examples use the set of values of the previous second depth estimation to make the first depth estimation and the second. It includes stochastically fusing with the depth estimation of 2. This eliminates the need to process the image using the neural network architecture 620.

In such an embodiment, the first depth estimation may be generated more frequently than the second depth estimation (the second depth estimation may be slower because it uses the neural network architecture 620). In some cases, the fusion depth estimate 680 may be refined based on the updated first depth estimate and the existing second depth estimate. Therefore, the depth of the scene can be updated at a higher rate than in other cases where the depth of the scene is updated after both the first depth estimation and the second depth estimation have been updated. In fact, by generating the first depth estimation and the second depth estimation separately and then fusing the first depth estimation and the second depth estimation, the methods herein are described as other. It is more flexible and can be performed more efficiently than the method of.

FIG. 7A is a schematic diagram showing components of a computing system 700 that may be used to implement any of the methods described herein. The computing system 700 can be a single computing device (eg, desktop, laptop, mobile and / or embedded computing device), or is a distributed computing system distributed across multiple separate computing devices. Obtain (eg, some components may be performed by one or more server computing devices based on requests made over the network from one or more client computing devices).

The computing system 700 includes, for example, a video capture device 710 that provides frames of video including scene observation. The computing system 700 also includes a location and mapping concurrency (SLAM) system 720. SLAM systems in the field of robot mapping and navigation serve to build and update maps of unknown environments while at the same time locating robot devices associated with maps in the environment. For example, a robot device can be a device that builds, updates, and / or uses a map. The SLAM system 720 is configured to provide attitude data for the video capture device 710. The medium density multi-view stereo component 730 of the computing system 700 is configured to receive frames of attitude data and video to implement the geometric reconstruction engine described in the other embodiments above. The medium density multiview stereo component 730 can be said to be the aforementioned "medium density", the term "multiview stereo" in which the component 730 uses a continuous frame of data from a monocular (eg non-stereo) camera. Instead, we show that it works to simulate a stereo image pair to identify depth data. In this case, the frame from the moving camera can provide different views of the common environment, which makes it possible to generate depth data as described above. The computing system 700 also includes a neural network circuit 740, which is, for example, an electronic circuit that implements the neural network architecture described with reference to the above embodiments. The computing system 700 also includes an image processing system 750 configured to implement the fusion engine of the embodiments of the present specification. The image processing system 750 probabilistically fuses the first depth data from the medium density multi-view stereo component 730 and the second depth data from the neural network circuit 740 to acquire the fused depth data. ..

FIG. 7B is a schematic diagram showing the components of the robot device 760 according to the embodiment. The robot device 760 includes the computing system 700 of FIG. 7A. The robot device 760 also includes one or more actuators 770 that allow the robot device 760 to interact with the surrounding three-dimensional environment. At least a portion of the surrounding 3D environment may be shown in the scene captured by the video capture device 710 of the computing system 700. In the case of FIG. 7B, the robot device 760 may be configured to capture video data as the robot device navigates a particular environment (eg, by device 130 of FIG. 1A). However, in another case, the robot device 760 may scan the environment or manipulate video data received from a third party such as a mobile device or a user with another robot device. When the robot device 760 processes video data, the robot device 760 may be configured to generate a depth estimate of the scene, for example a fusion depth estimate.

The robot device 760 also includes an interaction engine 780 that includes at least one processor that controls one or more actuators 770. The interaction engine 780 of FIG. 7B is configured to interact with the surrounding 3D environment using fusion depth estimation. The interaction engine 780 can use fusion depth estimation to control one or more actuators to interact with the environment. For example, fusion depth estimation can be used to grab objects in the environment and / or avoid collisions with barriers such as walls.

Examples of functional components described herein with reference to FIGS. 7A and 7B may include dedicated processing electronic devices and / or by computer program code executed by the processor of at least one computing device. Can be carried out. In some cases, one or more embedded computing devices may be used. The components described herein may include at least one processor that operates in association with memory to execute computer program code loaded on a computer readable medium. The medium may include solid state storage such as erasable programmable read-only memory and the computer program code may include firmware. In other cases, the component may include a well-configured system-on-chip, application-specific integrated circuit, and / or one or more well-programmed field programmable gate arrays. In one case, the component may be implemented by a computer program code and / or dedicated processing electronic device within a mobile computing device and / or a desktop computing device. In one case, as in the previous case, or in place of the previous case, the component may be implemented by one or more graphical processing units that execute computer program code. In some cases, the component may be implemented by one or more functions implemented in parallel, eg, on multiple processors and / or cores of a graphics processing unit.

FIG. 8 is a schematic diagram showing Example 800 of various functions described with reference to FIGS. 1-7. FIG. 8 shows an example of a three-dimensional (3D) environment 805 that can be referred to as a scene. The 3D environment 805 includes a capture device 810 such as the capture device 120 of FIG. 1, and two objects 815 and 820. The capture device 810 is configured to capture the observations of the 3D environment 805 (eg, in the form of a still image or video). These observations include, for example, observations of objects 815, 820, positions of objects 815, 820 relative to each other, and other objects or features in a 3D environment (surfaces supporting objects 815, 820, or objects 815, 820). It may indicate the position of objects 815, 820 with respect to (such as the wall behind). An example of a video frame 825 showing an observation of the 3D environment 805 captured by the capture device 810 is also shown in FIG. As shown, two objects 815 and 820 are displayed within frame 825 of the video.

FIG. 8 also schematically shows an example of a first depth estimation 830 obtained by a geometric reconstruction engine. As shown, the presence of objects 815,820 in the scene is indicated by contours 832, 834 in the first depth estimation 830. Therefore, the first depth estimation 830 makes it possible to identify, for example, boundaries or other edges in an image (eg, corresponding to possible sudden changes in depth at the edges of objects in the scene).

A second depth estimate 835 obtained by the neural network architecture is also schematically shown in FIG. As illustrated, the presence of objects 815,820 in the scene is indicated by shading 836, 838 in the second depth estimation 835. For example, a shaded grayscale value indicates the relative depth of a portion of an object.

In the embodiment of FIG. 8, the two objects 815 and 820 project toward the capture device 810. For example, in FIG. 8, the first object 815 is a cylinder with a vertically extending longitudinal axis. Therefore, the center of the first object 815 bulges toward the capture device 810, and its side surface retracts from the capture device 810 (when viewed from the capture device 810). The shape of the first object 815 is captured by the second depth estimation 835, indicating that the depth of the first object 815 is reduced relative to the capture device 810 towards the center of the first object 815. (Indicated by the shaded area 836 that becomes darker towards the center of the first object in the second depth estimation 835). However, the edges of the first object are sharper or sharper in the first depth estimation 830 than in the second depth estimation 835. This is because the first depth estimation 830 can more accurately capture the depth of the high textured area of the scene, while the second depth estimation 835 more accurately captures the depth of the low textured (ie smooth) area of the scene. Indicates that it can be captured. This is further apparent from the second depth estimation 835, which shows that there are shadows in the upper left and lower left corners, and that these areas have a depth difference compared to the other areas of the scene. This difference is not identified in the first depth estimation 830 because it is a relatively subtle or small depth change.

FIG. 8 also shows a first depth gradient estimation 840 in the first direction (horizontal in this embodiment) and a second depth gradient estimation in the direction orthogonal to the first direction (vertical in this embodiment). An embodiment with 845 is shown schematically. The presence of objects 815 and 820 is indicated by arrows 842 and 844 in the first depth gradient estimation 840, respectively. The presence of objects 815 and 820 is indicated by arrows 846 and 848 in the second depth gradient estimation 845, respectively. As described with reference to the first depth estimation 830 and the second depth estimation 835, the object 815 inflates with respect to the capture device 810 along its longitudinal axis. This is because in the first depth gradient estimation 840, the arrows 842 are closer to each other in the central region than in the lateral direction of the object 815, where the object 815 curves more rapidly in the receding direction from the capture device 810 due to its cylindrical shape. (Indicates that the change in depth gradient is not rapid).

FIG. 9 is a flow diagram illustrating an exemplary method 900 for estimating the depth of a scene. Method 900 includes a first action 910 that produces a first depth estimate. A first depth estimate can be generated using the geometric reconstruction of the scene, and the geometric reconstruction of the scene is configured to output an uncertainty measurement of the first depth estimation. In the second operation 920, the neural network architecture is used to generate the second depth estimation. The neural network architecture is configured to output a second depth estimation uncertainty measurement. In the third operation 930, the first depth estimation and the second depth estimation are stochastically fused using uncertainty measurement to generate a fused depth estimation of the scene. The method 900 of FIG. 9 can be implemented using any of the systems described herein.

FIG. 10 is a flow diagram illustrating a further exemplary method 1000 for estimating the depth of a scene. Image data is acquired in the first operation 1010. Image data can be obtained from a capture device configured to capture an image of the scene. In the second operation 1020, the attitude estimation of the capture device while acquiring the image data of the first operation 1010 is generated. In the third operation 1030, a geometric reconstruction engine is used to obtain a medium density estimate of the depth of the scene, as described, for example, with reference to other embodiments herein. In the fourth operation 1040, as described with reference to FIG. 6, it is determined whether or not the image captured by the capture device is a key frame. If it is a key frame, a neural network output is generated in the fifth action 1050 to generate a second depth estimate. However, if the image does not correspond to a key frame, then in operation 1060, the existing neural network output (eg, obtained in the previous image) is used instead. The neural network output includes, for example, a second depth estimation and at least one surface orientation estimation, and an uncertainty measurement associated with each of the second depth estimation and at least one surface orientation estimation. Finally, in the seventh operation 1070, the first depth estimation of the third operation 1030 and the second depth estimation of the fifth operation 1050 or the sixth operation 1060 are, for example, the first depth estimation. And the second depth estimation are fused in a probabilistic way using the uncertainty measurements associated with each. In this embodiment, at least one surface orientation estimation and corresponding uncertainty measurement are also used during the fusion operation to obtain the fusion depth map and scale factor. The method 100 of FIG. 10 can be implemented using, for example, the system 600 of FIG.

FIG. 11 is a schematic diagram showing an embodiment 1100 of a processor 1110 and a non-temporary computer-readable storage medium 1120 including a computer executable instruction 1130. When executed by the processor 1110, the computer executable instruction 1130 causes a computer device such as a computing device equipped with the processor 1110 to estimate the depth of the scene. The command may result in a method similar to the exemplary method described above. For example, the computer-readable storage medium 1120 may be configured to store a plurality of first depth data 1140s that may be acquired for the plurality of reference frames, as described with reference to FIG. The computer-readable storage medium 1120 may also be configured to store a second depth estimate of keyframes 1150. The first depth data and the second depth data are stochastically fused, and a fusion depth estimation can be obtained. In FIG. 11, the first depth data 1140 and the second depth data 1150 are shown to be stored in the computer-readable storage medium 1120, but in other embodiments, the first depth data 1140 and the second depth data 1140. At least one of the depth data 1150 may be stored in storage external to the computer readable storage medium 1120 (but accessible by the computer readable storage medium 1120).

FIG. 12 is a schematic diagram showing the fusion of the first depth estimation and the second depth estimation of the scene according to a further embodiment. In FIG. 12, the geometric reconstruction is used to generate a first depth probability volume 1200 for the scene. The first depth probability volume 1200 is each of the first plurality of depth estimates (in this case including the first depth estimates discussed in other embodiments herein) and the first plurality of depth estimates. Includes a first plurality of uncertainty measurements, each associated with a depth estimate. Therefore, the first plurality of uncertainty measurements include a first depth estimation uncertainty measurement.

A first depth probability volume 1200 is schematically shown in FIG. 12, which is a simplified example for ease of illustration. In FIG. 12, the frame representing the observation of the scene contains _nine pixels labeled P1 to _P9 in FIG. Each of the pixels corresponds to an observation of different parts of the scene. For each of the pixels in FIG. 12, there are three depth estimates labeled D ₁ , D ₂ , and D ₃ (but in other embodiments there may be more or less depth estimates per pixel. be). Each depth estimate in FIG. 12 is associated with each uncertainty measurement. In FIG. 12, the uncertainty measurement associated with the _mth depth estimation Dm for the nth pixel P _n is labeled _unm . FIG. 12 shows an uncertainty measurement of the pixels in the top row (P ₁ , P ₂ , and P ₃ ). However, it should be understood that depth estimates for other pixels of the frame ( _P4-9 ) also have a corresponding uncertainty measurement (not shown in FIG. ₁₂ ). In this embodiment, the 3D configuration of the uncertainty measurement associated with each depth estimation for the 2D pixel array forms a 3D stochastic volume.

In FIG. 12, the uncertainty measurement associated with a given depth estimate of the first plurality of depth estimates of the first depth probability volume 1200 is given in a given area of the scene (its observation is given). Represents the probability that (captured in pixels) is present at the depth represented by a given depth estimate of the first plurality of depth estimates. Therefore, in FIG. 12, u ₁₁ represents the probability that the region of the scene captured by the first pixel P ₁ exists at a depth corresponding to the depth of the first depth estimation D ₁ .

FIG. 12 also includes a second depth probability volume 1202, which is generated for the scene using a neural network architecture. The second depth probability volume 1202 in this case is otherwise similar to the first depth probability volume 1200, with a second plurality of depth estimates including a second depth estimate and a second plurality of depth estimates. Includes a second plurality of uncertainty measurements associated with each depth estimation of. Therefore, the second plurality of uncertainty measurements include a second depth estimation uncertainty measurement. With respect to the first depth probability volume 1200, the uncertainty measurement associated with a given depth estimate of the second plurality of depth estimates is that a given area of the scene is the second plurality of depth estimates. Represents the probability of being at the depth represented by our given depth estimation.

In the embodiment of FIG. 12, the first depth probability volume 1200 and the second depth probability volume 1202 are associated with the same respective depth estimates (D ₁ , D ₂ , and D ₃ ) for a given pixel. Includes uncertainty measurements. However, it should be understood that in other embodiments, the depth estimates of the first depth probability volume 1200 and the second depth probability volume 1202 may differ from each other. For example, one of the first depth probability volume 1200 and the second depth probability volume 1202 may have a larger number of depth estimates and / or different depth estimates.

Since the accuracy of the geometric reconstruction for a given part of the scene and the accuracy of the neural network architecture are usually different, the first depth probability volume 1200 and the second depth probability volume 1202 give a given depth estimate. It should be understood that the associated uncertainty measurements are usually different. This can result in different probability distributions for a given part of the scene, depending on whether geometric reconstruction or neural network architecture is used. For example, if a given technique (with geometric reconstruction, or the use of a neural network architecture) cannot accurately characterize the depth of a given part of the scene, then a pixel representing a given part of the scene The associated depth probability distribution can be relatively uniform, making it difficult to determine the most likely depth of that part of the scene. Conversely, if a given technique can accurately determine the depth of a given part of the scene, then the depth probability distribution is a depth estimate that corresponds to the depth of a given part of the scene, with sharper peaks. May have.

Depth estimation associated with the first depth probability volume 1200 and the second depth probability volume 1202 by fusing the uncertainty measurements associated with the first depth probability volume 1200 and the second depth probability volume 1202. It can be fused probabilistically, which produces a fusion depth probability volume 1204. This is schematically shown in FIG. 12, where the fusion depth probability volume 1204 is similar to the first depth probability volume 1200 and the second depth probability volume 1202, but with the depth estimation of the first depth probability volume 1200. Uncertainty measurement u _nm different from the value of uncertainty measurement u _nm of the first depth probability volume 1200 and the second depth probability volume 1202 by the probabilistic fusion with the depth estimation of the second depth probability volume 1202. Usually contains the value of. By probabilistically fusing the first depth probability volume 1200 and the second depth probability volume 1202 in this way, information about the depth of the scene from two different sources (geometric reconstruction and neural network architecture). Can be combined, and the accuracy of depth estimation is usually improved compared to the case where each source is used individually.

FIG. 13 is a schematic diagram of an exemplary system 1300 for acquiring a second depth probability volume 1302 that may be similar to or identical to the second depth probability volume 1202 of FIG. The system 1300 is configured to receive frames 1304 representing observations of the scene. The frame 1304 of this example is represented by image data representing an image of the scene. The neural network architecture 1306 of the system 1300 is configured to process frames 1304 to generate a second depth probability volume 1302, which in this case is a second plurality of depth estimates. 1308, and a second plurality of uncertainty measurements 1310 (in this case, the probabilities that a given area of the scene is at the depth represented by the depth estimation of the second plurality of depth estimates 1308, respectively. Represents).

The system 1300 of FIG. 13 is configured to output a second plurality of depth estimation 1308s, including a plurality of depth estimation sets associated with different parts of the image of the scene. Therefore, when using the system 1300 to output the second depth probability volume 1202 of FIG. 12, the depth estimates D ₁ , D ₂ , and D ₃ for a given pixel are considered to correspond to the depth estimation set. obtain. Thus, the system 1300 of FIG. 13 can be used to process an image containing a plurality of pixels to generate a plurality of depth estimation sets associated with each of the different pixels of the plurality of pixels.

The neural network architecture 1306 of the embodiment of FIG. 13 is configured to output an uncertainty measurement 1310 associated with each depth estimate 1308 having a predefined value. In other words, rather than outputting a single depth value for each pixel, the neural network architecture 1306 of FIG. 13 is uncertain for each of the multiple predefined discrete depth estimates 1308 for each pixel. It is configured to output the sex measurement 1310. In this way, the neural network architecture 1306 outputs a discrete depth probability distribution over a given range for a given pixel, which is nonparametric in this case. This allows the neural network architecture 1306 to represent uncertainty about the expected depth (represented by the uncertainty measurement associated with a given depth estimate, in this case by the probability value). Will be). This also allows the neural network architecture 1306 to make multiple hypothetical depth predictions, which, when fused with the depth estimates obtained by geometric reconstruction, estimate the depth of the scene more accurately. It becomes possible to do.

There can be non-uniform spacing between the predefined values output by the neural network architecture 1306. In such a technique, the neural network architecture 1306 is configured to output a depth probability distribution with variable resolution over the depth range occupied by the depth estimation for a given pixel. For example, a predefined value may include multiple logarithmic depth values within a predefined depth range (a range that can be all or part of the depth range occupied by depth estimation). By using log-depth parameterization, the depth range can be evenly divided in logarithmic space. This provides higher depth resolution in areas closer to the capture device used to capture the observation of the scene and lower resolution in areas farther away.

For a given pixel in an image processed using the system 1300 of FIG. 13, the uncertainty measurement 1310 associated with each depth estimation of the second plurality of depth estimates 1308 is schematically shown in FIG. Shown. In FIG. 14, the uncertainty measurement is a probability density value shown on the y-axis 1400, and the depth estimation is a metric log depth value shown on the x-axis 1402. The depth estimation in FIG. 14 has discrete values. Therefore, FIG. 14 shows a discrete probability distribution 1400 in the form of a bar graph 1406.

In some cases, a continuous probability function can be obtained from the discrete probability distribution 1400 in order to reduce the discretization error and facilitate the acquisition of the depth estimation of the scene. The continuous probability function 1408 obtained from the discrete probability distribution 1400 is schematically shown in FIG. The continuous probability function 1408 can be a smooth function and is further discussed below with reference to FIG.

Revisiting FIG. 13, a variety of different neural network architectures can be used as the neural network architecture 1306. In one example, the neural network architecture 1306 includes a residual neural network (ResNet) encoder followed by three upsample blocks, where each upsample block is a bilinear upsampling layer, coupled with an input image, and then. Contains two convolutional layers, whereby the output has the same resolution as the input image representing the observation of the scene.

θは、ニューラルネットワークアーキテクチャ１３０６の重みの集合であり、Ｋは、奥行き範囲が離散化されるビンの数であり、ｋ_ｉ ^＊は、ピクセルｉについてのグラウンドトゥルース奥行きを含むビンのインデックスであり、ｐ_θ，ｉ（ｋ_ｉ ^＊＝ｊ）は、グラウンドトゥルース奥行きがビンｊ内である確率に関するニューラルネットワークアーキテクチャ１３０６の予測である。しかし、これは単なる例に過ぎず、他の実施例では、他の損失関数が使用され得る。 A sequence loss function can be used to train the neural network architecture 1306 of FIG. 13 to predict the probability values associated with the discrete depth estimation. An example of a suitable sequence loss function L (θ) is as follows.

θ is the set of weights in the neural network architecture 1306, K is the number of bins whose depth range is discretized, and ki ^* is the index of bins containing the ground truth depth for pixel _i . p _{θ, i (} ki ^* = j) is a prediction of the neural network architecture 1306 regarding the probability that the ground truth depth is within bin j. However, this is just an example, and other loss functions may be used in other embodiments.

Looking at FIG. 15, FIG. 15 is a schematic diagram of an exemplary system 1500 for acquiring a first depth probability volume 1502 that may be similar to or identical to the first depth probability volume 1200 of FIG. System 1500 of FIG. 15 processes a first frame 1504 representing a first observation of a scene, and a second frame 1506 representing, for example, a second observation of the scene before or after the first observation of the scene. It is configured as follows. The first and second observations of the embodiment are finally partially overlapped (eg, both include observations of the same part of the scene).

The first frame 1504 and the second frame 1506 are processed by the photometric error calculation engine 1508 to generate a set of photometric errors 1510 for each of the plurality of parts of the first frame 1504, each of which has a photometric error 1510. Associated with each different depth estimate of the first plurality of depth estimates 1512. The photometric error warps the first frame 1504 to the second frame 1506 for each of the first plurality of depth estimates 1512 and identifies the difference between the warped first frame 1504 and the second frame 1506. Can be obtained by doing so. In some cases, the difference is, for example, the sum of the squared differences between the pixel values of the warped first frame 1504 and the pixel values of the second frame 1506 for a patch of pixels of 3x3 pixel size. , This is just an example. Warping the first frame 1504 in this way is to map the pixels of the first frame 1504 to the corresponding positions in the second frame 1506, as described, for example, with reference to FIG. Can be considered corresponding. For example, the photometric error calculated iteratively for each depth value to identify the depth values that minimize the photometric error in the embodiment described with reference to FIG. 2 is the photometric error in the embodiment of FIG. It can be output as a set of metering errors 1510 by the calculation engine 1508. The embodiment of FIG. 2 involves calculating the uncertainty measurement associated with the depth estimation obtained by minimizing the photometric error, eg, using the Jacobian term. In contrast, in the embodiment of FIG. 15, the set of photometric errors 1510 themselves are treated as their respective uncertainty measurements associated with each depth estimate.

The warp of the first frame 1504 is intended to duplicate the second observation of the scene captured in the second frame 1506 (eg, the second of the cameras while capturing the second frame 1506). As observed with a camera that has the same posture as the posture of). The first frame 1504 is thus transformed for each of the first plurality of depth estimates 1512 (each of the depth estimates is the first orientation of the camera while capturing the first frame 1504). Is the hypothetical depth of the scene based on). Normally, the depth of the scene relative to the first pose is non-uniform, but assuming the entire scene is at the same depth, the measurement error of the depth estimation is calculated pixel by pixel (or image patch by image). By doing so, the warp can be executed more efficiently. This technique can be iterated over for the first plurality of depth estimates 1512 for each of the plurality of pixels in the first frame 1504 to generate a cost volume, from the cost volume to the first depth probability volume 1502. Can be obtained. It should be understood that the first and second poses of the camera can be obtained using any suitable method, as described with reference to FIG.

Each probability value is output by the neural network architecture 1306 to simplify the fusion of the first depth probability volume 1502 and the second depth probability volume obtained using the system 1300 of FIG. The midpoint of the depth bin can be used as the respective depth estimate for the first depth probability volume 1502. However, in other embodiments it does not have to be.

In some embodiments, the first frame 1504 and the second frame 1506 are normalized before the first frame 1504 is warped and / or before the set of photometric errors 1510 is calculated. For normalization, for each of the first frame 1504 and the second frame 1506, the average pixel value is subtracted from each of the pixel values, and the output values are each of the first frame 1504 and the second frame. It can be done by dividing by the standard deviation of 1506. This allows the warped first frame 1504 and second frame 1506 with respect to a given depth estimation without being overly affected by changes in illumination between the first and second observations of the scene. It is possible to more accurately identify the fundamental photometric difference of.

As mentioned above, the set of photometric errors 1510 acquired by the photometric error calculation engine 1508 in FIG. 15 can be considered to form a cost volume. To obtain the first depth probability volume 1502 from the set of metering errors 1510, the system 1500 of FIG. 15 includes a scaling engine 1512, which sets the metering error 1510 to each probability value 1514 (first plurality). It is configured to scale to (which can be considered to correspond to the uncertainty measurement associated with the depth estimation) of the depth estimation 1512. In one example, scaling is the negative squared metering error of each pixel such that after scaling the sum of the negative squared metering errors for each of the first plurality of depth estimates 1512 for a given pixel is 1. Includes scaling the values of individually. The scaled values are then used as the respective probability values 1514 associated with a given depth estimate of the first plurality of depth estimates 1512, thereby producing a first probability volume 1502. obtain.

FIG. 16 is a flow diagram illustrating an exemplary method 1600 for obtaining a fusion depth estimate of a scene, where the observation of the scene is captured in multiple frames, each containing an array of pixels. The fusion depth estimation is obtained using a first depth probability volume and a second depth probability volume as described with reference to FIGS. 12-15.

Item 1602 in FIG. 16 includes acquiring the fusion probability volume. In item 1602, the first depth estimation and the second depth estimation (forming a part of the first depth probability volume and the second depth probability volume, respectively) are the first plurality of uncertainty measurements. By combining with a second plurality of uncertainty measurements, they are stochastically fused to produce a fusion probability volume. The first plurality of uncertainty measurements and the second plurality of uncertainty measurements can be combined in a variety of different ways. In one case, where the first plurality of uncertainty measurements and the second plurality of uncertainty measurements are the probability values associated with the first depth probability volume and the second depth probability volume, the first plurality. By combining (eg, multiplying) the probability value associated with the depth estimation of the depth estimation with the probability value associated with the corresponding depth estimation of the second plurality of depth estimations, the depth estimation By acquiring the fusion value for each, the fusion probability volume is acquired. In some cases, the fusion values for each of the depth estimates for a given pixel are then scaled to a total of 1 to generate fusion probability values for each of the depth estimates. However, in other cases, the probability values associated with, for example, the first plurality of depth estimates and the second plurality of depth estimates have already been previously scaled to a total of 1 for a given pixel. If so, it can be omitted.

In the embodiment of FIG. 16, suitable functions (items 1604 and 1606 of FIG. 16) are used to avoid quantification of the depth estimation of the scene obtained using the fusion probability volume and in subsequent optimization steps. The depth probability function is obtained using the fusion probability volume to obtain (referred to and further discussed). The depth probability function represents the parameterization of the fusion probability volume and makes it possible to obtain continuous depth values (not just a discrete depth estimation of the fusion probability volume). Depth probability functions can be obtained using any suitable technique for parameterizing discrete distributions, such as the kernel density estimation (KDE) technique, which uses Gaussian basis functions.

In items 1604 and 1606 of FIG. 16, the fusion depth estimation of the scene is obtained from the fusion probability volume (in this case, it is obtained from the depth probability function obtained from the fusion probability volume, but this is only an example. ). In the embodiment of FIG. 16, acquiring the fusion depth estimation of the scene comprises optimizing the cost function in item 1604. The cost function in this case includes a first cost term obtained using the fusion probability volume and a second cost term that includes a local geometric constraint on the depth value. The cost function c (d) can be expressed as follows.
c (d) = c ₁ (d) + λc ₂ (d)
d is the estimated depth value, c ₁ (d) is the first cost term, c ₂ (d) is the second cost term, and λ is the second cost for the cost function. A parameter used to adjust the contribution of a term. The parameter λ can be adjusted experimentally to obtain a good estimate of the depth value. A suitable value for the parameter λ in one case is 1 × ¹⁰⁷ , but this is just an example.

ｆ_ｉ（ｄ_ｉ）は、奥行きｄ_ｉで評価された、所与の入力フレーム（シーンの観察を表す）のピクセルｉについての奥行き確率関数の出力である。 The first cost term depends on the fusion probability volume and, in the embodiment of FIG. 16, depends on the depth probability function obtained from the fusion probability volume. In this case, the first cost term can be expressed as:

f _i ( _di ) is the output of the depth probability function for pixel _i of a given input frame (representing the observation of the scene), evaluated at depth di.

By fusing the first depth-probability volume with the second depth-probability volume, the fusion-probability volume is usually more locally consistent than when geometric reconstruction or neural network architecture is used alone. Become. In the embodiment of FIG. 16, local consistency is improved by including a second cost term that can be considered a normalization term. The second cost term in this case imposes local geometric constraints during the optimization of the cost function, which better maintains the local geometry.

A system 1700 for obtaining fusion depth estimates 1702 using method 1600 of FIG. 16 is schematically shown in FIG. The fusion depth probability volume 1704 is input to the depth estimation engine 1706 of the system 1700. The depth estimation engine 1706 performs the cost function optimization of item 1604 of FIG. The first cost term of the cost function depends on the fusion depth probability volume 1704 input to the depth estimation engine 1706 and can be expressed in this case using equation c ₁ (d) above. Therefore, in FIG. 17, the depth estimation engine 1706 is configured to obtain a depth probability function from the fusion depth probability volume 1704 in order to calculate the first cost term. However, in other cases, the first cost term can be obtained from the fusion depth probability volume 1704 itself, or the depth estimation engine 1706 is configured to receive a depth probability function rather than a fusion depth probability volume 1704. obtain.

The system 1700 of FIG. 17 also includes an additional neural network architecture 1708, which is to receive an input frame 1710 representing an observation of the scene in which the fusion depth estimation is generated, and a second cost term of the cost function. It is configured to generate the geometric constraint data 1712 used in the generation. The input frame 1710 in this example is an input frame processed by the neural network architecture to generate a second depth probability volume and is processed by a geometric reconstruction engine to generate a first depth probability volume. It is one of the frames that are made, but this is just an example.

The geometric constraint data 1712 of the embodiment of FIG. 17 represents surface orientation estimation and occlusion boundary estimation. A second cost term is generated using surface orientation estimation and occlusion boundary estimation. For example, the surface orientation estimation represents a surface normal for a given pixel in input frame 1710 and is predicted by the further neural network architecture 1708. Any well-trained neural network architecture can be used as a further neural network architecture 1708, as will be appreciated by those of skill in the art. Using surface orientation estimation in the second cost term improves the maintenance of local geometry in the fusion depth estimation obtained by optimizing the cost function. For example, if the surface orientation estimates are similar for adjacent pixels (eg, these pixels have similar orientations, indicating that a surface of a continuous plane is expected), the second cost term is usually smaller.

However, scenes usually contain depth discontinuities at the boundaries of objects (which can be referred to as occlusion boundaries). At such boundaries, the surface orientation estimates of adjacent pixels in the input frame that represent the observation of the scene are usually different from each other. Surface orientation estimates can be unreliable in such areas, as some of the objects in these areas can be occluded by sudden changes in the depth of the objects at the object boundaries. Therefore, observations of these parts of the object cannot be present in the input frame, which is the reliability of the surface orientation estimation and the cost based on the difference between the surface orientation estimates of adjacent pixels of the image representing the observation of the scene. It can affect the reliability of a term.

To compensate for this, the second cost term in the embodiment of FIG. 17 masks the normalization term at the occlusion boundary. In other words, for pixels that correspond to unreliable regions of input frame 1710, such as pixels that correspond to occlusion boundaries, the second cost term contributes less and corresponds to more reliable regions. In order to adjust the contribution of the second cost term for each pixel of the input frame 1710 so that the contribution of the second cost term is higher, the second cost term may be a value of 0 to 1, or the like. Weighted by value. For example, for pixels on the occlusion boundary, the second cost term can be weighted with a zero value weight, so for these pixels the second cost term contributes to the optimization of the cost function. do not do.

In some cases, the additional neural network architecture 1708 outputs the probability that a given pixel belongs to the occlusion boundary as an occlusion boundary estimate. In such cases, pixels can be considered to be on occlusion boundaries if this probability is greater than or equal to a predetermined threshold, such as 0.4.

ｂ_ｉ∈｛０、１｝は、入力フレーム１７１０のピクセルｉのオクルージョン境界推定に基づいたマスクの値であり、＜．，．＞は、ドット積演算子を表し、

は、さらなるニューラルネットワークアーキテクチャ１７０８により出力された表面配向推定であり、Ｋは、入力フレーム１７１０をキャプチャするのに使用されたカメラに関連付けられた固有パラメータを表す行列であり（時にカメラ固有行列と称される）、

は、ピクセルｉの均一ピクセル座標を表し、Ｗはピクセル単位の画像の幅である。 In the embodiment of FIG. 17, the second cost term c ₂ (d) generated by the depth estimation engine 1706 and used to optimize the cost function can be expressed as:

b _i ∈ {0, 1} is the mask value based on the occlusion boundary estimation of the pixel i of the input frame 1710, and <. ,. > Represents the dot product operator

Is a surface orientation estimation output by the further neural network architecture 1708, where K is a matrix representing the camera-specific parameters used to capture the input frame 1710 (sometimes referred to as the camera-specific matrix). Will be),

Represents the uniform pixel coordinates of pixel i, and W is the width of the image in pixel units.

In FIG. 17, the depth estimation engine 1706 uses gradient descent to optimize the cost function and obtain a fusion depth estimate 1702 corresponding to the depth value d that minimizes the value of the cost function (item 1606 in FIG. 16). ). However, this is just an example, and in other cases fusion depth estimates can be obtained using different optimization techniques.

FIG. 18 is a flow diagram illustrating an exemplary method 1800 for obtaining a second fusion depth probability volume. Using method 1800 of FIG. 18, from the first fusion depth probability volume associated with the first frame of the video representing the first observation of the scene, the second of the video data representing the second observation of the scene. A second fusion depth probability volume associated with a frame can be obtained, the first frame being, for example, before or after the second frame. By using the first fusion depth probability volume to obtain the second fusion depth probability volume, information about the depth of the scene can be retained across multiple frames. Thereby, the depth estimation of the second frame can be improved as compared with the case of recalculating the depth estimation of the second frame without using the information from the first frame.

Since the first fusion depth probability volume represents the depth probability distribution of each pixel of the first frame, it is not obvious to incorporate the information represented by the first fusion depth probability volume into the second frame. To address this, item 1802 in FIG. 18 includes converting a first fusion depth probability volume into a first occupancy probability volume. The first fusion depth probability volume of the first frame can be obtained using any of the methods described with reference to FIGS. 12-17. The first occupancy stochastic volume can be considered as a occupancy-based stochastic volume, and thus for each depth along the light beam propagated from the camera in the first orientation associated with the capture of the first frame facing the scene. , There is a probability that the related points in space are occupied.

In one example, the first occupancy probability volume is on a voxel _{Sk, i} (eg, a ray associated with pixel i in the first frame, provided that the depth belongs to the bin j of the first depth probability volume. The first depth probability along is obtained by first identifying the probability that the (three-dimensional volume element corresponding to the depth estimation associated with the bin k of the volume) is occupied.

ｐ_ｉ（ｋ_ｉ ^＊＝ｋ）は、第１のフレームのピクセルｉについての第１の奥行き確率体積のビンｋの確率値であり、Ｋは、ピクセル単位の第１のフレームの幅である。 From this, the first occupancy probability volume p ( _{Sk, i} = 1) can be obtained using the following equation.

p _i ( _ki ^* = k) is the probability value of the bin k of the first depth probability volume for the pixel i of the first frame, and K is the width of the first frame in pixel units.

Item 1804 in FIG. 18 warps the first occupancy probability volume based on the stance data representing the posture of the camera while observing the scene, and the second occupancy probability associated with the second frame. Includes obtaining volume. Warping the first occupied probability volume can be otherwise similar to warping the first frame to obtain the photometric error 1510 described with reference to FIG. 15, and is usually the first. The first posture data representing the first posture of the camera while capturing one frame, and the second posture data representing the second posture of the camera while capturing the second frame. Is used as posture data. In this way, the first occupancy probability volume can be warped to the second frame. In some cases, the second frame may contain some pixels in which the corresponding warped first occupancy volume does not exist. For these pixels, a predetermined value (eg, the default value), such as a value of 0.01 (just an example), may be used for the occupancy probability.

この式を使用して、第２のフレームの複数のピクセルのそれぞれについて、第２の融合奥行き確率分布のそれぞれのビンの確率値が生成され得る。次に、第２の融合奥行き確率分布は、１つの光線に沿って分布の合計が１となるようにスケーリングされ、第２の融合奥行き確率体積が取得され得る。次に、第２のフレームの融合奥行き推定が、例えば図１６及び図１７を参照して説明されるように、第２の融合奥行き確率体積から取得され得る。 In item 1806 of FIG. 18, the second occupancy probability volume is converted to the second fusion depth probability volume associated with the second frame. This transformation is performed using the following equation, and the probability value _pi (ki ^* = _k ) of the bin k of the second depth probability distribution for the pixel i of the second frame can be obtained.

Using this equation, the probability values for each bin of the second fusion depth probability distribution can be generated for each of the plurality of pixels in the second frame. Next, the second fusion depth probability distribution can be scaled so that the sum of the distributions is 1 along one ray, and the second fusion depth probability volume can be obtained. The fusion depth estimation of the second frame can then be obtained from the second fusion depth probability volume, as described, for example, with reference to FIGS. 16 and 17.

FIG. 19 is a flow diagram illustrating an exemplary method 1900 for estimating the depth of a scene according to a further embodiment.

At item 1902, the geometric reconstruction of the scene is used to generate a first depth probability volume for the scene. The first depth probability volume is, for example, the same as or similar to the first depth probability volume described with reference to FIG. 12, and can be generated, for example, as described with reference to FIG.

At item 1904, a neural network architecture is used to generate a second depth probability volume for the scene. The second depth probability volume is, for example, the same as or similar to the second depth probability volume described with reference to FIG. 12, eg, generated as described with reference to FIGS. 13 and 14. obtain.

In item 1906, the fusion depth probability volume of the scene is generated using the first depth probability volume and the second depth probability volume, and in item 1908, the fusion depth probability volume of the scene is used. Depth estimation is generated. The generation of the fusion depth probability volume and fusion depth estimation of items 1906 and 1908 of FIG. 19 may be similar to or the same as the method of FIGS. 16 and / or FIG.

FIG. 20 is a schematic diagram of an image processing system 2000 for estimating the depth of a scene according to a further embodiment. The image processing system 2000 includes a fusion engine 2002, which comprises a first depth probability volume 2004 from the geometric reconstruction engine 2006 and a second depth probability volume 2008 from the neural network architecture 2010. Upon receiving, the first depth probability volume 2004 and the second depth probability volume 2008 are fused, and the fusion depth probability volume 2012 of the scene is output. The image processing system 2000 also includes a depth estimation engine 2014, which estimates the depth of the scene using the fusion depth probability volume 2012 (which can be referred to as fusion depth estimation 2016).

In the embodiment of FIG. 20, the image processing system 2000 is configured to process input frames 2018 representing each observation of the scene to generate a fusion depth estimate 2016. The input frame 2018 includes, for example, a first frame, which is by the neural network architecture 2010 to generate a second depth probability volume 2008, as described, for example, with reference to FIGS. 12-14. It is processed. The input frame 2018 may also include a second frame. In such an embodiment, both the first frame and the second frame are processed by the geometric reconstruction engine 2006, as described, for example, with reference to FIGS. 12 and 15. Depth probability volume 2004 can be generated. The generation of the fusion depth probability volume 2012 by the fusion engine 2002 and the generation of the fusion depth estimation 2016 by the depth estimation engine 2014 may be as described, for example, with reference to FIGS. 16 and 17.

The above embodiment should be understood as an example. Further examples are envisioned.

In the embodiment of FIG. 16, the cost function includes a first cost term and a second cost term. In other embodiments that are otherwise identical or similar to the embodiment of FIG. 16, the cost function may not include a second cost term, eg only a first cost term.

The fusion depth probability volume acquired for the first frame by the method 1900 of FIG. 19 or the system 2000 of FIG. 20 is warped as described with reference to FIG. 18 and the depth probability volume of the second frame is acquired. Can be done.

It should be understood that the estimation of the depth of the scene as described with reference to FIGS. 12-17 and 19 and 20 need not be performed for each observation of the scene. Alternatively, the depth can be estimated for a subset of observations (eg, a subset of frames) that can be referred to as keyframes. This can reduce the processing requirements. Similarly, the method of FIG. 19 does not need to be performed for each frame following a key frame whose depth has been estimated. For example, if the posture of the camera changes significantly between the first frame and the second frame, the method of FIG. 19 may be omitted.

For the first frame of the video that estimates the depth, the depth may be estimated using the neural network architecture as described above (eg, fusing the second depth probability volume with the first depth probability volume). Not by calculating the depth estimation from the second depth probability volume). After acquiring at least one additional frame of video, a first depth probability volume is calculated and fused with a second depth probability volume to obtain a fusion depth probability volume to generate a fusion depth estimate for the scene. obtain.

In the embodiments of FIGS. 16 and 17, a fusion depth estimate is generated using a cost function based on a depth probability function derived from the fusion depth probability volume. This technique tends to work more accurately in featureless regions that are susceptible to false minimums because the predictions from the neural network architecture are relatively uncertain. However, in other embodiments, the fusion depth estimation of a given pixel can be considered as the depth estimation of the pixel with the highest probability from the fusion depth probability volume.

In the embodiments of FIGS. 12-20, the fusion depth estimation is a high density depth estimation. However, in other cases, medium or low density depth estimates can be obtained by using similar methods or systems, eg, by processing a subset of the pixels of the input frame using the methods herein. Can be done. Additional or alternative, the depth estimation of each of the fusion depth estimates obtained according to the embodiment of any one of FIGS. 12 to 20 and the pixels of the input frame from which the fusion depth estimation was obtained. There can be one-to-one, one-to-many, or many-to-one mappings in between.

The embodiments of FIGS. 12-20 are described with reference to processing frames representing observations of the scene. However, it should be understood that these methods and / or systems may instead be used to process still images rather than frames of video.

The system 2000 of FIG. 19 and / or system 2000 of FIG. 20 is described herein, such as the capture device of FIGS. 1A-1C, the computing system 700 of FIG. 7A, and / or the robot device 760 of FIG. 7B. Either the system or the device can be used. Instructions for performing the method 1900 of FIG. 19 or performing the system 2000 of FIG. 20 may be stored in a non-temporary computer-readable storage medium as described with reference to FIG.

Any function described in connection with any one embodiment may be used alone or in combination with other functions described, and may be any of the embodiments. It may be used in combination with one or more functions of other embodiments, or it may be used in combination of one or more functions of any combination of any other embodiment of the embodiments. Please understand that. Further, equivalents and modifications not described above may be used without departing from the scope of the invention as defined in the appended claims.

Claims (35)

An image processing system that estimates the depth of a scene.
The first depth estimation and the second depth estimation from the neural network architecture are received from the geometric reconstruction engine, and the first depth estimation and the second depth estimation are stochastically fused. Equipped with a fusion engine that outputs the fusion depth estimation of the scene
The fusion engine is to receive the uncertainty measurement of the first depth estimation from the geometric reconstruction engine and the uncertainty measurement of the second depth estimation from the neural network architecture. Configured,
The fusion engine is configured to stochastically fuse the first depth estimate with the second depth estimate using the uncertainty measurement.
The system. The fusion engine receives surface orientation estimation and uncertainty measurement of the surface orientation estimation from the neural network architecture, and uses the uncertainty measurement of the surface orientation estimation and the surface orientation estimation to use the first. The system according to claim 1, wherein the depth estimation of the above and the second depth estimation are stochastically fused. For the surface orientation estimation,
Depth gradient estimation in the first direction and
Depth gradient estimation in the direction orthogonal to the first direction and
Surface normal estimation and
The system according to claim 2, wherein one or more of them are included. The one according to any one of claims 1 to 3, wherein the fusion engine is configured to specify a scale estimation when the first depth estimation and the second depth estimation are stochastically fused. System. The scene is captured in the first frame of the video data and
The second depth estimation for the first frame of the video data is received and
The first depth estimation includes a plurality of first depth estimations for the first frame of the video data, and at least one of the plurality of first depth estimations is the video data. Generated using a second frame of video data that is different from the first frame
The fusion engine is configured to process the second depth estimation and one of the plurality of depth estimates for each iteration and iteratively output the fusion depth estimation of the scene.
The system according to any one of claims 1 to 4. The system according to any one of claims 1 to 5, wherein the first depth estimation, the second depth estimation, and the fusion depth estimation each include a depth map for a plurality of pixels. The system according to any one of claims 1 to 6, wherein the first depth estimation is a medium density depth estimation, and the second depth estimation and the fusion depth estimation each include a high density depth estimation. .. With a monocular camera that captures frames of video data,
A tracking system that identifies the posture of the monocular camera while observing the scene,
With the geometric reconstruction engine,
The geometric reconstruction engine is configured to use the orientation from the tracking system and the frame of the video data to generate a depth estimate for at least a subset of the pixels from the frame of the video data. The geometric reconstruction engine is configured to minimize the photometric error and generate the depth estimation.
The system according to any one of claims 1 to 7. With the above neural network architecture
The neural network architecture comprises one or more neural networks and is configured to receive pixel values of frames of video data and make predictions.
To generate the second depth estimate, a depth estimate for each of the first sets of image portions,
With at least one surface orientation estimation for each of the second sets of image parts,
With one or more uncertainty measurements associated with each depth estimate,
With one or more uncertainty measurements associated with each surface orientation estimation,
Is expected,
The system according to any one of claims 1 to 8. It ’s a way to estimate the depth of the scene.
The geometric reconstruction of the scene is used to generate a first depth estimate of the scene, which outputs an uncertainty measurement of the first depth estimation. The above-mentioned generation, which is configured to be
Using a neural network architecture is to generate a second depth estimate of the scene, the neural network architecture being configured to output an uncertainty measurement of the second depth estimate. The above-mentioned generation and
Using the uncertainty measurement, the first depth estimation and the second depth estimation are stochastically fused to generate the fusion depth estimation of the scene.
The method described above. Before generating the first depth estimate,
Acquiring image data representing two or more views of the scene from the camera,
Including
Generating the first depth estimation is
Obtaining the attitude estimation of the camera and
To generate the first depth estimation by minimizing at least the photometric error which is a function of the attitude estimation and the image data.
including,
The method according to claim 10. Before generating the first depth estimate,
Acquiring image data representing one or more views of the scene from the camera,
Including
Generating the second depth estimation is
Receiving the image data with the neural network architecture
To generate the second depth estimate, the neural network architecture is used to predict the depth estimate for each set of image parts.
Using the neural network architecture to predict at least one surface orientation estimate for each set of image portions.
Using the neural network architecture to predict a set of uncertainty measurements for each depth estimate and each surface orientation estimate,
including,
The method according to claim 10. For the surface orientation estimation,
Depth gradient estimation in the first direction and
Depth gradient estimation in the direction orthogonal to the first direction and
Surface normal estimation and
12. The method of claim 12, comprising one or more of the above. Before generating the first depth estimate,
Acquiring image data from a camera that contains multiple pixels representing two or more views of the scene.
Including
Generating the first depth estimation is
Obtaining the attitude estimation of the camera and
To generate a medium density depth estimate that includes a depth estimate for a portion of the pixel in the image data.
Including
Generating the second depth estimation includes generating a high density depth estimation for the pixel in the image data.
Probabilistic fusion of the first depth estimation and the second depth estimation includes outputting a high density depth estimation for the pixel in the image data.
The method according to claim 10. The method is iteratively repeated and with respect to subsequent iterations.
The method comprises determining whether to generate the second depth estimation.
Probabilistic fusion of the first depth estimation and the second depth estimation is a set of values prior to the second depth estimation, depending on the determination that the second depth estimation is not generated. Including using
The method according to any one of claims 10 to 14. The method is applied to a frame of video data, and the probabilistic fusion of the first depth estimation and the second depth estimation is for a given frame of video data.
It comprises optimizing a cost function that includes a first cost term associated with the first depth estimation and a second cost term associated with the second depth estimation.
The first cost term includes a function of the fusion depth estimate, the first depth estimate, and the uncertainty value of the first depth estimate.
The second cost term includes a function of the fusion depth estimate, the second depth estimate, and the uncertainty value of the second depth estimate.
The cost function is optimized to identify the fusion depth estimate.
The method according to any one of claims 10 to 15. 16. The method of claim 16, wherein optimizing the cost function comprises identifying the scale factor of the fusion depth estimation, wherein the scale factor indicates the scale of the fusion depth estimation for the scene. Including using the neural network architecture to generate at least one surface orientation estimate for the scene.
The neural network architecture is configured to output an uncertainty measurement for each of the at least one surface orientation estimates.
The cost function includes a third cost term associated with the at least one surface orientation estimation.
The third cost term comprises a function of a fusion depth estimate, a surface orientation estimate, and an uncertainty value for each of the at least one surface orientation estimation.
The method of claim 16 or 17. The geometric reconstruction of the scene is configured to generate a first depth probability volume of the scene, where the first depth probability volume is.
A first plurality of depth estimates, including the first depth estimate,
The first plurality of uncertainty measurements associated with each of the first plurality of depth estimates, respectively.
Including
The uncertainty measurement associated with a given depth estimate of the first plurality of depth estimates is that a given area of the scene is the given given of the first plurality of depth estimates. Represents the probability of being in the depth represented by depth estimation
The neural network architecture is configured to output a second depth probability volume of the scene, where the second depth probability volume is.
A second plurality of depth estimates, including the second depth estimate,
A second plurality of uncertainty measurements associated with each of the second plurality of depth estimates, respectively.
Including
The uncertainty measurement associated with a given depth estimate of the second plurality of depth estimates is that a given area of the scene is the given given of the second plurality of depth estimates. Represents the probability of being in the depth represented by depth estimation,
The method according to claim 10. Generating the second depth estimation of the scene comprises using the neural network architecture to process image data representing the image of the scene to generate the second depth probability volume. ,
The second plurality of depth estimates include a plurality of depth estimation sets, each associated with a different portion of the image of the scene.
19. The method of claim 19. 19. The method of claim 19 or 20, wherein the second plurality of depth estimates include depth estimates having predefined values. 21. The method of claim 21, wherein there is a non-uniform spacing between the predefined values. 22. The method of claim 21 or 22, wherein the predefined values include a plurality of logarithmic depth values within a predefined depth range. Generating the first depth probability volume of the scene is
The first frame of the video data representing the first observation of the scene and the second frame of the video data representing the second observation of the scene are processed to form a plurality of parts of the first frame. To generate a set of metering errors for each, said generating and each metering error is associated with each different depth estimation of the first plurality of depth estimates.
Scaling the metering error and converting the metering error into their respective probability values,
The method according to any one of claims 19 to 23. Probabilistic fusion of the first depth estimation and the second depth estimation using the uncertainty measurement is to generate the first plurality of uncertainty measurements and the second plurality of uncertainties. The method of any one of claims 19-24, comprising generating a fusion probability volume in combination with certainty measurements. 25. The method of claim 25, wherein generating the fusion depth estimation of the scene comprises obtaining the fusion depth estimation of the scene from the fusion probability volume. To obtain the depth probability function using the fusion probability volume,
Using the depth probability function to obtain the fusion depth estimation,
25 or 26, the method of claim 25 or 26. Obtaining the fusion depth estimation involves optimizing the cost function, which is a cost function.
The first cost term obtained using the fusion probability volume and
A second cost term, including local geometric constraints on the depth value,
25. The method according to any one of claims 25 to 27. Receiving surface orientation estimates and occlusion boundary estimates from further neural network architectures,
Using the surface orientation estimation and the occlusion boundary estimation to generate the second cost term,
28. The method of claim 28. The fusion depth probability volume is a first fusion depth probability volume associated with a first frame of video data representing the first observation of the scene.
The method is
Converting the first fusion depth probability volume into the first occupancy probability volume,
Based on the posture data representing the posture of the camera while observing the scene, the first occupancy probability volume is warped and associated with the second frame of the video data representing the second observation of the scene. To obtain the second occupancy probability volume obtained,
Converting the second occupied probability volume into a second fusion depth probability volume associated with the second frame,
25. The method according to any one of claims 25 to 29. An image processing system that estimates the depth of a scene.
The first depth probability volume is received from the geometric reconstruction engine and the second depth probability volume is received from the neural network architecture, and the first depth probability volume and the second depth probability volume are fused. , A fusion engine that outputs the fusion depth probability volume of the scene,
With a depth estimation engine that estimates the depth of the scene using the fusion depth probability volume,
The system comprising. It ’s a way to estimate the depth of the scene.
Using the geometric reconstruction of the scene to generate the first depth probability volume of the scene,
Using a neural network architecture to generate a second depth probability volume for the scene,
By fusing the first depth probability volume and the second depth probability volume to generate the fusion depth probability volume of the scene,
Using the fusion depth probability volume to generate a fusion depth estimate for the scene,
The method described above. With a monocular capture device that provides video frames,
The position identification and mapping simultaneous execution system that provides the attitude data of the monocular capture device,
The system according to claim 1 or 31 and
A medium-density multi-view stereo component that receives the attitude data and video frames and implements the geometric reconstruction engine.
An electronic circuit that implements the neural network architecture and
A computing system. It ’s a robot device,
The computing system according to claim 33 and
With the one or more actuators that allow the robot device to interact with the surrounding three-dimensional environment, wherein at least a portion of the surrounding three-dimensional environment is shown in the scene. ,
An interaction engine comprising at least one processor controlling the one or more actuators.
Including
The interaction engine uses the fusion depth estimation to interact with the surrounding three-dimensional environment.
The robot device. A non-temporary computer-readable storage medium containing computer-executable instructions that, when executed by a processor, causes the computing device to perform the method of any one of claims 10-30. The non-temporary computer-readable storage medium.

Images