KR20220024517A - 3D scene reconstruction from 2D images - Google Patents

Abstract

This specification relates to reconstructing three-dimensional (3D) scenes from two-dimensional (2D) images using a neural network. According to a first aspect of the present specification, a method is described for generating a three-dimensional reconstruction of a scene having a plurality of objects from a single two-dimensional image, the method comprising: receiving a single two-dimensional image; identifying all objects in the image to be reconstructed and identifying the types of the objects; estimating a three-dimensional representation of each identified object; estimating a three-dimensional plane that physically supports all three-dimensional objects; and positioning all three-dimensional objects in space with respect to the support plane.

Description

3D scene reconstruction from 2D images

This specification relates to reconstructing three-dimensional (3D) scenes from two-dimensional (2D) images using a neural network.

Aside from the easier problem of faces, there is little research to integrate a 3D human body (or other objects) into augmented reality applications in real time. Bodies are generally more difficult than faces due to their articulated deformations, multiple occlusions and self occlusions, complex interactions with other objects and people, and wider appearance variability due to clothing.

Human pose estimation algorithms typically aim to locate specific sparse points in images, such as skeletal joints and facial landmarks, or more recently, dense surface level coordinates. Despite their increasing accuracy, such representations fall short of serving downstream applications such as augmented reality, motion capture, gaming, or graphics. They require access to a three-dimensional basal body surface and currently rely on multiple camera setups or depth sensors.

The study of morphable models demonstrated that accurate monocular surface reconstruction can be performed by using the low-dimensional parametric of the facial surface and appearance, and casting the reconstruction operation as an optimization problem. Extending this to more complex articular structures of the human body, monocular anatomical reconstruction has been extensively studied in the previous decade, along with part-based representations, sampling-based reasoning, spatiotemporal reasoning, and bottom-up/bottom-up methods. Monocular 3D reconstruction has witnessed a resurgence in the context of deep learning for general categories and especially for humans. Previous studies rely on efficient parameterization of the human body in terms of skinned linear models and in particular a skinned dyne linear (SMPL) model. Taking advantage of the fact that the SMPL model provides a low-dimensional, differentiable representation of the human body, these studies have investigated the reprojection error between SMPL-based 3D key points and 2D joint annotations, human segmentation masks and 3D volume projections. Systems were trained to regress model parameters by minimizing or even refining to the level of body parts.

In parallel, 3D human joint estimation has been developed with 3D convolutional neural network (CNN) based architectures that directly localize 3D joints in volume output space through hybrids of classification and regression, from classical motion-to-structure approaches. By going over it, there was a dramatic increase in accuracy.

Finally, a recent study on dense pose estimation suggests that it is possible to estimate dense correspondences between RGB images and SMPL models by training a general bottom-up detection system to associate image pixels with surface-level UV coordinates. showed that It should be noted that although the dense pose establishes a direct link between images and surfaces, it does not cover the underlying 3D geometry, but rather gives a strong hint about it. Other recent studies, such as SMPL, rely on parametric human models of shape, which allow describing 3D human body surfaces in terms of low-dimensional parametric vectors. Most of these studies train CNNs to regress deformable model parameters and update them in an iterative and time-required optimization process.

According to a first aspect of the present specification, a method is described for generating a three-dimensional reconstruction of a scene having a plurality of objects from a single two-dimensional image, the method comprising: receiving a single two-dimensional image; identifying all objects in the image to be reconstructed and identifying the types of the objects; estimating a three-dimensional representation of each identified object; estimating a three-dimensional plane that physically supports all three-dimensional objects; and positioning all three-dimensional objects in space with respect to the support plane.

The step of estimating the three-dimensional representation may be performed in a deep machine learning model, the deep machine learning model comprising an output layer and one or more hidden layers each applying a nonlinear transformation to the received input to generate an output. The deep machine learning model predicts three-dimensional landmark positions of multiple objects by connecting feature data from one or more intermediate layers of a neural network, and the predicted three-dimensional positions are simultaneously applied to the predicted types of objects depicted in each region. It is estimated.

The step of estimating the plane supporting the plurality of objects may be performed for a single frame using the estimated three-dimensional positions of all visible objects to reconstruct a two-dimensional plane passing through all the visible objects. The step of estimating a plane supporting multiple objects may be performed for a series of frames using relative camera pose estimation and plane localization using correspondences between points of successive frames.

The receiving step may further comprise receiving a plurality of images, wherein the steps of estimating three-dimensional representations of the plurality of objects and locating them in place are for each received image, eg, in real time is made of Processing may occur over multiple successive frames by combining the hidden layer responses in successive frames, eg, by averaging them.

Digital graphic objects can be synthetically added to the three-dimensional scene reconstruction with a given relationship to the estimated three-dimensional object positions, and then projected back into the two-dimensional image.

According to a further aspect of the present specification, a computational unit is described for generating a three-dimensional reconstruction of a scene having a plurality of objects from a single two-dimensional image, the computational unit comprising: a memory; and at least one processor, wherein the at least one processor is configured to perform the method according to the first aspect.

According to a further aspect of the present specification, to at least one processor of the computation unit, for causing a computation unit for generating a three-dimensional reconstruction of a scene having a plurality of objects from a single two-dimensional image to perform the method according to the first aspect A computer readable medium is described that stores a set of instructions executable by the

According to a further aspect of the present specification, a computer program product is described comprising instructions that, when executed by a computer, cause the computer to perform a method according to the first aspect.

According to a further aspect of the present specification, a method is described for training a deep machine learning model to generate a three-dimensional reconstruction of a scene having multiple objects from a single two-dimensional image, the method comprising: receiving a single two-dimensional image; step; obtaining a training signal for three-dimensional reconstruction through adapting a three-dimensional model of an object to a two-dimensional image; and using the resulting three-dimensional model fitting results as a supervisory signal for training the deep machine learning model.

Fitting the three-dimensional model may include projecting the three-dimensional representation onto a two-dimensional image plane to generate a projected representation; comparing the respective positions of the projected representation with the object of the single two-dimensional image; determining an error value based on the comparison; and adjusting the parameters of the fused three-dimensional representation based on the error value, wherein the comparing, measuring and adjusting steps are iteratively repeated until a threshold condition is satisfied. The threshold condition may be that the measured error value falls below a predetermined threshold value or that a threshold number of iterations is exceeded. The projecting step may be performed by taking into account the effects of perspective projection, and, if multiple views are available, using multiple views of the same object.

According to a further aspect of the present specification, a computational unit is described for generating a three-dimensional reconstruction of a scene having a plurality of objects from a single two-dimensional image, the computational unit comprising: a memory; and at least one processor, wherein the at least one processor is configured to perform according to a further aspect.

According to a further aspect of the present specification, by at least one processor of the calculating unit to cause a calculating unit for generating a three-dimensional reconstruction of a scene having a plurality of objects from a single two-dimensional image to perform the method according to the further aspect A computer readable medium storing a set of executable instructions is described.

According to a further aspect of the present specification, a computer program product is described comprising instructions that, when executed by a computer, cause the computer to perform a method according to the further aspect.

According to a further aspect of the present specification, a system is described for providing a three-dimensional reconstruction of a scene having a plurality of objects from a single two-dimensional image, the system comprising: a second for performing any of the methods of the first aspect 1 counting unit; and a second computation unit configured to perform the method according to a further aspect, wherein the second unit is configured to train the model with results of the first computation unit.

Embodiments and examples will be described with reference to the accompanying drawings, in which:
1 shows a schematic overview of an exemplary method for generating a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image;
2 shows a flow diagram of a method for generating a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image;
3 shows a schematic overview of an exemplary method for training a neural network to reconstruct a three-dimensional scene with multiple objects from a single two-dimensional image;
4 shows a flow diagram of a method for training a neural network for use in reconstructing a three-dimensional scene with multiple objects from a single two-dimensional image;
5 shows a further example of a method for training a neural network for use in three-dimensional scene reconstruction and object re-identification;
6 shows a schematic example of a system/apparatus for performing any of the methods described herein.

This specification describes methods and systems for reconstruction of a 3D scene containing multiple objects (eg, humans) from a single 2D image (eg, RGB image). Exemplary methods and systems described herein recover accurate 3D reconstructions of multiple objects at more than 30 frames per second on a mobile device (eg, a mobile phone), while also providing information regarding 3D camera position and world coordinates. can be restored The methods and systems described herein may be applied, for example, to real-time augmented reality applications involving the entire anatomy, where users attach to objects positioned on them, eg, a graphic asset attached to their hands. While allowing objects to control them, it also allows objects to interact with humans, eg, allowing balls to bounce off humans when they come into contact with their bodies.

This specification also describes CNN design choices that allow estimation of the 3D shape of multiple (potentially hundreds) objects (eg, humans) at more than 30 frames per second on a mobile device. Using the methods described herein, results are generated within a constant time of about 30 milliseconds per frame on a SnapDragon 855 Neural Processing Unit (NPU), regardless of the number of objects (eg, people) in the scene. can be obtained.

Various aspects described herein may contribute to these effects, alone or in combination.

In some embodiments, mesh distillation is used to construct a supervisory signal for monocular image reconstruction. A detailed time-required model fitting procedure is performed offline to recover the 3D interpretation of all images in the training set. The fitting results are then used to train one or more neural networks (eg, convolutional neural networks) to efficiently process incoming test images into a single feedforward pass through the network. This, without compromising speed at test time, is imposed by complementary ground truth signals during model fitting (e.g., sparse and dense reprojection error based on region alignment, key points and tight poses, respectively). Allows the integration of complex constraints.

In some embodiments, an efficient encoder-only neural network architecture is used for monocular three-dimensional human pose estimation. Early studies relied on parametric design models of the human surface, and decoder-based networks for thorough and accurate localization of human joints in 2D. Instead, the repurposed (standard) deep neural network used for image classification can be used, so that the last layer is configured to output the 3D coordinates for each vertex of the object (eg, human) mesh. This may substantially accelerate test time inference, and may also simplify deploying networks on mobile devices.

In some embodiments, the neural network will have a single stage fully convolutional architecture that densely processes the input 2D image and emits 3D pose estimates for multiple image positions in a single pass through the (standard) convolutional network. can This may result in an inference that is time independent of the number of objects (eg, people) in the scene, while also dramatically simplifying the smoothing of 3D reconstructions over time, which performs post smoothing Instead, it is possible to average the network layers over time.

In some embodiments, a person-based self-calibration method for estimating a floor position in a 3D scene by fitting a plane to the estimated 3D positions of objects/sub-objects (eg, a human foot) in a 2D image is used This allows recovery of the world geometry in a way that negates the scaling effects of the perspective projection, while also recovering the camera position relative to the plane. In turn, reconstructing the world geometry allows augmentation of the reconstructed scene, for example by inserting objects that can fall to the floor and bounce back while obeying the laws of physics. The method is, in both cases when the camera is stationary - in which case simultaneous localization and mapping (SLAM) methods fail - and also in both cases when the camera is moving, e.g., SLAM the estimated floor position. It can work by aligning with the planes restored by

In some embodiments, a distributed part based variant of this approach collects information about mesh parts from multiple image positions, as indicated by estimated object part positions. The body mesh is obtained by synthesizing together the partial-level meshes output by the neural network, thereby allowing better handling of occlusions, large joints, while maintaining exactly the same memory and computational cost, apart from the memory retrieval operation. .

In some embodiments, the neural network may additionally accommodate the task of object/person re-identification (REID). A teacher-student network distillation approach can be used to train a network to perform such tasks. REID embeddings are extracted from object (eg, human) crops from 2D images using a pre-trained REID network, and these are used as supervisory signals to train the REID branch of the neural network. This branch carries REID embeddings that mimic those of the training network, but can be fully convolutional, meaning that its execution speed is independent of the number of objects in the image.

1 shows a schematic overview of an exemplary method 100 for generating a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image. The method may be performed by one or more computing devices operating at one or more locations. For example, the method may be performed by a mobile computing device, such as a mobile phone.

One or more 2D images 102 each comprising a plurality of objects 104 of a given type (eg, a person) are input to the neural network 106 . In some embodiments, only a single input image 102 is used. The neural network 106 identifies objects 104 in the image and outputs data 108 that includes an estimated 3D representation 110 of each (potentially) identified object 104 in the input 2D image 102 . Process the input data to create it. The output data 108 may, in some embodiments, be at the estimated coordinates 112 of the bounding boxes of potential objects 104 in the input 2D image 102 and/or at locations within the input 2D image 102 . may further include probabilities of the existence of objects 104 of The output data 108 is further processed (ie, post-processed) to generate a three-dimensional reconstruction 116 of the scene within the input 2D image 102 . In some embodiments, the output data 108 may further include an embedding vector (not shown) associated with each object 104 in the image. The embedding vector provides the REID embedding of the object, so that a given object 104 can be identified and/or tracked across multiple input images 102 .

The 2D input image 102 , I contains a set of pixel values corresponding to a two-dimensional array. For example, in a color image, I ∈ R ^HxWx3 , where H is the height of the image in pixels, W is the width of the image in pixels, and the image has three color channels (e.g., RGB or CIELAB). have In some embodiments, the 2D image may be black and white/grayscale.

In some embodiments, method 100 may use multiple (eg, series) input 2D images 102 (eg, of multiple people in motion). The 3D representations 110 may take these changes into account. Multiple input 2D images 102 may be received, for example, in substantially real time (eg, at 30 fps) from a camera on a mobile device. Neural network 106 may process each of these input 2D images 102 individually, and the hidden layer responses for each input 2D image 102 are combined during post-processing when generating the 3D scene.

Neural network 106 takes 2D image 102 as input and processes it through a plurality of neural network layers to generate output data 108 . Neural network 106 is a deep machine learning model comprising an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to produce an output. The neural network may predict three-dimensional landmark positions of multiple objects by connecting feature data from one or more intermediate layers of the neural network. The predicted three-dimensional positions can be estimated simultaneously for the predicted type of object depicted in each region.

Each layer of neural network 106 includes a plurality of nodes (also referred to herein as “neurons”), each of which is associated with one or more neural network parameters (eg, weights and/or biases). . Each node takes as input the output from one or more nodes in the previous layer (or input to the first layer of the neural network) and applies a transform on that input based on parameters associated with that node. The transform may be a non-linear transform. Alternatively, some of the nodes may apply a linear transform.

Neural network 106 may include one or more convolutional layers, each configured to apply one or more convolutional filters to the output of previous layers in the network. In some embodiments, neural network 106 is fully convolutional. A neural network may include one or more fully connected layers, where each node in the fully connected layer receives input from all nodes in the previous layer. Neural network 106 may include one or more skip connections. The neural network 106 may include a residual neural network, for example, ResNet-50. The remaining network can be used as a backbone network.

The neural network 106 is a single stage system that jointly detects objects (eg, humans) and estimates the 3D shape of the objects by performing a single forward pass through the neural network 106 (which may be a fully convolutional neural network). can be Instead of cropping image patches around object detection regions and then processing them again, the task of extracting region-specific features is delegated to neurons in successive layers of neural network 106 with increasingly larger receptive fields. . A Lesnet-50 backbone is available which offers a good compromise between speed and accuracy. A-trous convolutions can be used (eg, L. Chen et al., "Deeplab: Semantic image segmentation with deep convolutional nets, the contents of which are incorporated herein by reference." , atrous convolution, and fully connected crfs", see PAMI, 2017.3), allowing to increase the spatial density in which object (eg, human) hypotheses are evaluated, and to reduce the number of missing detections.

The final layer of the neural network 106 (eg, a fully convolutional neural network) is the output data 108 (eg, It is responsible for predicting multiple outputs).

In some embodiments, one or more of the layers of the neural network 106 may be a 1x1 convolutional layer. For example, the final layer of a neural network may contain one or more 1x1 convolutions. Examples of such layers are "Fully Convolutional Networks for Semantic Segmentation" (E. Shelhamer et al., IEEE Trans. Pattern Anal. Mach. Intelll. 39(4): 640-651 (2017)) and "Overfeat" : Integrated recognition, localization and detection using convolutional networks" (P. Sermanet et al., 2nd International Conference on Learning Representations, International Conference on Learning Representations, 2014), the contents of which are hereby incorporated by reference. do.

The output data 108 includes a representation of the mesh for each detected object 110 . It may be in the form of a K = N x 3D vector that captures the shape of the object, where N is the number of nodes in the mesh. N may be 563, but other numbers of mesh nodes are possible. The output data 108 may further include a probability of the existence of an object (eg, a person) 114 . The output data 108 may further include corners of the bounding boxes 112 of objects in the input image 102 .

In some embodiments, the predictions of the neural network 106 may be object type specific. For example, neural network 106 may predict 3D positions of face parts, fingers, and limbs for humans, whereas different neural network 106 may predict wheel, window, door and light positions for automobiles. can be dedicated It should be understood that object landmarks that are expected to be present due to the identified object type may not be present in the image. For example, if the object type is a human, parts of the anatomy may not be visible due to postures or other items that obstruct the view.

The use of neural network 106 as described above can result in several advantages. First, the inference is efficient and bypasses the need for deconvolution-based decoders for high-accuracy pose estimation. Instead, this architecture is "encoder-only", reducing the time required to process an M x M (eg, 233 x 233) patch to a few milliseconds. Second, the resulting networks are easily portable to mobile devices because, in contrast to the chair body models used in previous studies, they rely exclusively on generic convolutional network layers. Third, it simplifies extending the resulting model so that the entire image can be processed in a fully convolutional manner instead of processing individual patches.

Beyond simplicity, the resulting architecture also simplifies performing temporal smoothing of the reconstructed 3D shapes. Instead of the complex parameter tracking based methods used previously, one can take a moving average of the second-to-last layer activations of the network 106 . This substantially stabilizes the 3D shapes recovered by the network, while requiring substantially no effort to integrate as a processing step into the mobile device.

In some embodiments, neural network 106 configures an object (eg, a human sternum) based on a high-dimensional feature vector (F) computed at a location (i) aligned with a predetermined point within the object (eg, the human sternum). The entire object mesh (V) of a person) can be output. The feature vector may be output by a layer in the neural network, and may correspond to, for example, activations of a hidden layer in the network. The mapping can be expressed as:

where M denotes the mapping of the feature vector (F) to the output mesh (V). M may, in some embodiments, be implemented as a linear layer in a neural network.

The simplicity of this 'centralized' (i.e., aligned with a predetermined single point within the object) method is that the detail of the object's (e.g. human) mesh through F[i] can serve as a computational bottleneck. It is offset by the difficulty of coefficients for both factors and variations. In such a way, the location of a given node in the output mesh 110 may be based on activations of neurons in the neural network associated with locations remote from the mesh node. This may introduce inaccuracies in the resulting output mesh 110 .

In some embodiments, the part-based method may alternatively be used to reconstruct the 3D representation 110 of the object. Partial-based approaches use a distributed approach to mesh regression, which can be substantially more accurate than centralized methods. The part-based approach is useful in situations where objects 104 may be partitioned into parts. For example, the human body may naturally be divided into parts, eg, head, torso, arms, hands, and the like.

Taking the human body as an example, neurons associated with locations closer to object parts (eg, hand, foot, etc.) distal from a predetermined point (eg, sternum) are each more distant than neurons associated with locations more distant. can deliver more reliable estimates for the mesh portions of . These closer neurons act like "distributed part experts" providing reliable information about body parts in their vicinity. Information from these distributed segment experts may be combined using a coordinator node located at a predetermined point (eg, the sternum) to determine the output mesh 110 . Information about a particular mesh node associated with a given part, coming from neurons associated with locations further away from the part (ie not associated with the part), may be suppressed/discarded during the determination of the output mesh 110 . This can be achieved using a sub-level attention matrix (A).

The partial positions providing information to the coordinator are determined by the human skeletal calculation stage and may associate the sternum position with the corresponding partial positions. The partial positions may be based on joint positions of the anatomy, for example.

In the part-based approach, the neural network 106 outputs a respective part mesh V[p] for each part p of the object 104 in a predetermined list of object parts. Each partial mesh V[p] is provided in its own coordinate system whose origin is offset with respect to a reference point (also referred to herein as a “coordinator node”) c (eg, the human sternum). . That is, {p ₁ , ... p _M } represents positions of each of the M parts associated with the coordinator (eg, sternum) position (c). The partial positions may be based on key points of objects, for example, joint positions within the human body.

In order to reconstruct the whole mesh V from the partial meshes V[p], the coordinator selects the nodes related to the part p from V[p] using the partial level attention matrix A. In some embodiments, A is a binary matrix indicating which part should be used for each vertex in the final mesh (V). For such an attention matrix, the final mesh V can be reconstructed for each mesh vertex v in the final mesh using:

where V[c,v] denotes the position of the mesh node v with respect to the predetermined point c, and p _v is the node v (i.e., the node with A[p,v]=1 in the binary example). ) denotes the position of the part that is an "active expert" for , and V[p,v] denotes the position of the mesh node v in the mesh associated with the part p. (cp _v ) is an offset vector describing the relative position between the portion and the reference position c (eg, the sternum).

In some embodiments, the sub-level attention matrix A may be a learned matrix determined during training. In such an example, the location of a given mesh node v may be based on information from one or more portions. Thus, using equation (2) rather than equation (3) may result in a weighted sum for more than one part.

In some embodiments, the output data 108 is, in some embodiments, the estimated coordinates 112 of the bounding boxes of potential objects 104 in the input 2D image 102 and/or the input 2D image ( and probabilities of the existence of objects 104 at locations within 102 .

The estimated 3D representation 110 of each (potentially) identified object 104 in the input 2D image 102 is, in some embodiments, in the form of an N-vertex mesh (V). For example, the estimated 3D representation 110 of each identified object 104 may be in the form of an (Nx3)-dimensional vector providing the 3D coordinates of the N vertices in the mesh. In some embodiments, N=536, but other numbers of mesh nodes are possible.

The coordinates of the bounding boxes are, for each potential object, the x and y positions of the vertices of the bounding box in the input 2D image 102 , e.g., {(x ₁ , y ₁ ), for each potential object (x ₂ , y ₂ ), (x ₃ , y ₃ ), (x ₄ , y ₄ )}. Each bounding box may be associated with a corresponding probability that the bounding box will contain an object of a given type (eg, a person).

A 3D reconstruction 116 of the scene in the input 2D image 102 is created by taking into account the perspective projection and negating its effects when estimating the coordinates in the world. The 3D reconstruction 116 creates a 3D plane 118 (also referred to herein as a “support plane”) that physically supports the objects 104 (eg, floor, ground) identified in the image 102 . It is created by estimating and placing all 3D objects 110 in space with respect to the support plane.

Estimating the 3D plane 118 that physically supports the objects 104 identified in the image 102 may be based on the assumption that the objects (eg, humans) are supported by a single physical plane. In some embodiments, this assumption is relaxed by using mixed models and using predictive maximization to assign different humans to different planes.

Estimating the 3D plane 118 may also be based on assuming that the heights of objects (eg, humans) are approximately equal. This latter assumption may be relaxed if a series of input images 102 are available in which individual objects can be tracked over time and the effects of perspective projection over time are monitored for each object.

To place the 3D objects 110 in space with respect to the support plane, the scaling of each mesh is estimated to bring it into world coordinates. The scale is inversely proportional to the distance of the object (eg, a person) from the camera capturing the scene in the input image 102 . As such, scaling can be used to position the mesh in world coordinates along the line connecting the mesh vertices to the camera center.

As an example, 3D objects 110 may be provided by neural network 106 in the form of meshes estimated under orthographic projection. Assuming the i-th mesh in the scene with vertices (m _i ={v _i,1 ,..., v _i,K }), where v _i,k = (x _i,k , y _i,k , z _i,k ) ∈ R ³ are the mesh coordinates estimated under scaled orthographic projection and scale s _i , and mesh vertices are located in 3D world coordinates by negating the effects of perspective projection. For example, the world coordinates of the k-th point in the i-th mesh (V _i,k =(X _i,k , Y _,i,k , Z _,i,k )) are the world coordinates by the inverse of the scale factor. From the corresponding mesh coordinates estimated under the scaled orthographic projection by "pushing back" the mesh depth in can be estimated. Symbolically, this can be expressed as:

Here, the camera calibration matrix has a center (c _x = W/2 and c _y = H/2, where W and H are image dimensions, i.e., image width and height) and focal length f. These can be set manually or by camera calibration.

For each mesh corresponding to the identified object, the lowest (ie, lowest Y value) is determined and used to estimate the point of contact between the object and the support plane (eg, floor). Once at least four of those points have been determined (collectively denoted M), the determined points can be used to estimate the support plane in world coordinates. In some embodiments, least squares method may be used to estimate the plane in world coordinates, for example:

Here, V ₁ =(a, b, c) is a vector perpendicular to the support plane. This vector may, in some embodiments, be normalized. The world coordinate axes R=[v ₁ ^T , v ₂ ^T , v ₃ ^T ] ^T have two complementary directions V ₂ and V ₃ , both of which are relative to V ₁ and to each other. is orthogonal to - is defined by finding In some embodiments, V ₂ is selected in the direction of Z and V ₃ =V ₁ xV ₂ . Vectors V ₂ and V ₃ may, in some embodiments, be normalized, ie, the set {V ₁ , V ₂ , V ₃ } forms an orthogonal normal basis.

The world coordinate center T can also be assigned as a 3D point lying on a plane. For example, this could be set to a point that is 3 meters from the camera, projected as y=H/2.

In some embodiments, the above content may be used to define a transformation between the world coordinate system and pixel locations within the 2D image 102 . For example, the world-to-camera transformation and transformation and camera calibration matrices in a single 3x4 perspective projection matrix P can be given as:

Here, the direction (V) vectors appear as rows rather than columns, since an inverse rotation matrix is being used, R ⁻¹ = R ^T .

Using homogeneous coordinates, world coordinates (C) can be transformed into pixel coordinates (c) using:

Coordinate transformation may, in some embodiments, be used to introduce objects into a scene/image that obey the laws of physics and/or interact with other objects in a meaningful/realistic way. This can lead to interactive applications that require real-world coordinates, eg, augmented reality games, eg, games in which a player attempts to hit another player with a sword or laser beam. In some embodiments, 3D reconstructed meshes may be projected back to the input image 102 , showing the accuracy of the 3D human pose estimates.

In embodiments where multiple input 2D images 102 are used, estimating the plane supporting the multiple uses relative camera pose estimation and plane localization using correspondences between points in successive frames. Thus, it can be performed on a series of frames.

Processing (eg, post-processing) may occur over multiple consecutive frames by combining the hidden layer responses in consecutive frames, eg, by averaging them.

According to some of the example embodiments, the composite objects may be controlled by, or interact with, objects (eg, a person) in the input image. For example, the added composite object could be a sword, or laser beam, that the person is controlling with his arm in a computer game, or an object that moves towards the person and bounces back when it comes into contact with an estimated three-dimensional person location.

This method may be integrated into a graphics engine, such as Unity. The user can interact in real time with his own images/videos and/or view meshes superimposed on himself.

For example, in video, combining multiple measurements over time taken from successive input images 102 can be accomplished while combining methods with camera tracking, as simultaneous localization and mapping (SLAM) allows. , provides more improved estimates.

In the case of a combination of deformable objects such as humans and robust 3D scenes, both are reconstructed into metric coordinates.

2 depicts a flow diagram of an exemplary method for generating a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image. The methods may be performed by one or more computers operating at one or more locations. The method may correspond to the method described with respect to FIG. 1 .

In operation (2.1), a 2D image is received. A 2D image comprises an array of pixel values, for example an RGB image. A single 2D input image may be received. In some embodiments, a series of single images may be received.

In operation 2.2, objects in the image to be reconstructed are identified. Object types may also be identified. Operation 2.2 may be performed by a neural network, eg, one or more layers of the neural network described above with respect to FIG. 1 .

In operation (2.3), a 3D representation of each identified object is estimated. The 3D representation may be in the form of a mesh, for example, a K=Nx3D vector of N vertex positions in 3D. Operation 7.3 may be performed by a neural network, eg, one or more layers of the neural network described above with respect to FIG. 1 .

The operation of estimating the 3D representation may be performed in a deep machine learning model, e.g., a neural network, comprising an output layer and one or more hidden layers each applying a non-linear transformation to the received input to produce an output. . A deep machine learning model can predict 3D landmark positions of multiple objects by concatenating feature data from one or more intermediate layers of a neural network. Predicted 3D positions are simultaneously estimated for the predicted type of object depicted in each region.

When a series of input images are received, a step of estimating three-dimensional representations of multiple objects and placing them on a plane is performed for each received image. If the images are received in substantially real-time (eg, 30 fps), these operations may be performed in substantially real-time.

In operation (2.4), a 3D plane that physically supports all three-dimensional objects is assumed. Operation 2.4 may be performed at a post-processing stage, ie after processing of the input image by the neural network. Methods for estimating the support frame are described in more detail above with reference to FIG. 1 .

Estimating a plane supporting multiple objects may be performed for a single frame using the estimated three-dimensional positions of all visible objects to reconstruct a two-dimensional plane passing through all visible objects. For example, the support plane may be estimated based on estimated positions of contact points between the objects and the plane, eg, positions of the feet of people identified in the input image. Estimating the plane supporting multiple objects may be performed for a series of frames using relative camera pose estimation and plane localization using correspondences between points of successive frames.

In operation 2.5, the 3D objects are positioned in space with respect to the support plane. Operation 2.5 may be performed in a post-processing step, ie after processing of the input image by the neural network. Methods of positioning 3D objects relative to the support frame are described in more detail above with reference to FIG. 1 .

3 shows a schematic overview of an exemplary method 300 for training a neural network to reconstruct a three-dimensional scene with multiple objects from a single two-dimensional image. The methods may be performed by one or more computers operating at one or more locations.

The 2D training image 302 is obtained from a training dataset including a plurality of 2D images. The 2D images include one or more objects 304 of a given type (eg, one or more humans). The 3D deformable model is fitted (eg, iteratively fitted) to objects in the input image 302 to extract one or more supervisory signals 306 for monocular 3D reconstruction. A training image 302 is input to a neural network 308 and current parameters of the neural network 308 to generate output data 310 comprising candidate 3D representations of each (expected) object in the input training image 302 . It is processed through a series of neural network layers using The output data is compared with the supervisory signals 306 of the input image 302 to determine parameter updates 312 for the neural network 308 .

The method may be iteratively performed until a threshold condition is satisfied.

The input images 302 and the neural network 308 may be of the same form as described with respect to FIG. 1 .

During the deformable model fitting/optimization stage, all of the available 2D and 3D ground truth data is used to provide energy terms that are minimized to obtain an estimated 3D shape (ie, supervisory signal 306 ). As an example, "Holopose: Holistic 3d human reconstruction in-the-wild" (R. Alp Guler and I. Koki), using CNN-style iterative fitting of deformable models to 2D images. The method described in I. Kokkinos, The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. 2, 3, the contents of which is incorporated herein by reference) may be used.

Fitting the three-dimensional model may be performed by projecting the three-dimensional representation onto a two-dimensional image plane to generate a projected representation. The projecting step may be performed by taking into account the effects of perspective projection, and, if multiple views are available, using multiple views of the same object. It can then compare the respective positions of the projected representation to the object in a single two-dimensional image, and based on the comparison, for example, the difference of the object landmark positions in the training image 302 and the reprojected image. By determining the error value can be determined. Then, the parameters of the fused three-dimensional representation are adjusted based on the error value. The projection, comparison, measurement and adjustment may be iteratively repeated until the measured error value is below a predetermined threshold value or a threshold number of iterations is exceeded.

Once the deformable model fitting converges, the estimated 3D surface is used as a supervisory signal 306 or target for training the network 308 . For example, a “low-poly” representation of objects 304 (eg, anatomy) is a representation of a significant object (eg, human landmarks (eg, face parts, limb joints, etc.) ))), representing the mesh in terms of N (eg, N=536) vertices.

From the fittings of the deformable model to the image, the estimated 3D positions of its vertices around each object present in the image are measured. The neural network 308 is then trained to regress this K=Nx3 dimensional vector when present with the object 304 in its receptive field. In the presence of multiple objects, different responses are expected at different image locations. The positions of the bounding boxes of objects in the input image 302 and/or the presence/absence of objects at positions in the input image may additionally be used as supervisory signals 306 .

The task of extracting region-specific features is delegated to neurons in successive layers with increasingly larger receptive fields. A Lesnet-50 backbone with a good compromise between speed and accuracy can be used. Atros convolutions can be used, allowing an increase in the spatial density at which human hypotheses are evaluated, and a decrease in the number of missing detections. The final layer of the resulting fully convolutional layer is responsible for predicting multiple outputs from each of its neurons, corresponding to the properties of the person assumed to be at each location. In some embodiments, the output data 310 further includes a probability of the existence of an object (eg, a person). Corners of the object bounding box may also be regressed and form part of the output data 310 .

Comparison of the output data to the supervisor signals 306 of the input image 302 may be performed using a loss/objective function (L). The loss function includes terms that penalize differences between the candidate 3D representation and the 3D representation in the supervisory signal 306 . Many examples of such loss functions are known in the art, eg, L ₁ or L ₂ losses. The loss function may also include terms that penalize the differences between the positions of the candidate bounding boxes output by the neural network 308 and the ground truth bounding box positions in the input training image 302 . Many examples of such loss functions are known in the art, eg, L ₁ or L ₂ losses. The loss function may also include terms that penalize the output probabilities of the neural network 308 based on the actual presence/absence of objects at that location. Many examples of such loss functions are known in the art, eg, classification loss functions.

In some embodiments, the loss may only penalize the 3D representation and bounding box position in the presence of objects (eg, people) - in the absence of objects, the box and 3D shape take on arbitrary values. However, it is understood that the object detector will thin out the resulting hypotheses.

Parameter updates 312 may be determined by applying an optimization procedure, eg, stochastic gradient descent, to the loss/objective function. The loss function may be averaged over the batch of training images before determining each set of parameter updates. The process may be repeated across multiple batches in the training dataset.

In embodiments where a part-based approach to mesh estimation is used, each of the 3D representations in the supervisor signal 306 may be partitioned into parts. The division of each supervisory signal 306 into portions may be based on key points of the object 304 . For example, the supervisory signal 306 may be segmented into body parts (eg, hands, feet, etc.) based on joint positions. When determining the parameter updates 312 the partial mesh V[p] output by the neural network 308 is compared with the corresponding partial meshes in the supervisory signal 306 .

4 shows a flow diagram of a method for training a neural network for use in reconstructing a three-dimensional scene with multiple objects from a single two-dimensional image. The methods may be performed by one or more computers operating at one or more locations. The method may correspond to the training method described with reference to FIG. 3 .

In operation 4.1, one or more two-dimensional images are received. The 2D images may be in the form of the images described with reference to FIG. 1 . Each 2D image includes one or more objects of a given type (eg, human).

In operation (4.2), a training signal for three-dimensional reconstruction is obtained through adaptation of a three-dimensional model of an object to each two-dimensional image. This effectively creates a labeled training dataset comprising a plurality of 2D images, each associated with one or more 3D representations of objects (eg, humans) in the images.

Fitting the three-dimensional model may include projecting the three-dimensional representation onto a two-dimensional image plane to produce a projected representation; comparing the respective positions of the projected representation with the object of the single two-dimensional image; determining an error value based on the comparison; and adjusting the parameters of the fused three-dimensional representation based on the error value, wherein the comparing, measuring and adjusting steps are iteratively repeated until a threshold condition is satisfied. The threshold condition may be that the measured error value falls below a predetermined threshold value and/or that a threshold number of iterations is exceeded. The projecting step may be performed by taking into account the effects of perspective projection, and, if multiple views are available, using multiple views of the same object.

In operation (4.3), the resulting 3D model fitting results are used as supervisory signals for training of the deep machine learning model. The deep machine learning model may be trained as described above with respect to FIG. 3 .

5 shows a further example of a method 500 for training a neural network for use in three-dimensional scene reconstruction and object re-identification. The methods may be performed by one or more computers operating at one or more locations.

3 , the method 500 includes training a branch of a neural network 508 to output one or more re-identification (REID) embedding vectors 516a , 516b as an additional portion of output data 510 . It is an extension of the method described and illustrated above. As such, any of the features described with respect to FIG. 3 may be further combined with the features described with respect to FIG. 5 . Neural network 508 trained according to the methods described in connection with FIG. 5 is endowed with the ability to maintain object/person identity across a series of images. This can be important for video game experiences, as well as the accumulation of information over time and smoothing of object/person specific parameters, where every user is consistently associated with a unique character over time.

Person/object re-identification is generally addressed via two-stage architectures in the literature, where people/objects are first detected by a person/object detection system, and for every detected person/object, the image is It is pinged and sent as input to a separate human identification network. The latter provides a higher-order embedding that acts like a unique person/object signature that is trained to be invariant in disturbance parameters such as camera position, person/object pose, lighting, etc. This strategy has a complexity that scales linearly in the number of people, and is also difficult to implement as a two-stage on mobile devices.

Instead, the neural networks described with respect to Figures 1-4 can be extended to accommodate the task of object/person re-identification (REID) using a teacher-student network distillation approach. REID embeddings are extracted from object/human crops using a pre-trained REID network. These REID embeddings are used as supervisory signals to train the REID branch for the neural network. This branch carries REID embeddings that mimic those of the training network, but can be, for example, fully convolutional, meaning that its execution speed is independent of the number of objects.

Combining a network trained in such a way with a re-identification and tracking algorithm ensures that object/person identities are maintained in scenes with high person overlap, interactions, and occlusion, even when the user temporarily disappears from the scene. allow that Examples of such re-identification and tracking algorithms are "Memory based online learning of deep representations from video streams" (F. Pernici et al., CVPR 2018, Salt Lake City, UT, USA, 18 June 2018- 22, pages 2324-2334), the contents of which are hereby incorporated by reference. Maintaining object/person identification across multiple images in a scene may allow different virtual objects/skins to be consistently associated with each object/person in an image over time.

The 2D training image 502 is obtained from a training dataset including a plurality of 2D images. The 2D images are one or more objects 504a, 504b of a given type (e.g., humans) (in general there may be any number of objects, but in the example shown, two objects 504a, 504b). includes The 3D deformable model is fitted (eg, iteratively fitted) to objects in the input image 502 to extract one or more supervisory signals 506 for monocular 3D reconstruction. Additionally, each object 504a, 504b is cropped from the image (eg, using a ground truth bounding box for the object) and individually input to a pre-trained REID network 518 . The pre-trained REID network 518 processes the cropping of each input object 504a, 504b to encode the "teacher" re-identification embedding 520a, 520b (input image) to encode the identity of the input object 504a, 504b. For object i in 502, denoted by e ^T _i ). ^The teacher re-identification embeddings e _Ti , 520a , 520b are used as supervisory signals to train the re-identification branch of the neural network 508 .

Candidate 3D representations 514a, 514b of each (expected) object in the input training image 502 and "student" re-identification embeddings 516a, 516b of each (expected) object in the input training image 502 A training image 502 is input to a neural network 508 and processed through a series of neural network layers using the current parameters of the neural network 508 to generate output data 510 containing it. The output data 510 is compared with the supervisory signals 506 of the input image 502 and the teacher embeddings 520a , 520b to determine parameter updates 512 for the neural network 508 .

Each of the teacher REID embeddings 520a, 520b and the student REID embeddings 516a, 516b is a high-dimensional vector representing the identity of an individual object. For example, each REID embedding may be an N-dimensional vector. N may be, for example, 256, 512 or 1024.

When the loss/objective function is used to determine the parameter updates 512, the loss/objective function re-identifies comparing each teacher REID embedding 520a, 520b to its corresponding student REID embedding 516a, 516b. may include losses. The re-embedding loss may be an L ₁ or L ₂ loss between each teacher REID embedding 520a, 520b and its corresponding student REID embedding 516a, 516b.

6 shows a schematic example of a system/apparatus for performing any of the methods described herein. The depicted system/apparatus is an example of a computing device. It will be appreciated by those of ordinary skill in the art that other types of computing devices/systems, such as a distributed computing system, may alternatively be used to implement the methods described herein. One or more of these systems/devices may be used to perform the methods described herein. For example, a first computing device (eg, a mobile computing device) may be used to perform the method described above with respect to FIGS. 1 and 2 and/or a second computing device may be used with respect to FIGS. 3-5 . Thus, it can be used to perform the method described above. The model used in the first computation unit may have been trained using the second computation unit.

Device (or system) 600 includes one or more processors 602 . One or more processors control the operation of other components of system/device 600 . The one or more processors 602 may include, for example, general-purpose processors. The one or more processors 602 may be single-core devices or multi-core devices. The one or more processors 602 may include a central processing unit (CPU) or a graphics processing unit (GPU). Alternatively, the one or more processors 602 may include specialized processing hardware, eg, a RISC processor or programmable hardware with embedded firmware. Multiple processors may be included.

The system/device includes working or volatile memory 604 . One or more processors may access volatile memory 604 to process data and control storage of data within the memory. Volatile memory 604 may include any type of RAM, for example, static RAM (SRAM), dynamic RAM (DRAM), or may include flash memory such as an SD card.

The system/device includes non-volatile memory 606 . The non-volatile memory 606 stores in the form of computer-readable instructions a set of operational instructions 608 for controlling the operation of the processors 602 . Non-volatile memory 606 may be any type of memory, such as read-only memory (ROM), flash memory, or magnetic drive memory.

The one or more processors 602 are configured to execute operational instructions 608 that cause the system/device to perform any of the methods described herein. The operating instructions 608 may include code related to the basic operation of the system/device 600 as well as code related to hardware components (ie, drivers) of the system/device 600 . Generally speaking, the one or more processors 602 execute one or more instructions of operational instructions 608 stored permanently or semi-permanently in non-volatile memory 606 , generated during execution of the operational instructions 608 . Volatile memory 604 is used to temporarily store data.

Implementations of the methods described herein may be realized as digital electronic circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These include computer program products (eg, magnetic disks) comprising computer readable instructions that, when executed by a computer, such as described in connection with FIG. 6 , cause a computer to perform one or more of the methods described herein. , optical disks, memory, software stored on programmable logic devices).

Any system feature as described herein may also be provided as a method feature, and vice versa. As used herein, functionally described features may alternatively be expressed in terms of their corresponding structures. In particular, method aspects may be applied to system aspects and vice versa.

Moreover, any, some and/or all features in one aspect may be applied to any, some and/or all features in any other aspect in any suitable combination. It should also be understood that certain combinations of the various features described and defined in any of the aspects of the invention may be independently implemented, supplied, and/or utilized.

While several embodiments have been shown and described, it will be understood by those skilled in the art that changes may be made to these embodiments without departing from the principles of the disclosure, the scope of which is defined in the claims.

The various illustrative embodiments described herein are, in one aspect, in a computer program product embodied in a computer-readable medium comprising computer-executable instructions, such as program code, being executed by computers in networked environments. described in the general context of method steps or processes that may be implemented by Computer-readable media includes removable and non-removable storage devices including, but not limited to, read-only memory (ROM), random access memory (RAM), compact disks (CDs), digital versatile disks (DVD), and the like. may include Generally, program modules may include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code for executing the steps of the methods disclosed herein. A particular set of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps or processes.

In the foregoing specification, embodiments have been described with reference to numerous specific details that may vary from implementation to implementation. Certain adaptations and modifications of the described embodiments may be made. Other embodiments may be apparent to those skilled in the art in view of this specification and practice of the invention disclosed herein. This specification and examples are intended to be regarded only as examples having the true scope and spirit of the invention as indicated by the following claims. Also, it is intended that the series of steps shown in the figures are for illustrative purposes only, and are not intended to be limited to any particular order of steps.

As such, a person skilled in the art may understand that these steps may be performed in a different order while implementing the same method.

In the drawings and herein, exemplary embodiments have been disclosed. However, many variations and modifications may be made to these embodiments. Accordingly, although specific terms are employed, they are used in a generic and descriptive sense only and not for purposes of limitation, the scope of the embodiments being defined by the exemplary embodiments presented herein.

Claims (18)

A method for generating a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image, comprising:
receiving a single two-dimensional image;
identifying all objects in the image to be reconstructed and identifying the types of objects;
estimating a three-dimensional representation of each identified object;
estimating a three-dimensional plane that physically supports all three-dimensional objects; and
and positioning all three-dimensional objects in space with respect to the support plane. According to claim 1,
estimating the three-dimensional representation is performed in a deep machine learning model, the deep machine learning model comprising an output layer and one or more hidden layers each applying a nonlinear transformation to a received input to generate an output; method. 3. The method of claim 2,
The deep machine learning model predicts three-dimensional landmark positions of a plurality of objects by connecting feature data from one or more intermediate layers of the neural network, and the predicted three-dimensional positions are the predicted three-dimensional positions of an object depicted in each region. A method that is simultaneously estimated for a type. According to claim 1,
and estimating a plane supporting the plurality of objects is performed for a single frame using the estimated three-dimensional positions of all visible objects to reconstruct a two-dimensional plane passing through all visible objects. 12. The method of claim 11,
wherein estimating a plane supporting the plurality of objects is performed for a series of frames using relative camera pose estimation and plane localization using correspondences between points of successive frames. 6. The method according to any one of claims 1 to 5,
The receiving step further comprises receiving a plurality of images, wherein the steps of estimating the three-dimensional representations of a plurality of objects and locating them are performed for each received image, for example in real time. , method. 7. The method of claim 6,
wherein the processing occurs over a plurality of consecutive frames by combining the hidden layer responses in successive frames, eg, by averaging them. 8. The method according to any one of claims 4 to 7,
digital graphic objects are synthetically added to the three-dimensional scene reconstruction with a given relationship to the estimated three-dimensional object positions and then projected back into the two-dimensional image. A first computational unit for generating a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image, comprising:
The first calculation unit includes a memory; and at least one processor, wherein the at least one processor is configured to perform the method of any one of claims 1 to 8. A computer readable medium storing a set of instructions, comprising:
The set of instructions causes the first computation unit to perform a method according to any one of claims 1 to 8, to generate a three-dimensional reconstruction of a scene with a plurality of objects from a single two-dimensional image. A computer-readable medium executable by at least one processor of the first computing unit. A computer program product comprising:
A computer program product comprising instructions that, when the program is executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. A method for training a deep machine learning model to generate a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image, the method comprising:
receiving a single two-dimensional image;
obtaining a training signal for three-dimensional reconstruction through adapting a three-dimensional model of an object to the two-dimensional image, and using the resultant three-dimensional model fitting results as a supervisory signal for training the deep machine learning model , method. 13. The method of claim 12,
The fitting of the three-dimensional model may include generating a projected representation by projecting the three-dimensional representation onto a two-dimensional image plane;
comparing the respective positions of the projected representation to the object in the single two-dimensional image;
determining an error value based on the comparison; and
adjusting parameters of the fused three-dimensional representation based on the error value, wherein the comparing, measuring and adjusting steps are performed until the measured error value is below a predetermined threshold value or a threshold number of iterations is exceeded. iteratively repeated - carried out by a method. 14. The method of claim 13,
wherein the projecting is performed by taking into account effects of perspective projection and, if multiple views are available, using multiple views of the same object. A second computational unit for generating a three-dimensional reconstruction from a single two-dimensional image to estimate a three-dimensional representation of an object contained in the single two-dimensional image, comprising:
The second calculation unit may include: a memory; and at least one processor, wherein the at least one processor is configured to perform the method of any one of claims 12 to 14. A computer readable medium storing a set of instructions, comprising:
15. The set of instructions comprises: to cause a second computation unit to perform a method according to any one of claims 12 to 14, to update a fused three-dimensional representation of an object contained in a single two-dimensional image. A computer-readable medium executable by at least one processor of a second computation unit. A computer program product comprising:
15. A computer program product comprising instructions that, when the program is executed by a computer, cause the computer to perform the method of any one of claims 12 to 14. A system for providing a three-dimensional reconstruction of a scene with multiple objects from a single two-dimensional image, comprising:
a first calculation unit according to claim 9; and
A second calculation unit according to claim 15, comprising:
and the first calculation unit is trained with the results of the second calculation unit.

Images