RU2791587C1 - Method for providing computer vision - Google Patents

Abstract

FIELD: navigation of mobile robots for mobile applications.

SUBSTANCE: method performs scene understanding and object recognition. The method is proposed in which a point cloud is taken, reconstructed or obtained from depth sensors. First, this point cloud is represented as a voxel representation. The voxel representation is then processed with a 3D sparse convolutional "skeleton". Next, a 3D sparse convolution "neck" is used to extract features at different levels of the "skeleton". The resulting features are processed through a 3D sparse convolutional "head" with weights common to different feature levels. At each feature level, the "head" is responsible for predicting objects of a certain scale. The "head" gives the bounding box options for each position. Finally, all predictions received are filtered to select the most likely ones.

EFFECT: increasing the accuracy of object detection and reducing the scene processing time.

3 cl, 4 dwg, 4 tbl

Description

The field of technology to which the invention belongs

The invention relates to computer vision, in particular to the navigation of mobile robots, for mobile applications that perform scene understanding and object recognition.

Description of the Prior Art

Detection of 3D objects from point clouds is aimed at the simultaneous location and recognition of 3D objects from a provided set of 3D points. This method is widely used in autonomous driving, robotics and augmented reality as the main method for understanding 3D scenes.

While the 2D methods ([26], [32]) operate on dense fixed-size arrays, the 3D methods must operate on irregular, unstructured 3D data of arbitrary size. Therefore, 2D data processing methods cannot be applied directly to 3D object detection, and 3D object detection methods ([10], [22], [19]) use innovative approaches to 3D data processing.

The problem of convolutional methods for detecting 3D objects is scalability: processing large-scale scenes requires either an unreasonably large amount of computing resources or too much time. Other methods resort to representing voxel data and using sparse convolutions; however, these methods solve scalability issues at the expense of detection accuracy. In other words, there is no method for detecting 3D objects that provides accurate estimates and at the same time has good scaling.

Modern methods for detecting 3D objects are designed for either indoor or outdoor applications.

Methods for indoor and outdoor applications are developed almost independently of each other using domain-specific data processing methods. Many modern methods for outdoor applications [30], [13], [35] project 3D points onto a general perspective plane, thereby reducing the task of detecting 3D objects to detecting 2D objects. These methods, of course, take advantage of rapidly developing 2D object detection algorithms. In the presence of a projection of the general perspective in [14], it is processed by a completely convolutional method, and in [31], an anchorless 2D approach is used. Unfortunately, these approaches, which have proved to be effective for both 2D object detection and outdoor 3D object detection, are difficult to adapt to indoor applications, as this would require an unreasonably large amount of memory and computing resources. Various strategies for processing 3D data have been proposed to address performance issues. Currently, three approaches dominate in the field of 3D object detection - based on voting, based on the converter and 3D convolution. The following discusses each of these approaches in detail; a brief overview of anchorless methods is also presented.

Voting-Based Methods

The first method in which voting by points was introduced to detect 3D objects was the VoteNet method [22]. VoteNet processes 3D points using Point-Net [23], assigns a group of points to each candidate object according to their voting center, and computes feature features from each group of points. Of the many followers of VoteNet, the main progress has been in modern grouping and voting strategies applied to PointNet features. BRNet [4] refines voting results using representative points from voting centers, which improves the capture of fine local structural features. MLCVNet [29] introduces three context modules into the voting and classification steps of VoteNet to encode context information at different levels. H3DNet [33] improves the procedure for generating a group of points by predicting a hybrid set of geometric primitives.

VENet [28] contains an attention mechanism and introduces a vote weighting module trained with a new vote attractiveness loss.

All voting-based methods of the VoteNet type are limited by design. First, they do not scale well: since their performance depends on the amount of input data, they tend to slow down as scenes get larger. In addition, many voting-based methods implement voting and grouping strategies as custom layers, making it difficult to reproduce or debug these methods or port them to mobile devices.

Transducer based methods

Recent transformer-based methods use end-to-end learning and direct extension to inference instead of heuristics and optimization, making them less domain-specific. In GroupFree [16], the "head" of VoteNet is replaced by a transformer module that iteratively updates the positions of object requests and collects intermediate detection results. 3DETR [19] was the first 3D object detection method implemented as an end-to-end trainable transducer. However, the more advanced converter-based methods still have scalability issues similar to the earlier voting-based methods. In contrast, the proposed method is fully convolutional, so it is faster and easier to implement than the vote-based and transformer-based methods.

3D Convolutional Methods

The voxel representation allows efficient processing of cube-growing sparse 3D data. Voxel-based 3D object detection methods ([12], [18]) convert points to voxels and process them using 3D convolutional networks. However, dense volumetric features still consume a lot of memory, and 3D convolutions require large computational resources. In general, processing large scenes requires a lot of resources and cannot be done in a single pass.

GSDN [10] solves performance problems with sparse 3D convolutions. It has an encoder-decoder architecture in which both encoder and decoder parts are built from sparse 3D convolutional blocks. Compared to standard vote-and-transformer convolutional approaches, GSDN is significantly more memory efficient and scales to large scenes without sacrificing point density. The main disadvantage of GSDN is its accuracy: this method is comparable in quality to VoteNet, but is significantly inferior to the current level of development [16].

GSDN uses 15 aspect ratios for the bounding boxes of 3D objects as anchors. If GSDN is trained in unanchored setups with one aspect ratio, then its accuracy is reduced by 12%. Unlike GSDN, the proposed method does not contain an anchor, but takes advantage of sparse 3D convolutions.

Anchorless object detection based on RGB

When detecting 2D objects, anchorless methods compete with standard anchor methods. FCOS [26] performs 2D object detection using pixel-by-pixel prediction and shows a significant improvement over its anchor-based predecessor, RetinaNet [15]. FCOS3D [27] trivially adapts FCOS by adding additional targets for monocular detection of 3D objects. ImVoxelNet [24] solves the same problem with a FCOS head built from standard (non-sparse) 3D convolution blocks. The present invention adapts the ideas of the mentioned anchorless methods for processing sparse irregular data.

In addition to scalability and accuracy, an ideal 3D object detection method should be able to handle objects of arbitrary shape and size without additional hacks and manually tuned hyperparameters. Previous assumptions on the bounding boxes of 3D objects (such as aspect ratios or absolute sizes) limit generalization and increase the number of hyperparameters and trainable parameters.

SUMMARY OF THE INVENTION

Recently, more and more attention has been paid to the detection of 3D objects from 3D point clouds (computer vision) due to the possibility of its application in such promising areas as robotics and augmented reality.

A non-anchor method is proposed, in which priors on objects are not set, and 3D object detection is carried out solely on the basis of data. In addition, a new parameterization of the oriented bounding box (OBB) inspired by the Möbius strip is introduced, which reduces the number of hyperparameters. To prove the effectiveness of the parameterization, the authors performed experiments on SUN RGB-D with several methods for detecting 3D objects and provided data on improving the results for all these methods.

A method for providing computer vision of a robotic device is proposed, which is implemented in a computing device of a robotic device, moreover, the robotic device has a camera with depth sensors and an RGB camera, a central processor, internal memory, RAM, and the method includes the following steps :

capturing a real scene with objects in the scene using a depth sensor camera and an RGB camera;

represent the captured real scene as a cloud of a set of N points, each of which is represented by its own coordinate and color, and

a cloud of a set of N points is introduced as data into the neural network, and the neural network consists of a "skeleton" part, a "neck" part and a "head" part,

where the "skeleton" part, the "neck" part and the "head" part are the 3D sparse convolutional parts of the neural network;

in a neural network, the following steps are performed:

by means of the "skeleton" part, they represent a point cloud in the form of a volumetric pixel representation of the input data;

by means of the "skeleton" part, the volumetric pixel representation of the input data is processed to obtain four-dimensional tensors;

through the convolutional part of the "neck" process four-dimensional tensors to extract numeric features of objects in the scene;

the extracted features of the objects in the scene are processed by the head part to obtain predictions of the positions and categories of objects in the scene, and for each object in the scene, the head part produces predictions, where the prediction contains:

object classification probability, object bounding box regression parameters, and object centering within the object bounding box;

by means of the "head" part, filtering all received predictions by comparing the predictions by the probability of classifying the object to select the most probable prediction, while the most probable prediction is considered the final estimate of the position and category of the object in the scene and is characterized by data regarding the position and category of the object in the scene;

outputting data from the neural network regarding the position, orientation, and category of the object in the scene as a numerical representation of the scene representing computer vision for the robotic device.

In this case, the method may comprise an additional step, at which: converting the numeric representation of the scene into a scene image, wherein the computer device further comprises a screen, and the scene image is displayed on the screen for the user.

A computer-readable medium is provided, containing a program code that reproduces the method according to claim 1 when it is implemented in a computer device.

Meanwhile, the above-described method performed by the electronic device can be performed using artificial intelligence (AI). The AI-related function can be performed by non-volatile memory, volatile memory, and a processor.

The processor may include one or multiple processors. Currently, one or more processors may be a general purpose processor such as a central processing unit (CPU), an application processor (AP) and the like, a graphics-only processing unit such as a graphics processing unit (GPU), a computer vision processor (VPU). ) and/or a dedicated artificial intelligence processor such as a neural processing unit (NPU).

These one or more processors control the processing of input data according to a predefined operating rule or artificial intelligence (AI) stored in non-volatile memory and non-volatile memory. A predetermined operating rule or artificial intelligence is provided through training.

In this context, provisioning by learning means that by applying a learning algorithm to a set of training data, a predetermined operating rule or AI with a desired performance is created. The training may be performed directly on the device running the AI according to the embodiment and/or may be implemented by a separate server/system.

An AI can be made up of many layers of a neural network. Each layer has a plurality of weights and performs a layer operation by computing the previous layer and a plurality of weights operation. Examples of neural networks include, without limitation, Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network ( BRDNN), Generative Adversarial Networks (GANs), and Deep Q-Nets. A neural network can be implemented in hardware or firmware.

The learning algorithm is a method for training a given target device (eg, a robot) using a plurality of training data to induce, allow, or control the target device to perform a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, partially supervised learning, or reinforcement learning.

According to the invention, the object recognition method can obtain recognition output by using image data as input to artificial intelligence. Artificial intelligence can be obtained through training. In this context, "learned" means that a predetermined operating rule or artificial intelligence that performs the required function (or purpose) is obtained by learning a basic artificial intelligence on a set of pieces of training data using a training algorithm. Artificial intelligence can include many layers of a neural network. Each of the plurality of neural network layers includes a plurality of weight values and performs neural network calculations by calculating between the calculation result of the previous layer and the plurality of weight values.

Visual understanding is a method of recognizing and processing things similar to human vision, and includes, for example, object recognition, object tracking, image search, person recognition, scene recognition, 3D reconstruction/localization or image enhancement.

According to the invention, the proposed method can use artificial intelligence, performed by using data. The processor may perform a pre-processing operation on the data to convert it into a form suitable for use as input to an artificial intelligence model. Artificial intelligence can be obtained through training. In this context, "learned" means that a predetermined operating rule or artificial intelligence that performs a desired function (or goal) is obtained by training the underlying artificial intelligence with several pieces of training data using a learning algorithm. Artificial intelligence can include many layers of a neural network. Each of the plurality of neural network layers includes a plurality of weight values and performs neural network calculations by calculating between the calculation result of the previous layer and the plurality of weight values.

Reasoning prediction is a method of generating logical reasoning and obtaining predictions by identifying information, which includes, for example, knowledge-based reasoning, optimization prediction, preference-based planning, or recommendations.

Brief description of the drawings

The above and/or other aspects will become more apparent from the description of exemplary embodiments with reference to the accompanying drawings, which depict the following:

Fig. 1 is a general scheme of the proposed method.

Fig. 2 - examples of objects with an indefinite track angle.

Fig. 3 - the result of the proposed method with the ScanNet dataset.

Fig. 4 - dependence of detection accuracy on inference speed, measured in frames per second, for the original and modified FCAF3D in comparison with known methods for detecting 3D objects.

Detailed description

The present invention relates to computer vision and is the first in its class fully convolutional anchorless method for detecting 3D objects indoors, called FCAF3D. FCAF3D is a simple, efficient and scalable method for detecting 3D objects from point clouds.

A method for providing computer vision of a robotic device is proposed. This method can be implemented in the computer device of the robotic device. The robotic device contains a camera with depth sensors and an RGB camera, a central processor, internal memory, and RAM. Depth camera and RGB camera capture a real scene with 3D objects in that scene. The proposed method allows to detect and recognize the category and position of 3D objects in the captured scene.

The proposed solution is designed for scene analysis and object recognition. The results obtained can be used in a wide range of tasks where decisions are made based on the scene and its objects. For example, software based on the proposed method can provide mobile robotic devices (eg, mobile navigation robots) with spatial information for trajectory planning, grabbing and manipulating objects. In addition, the proposed solution can be used in a mobile application to automatically generate scene hints.

The proposed solution is designed to detect and recognize 3D objects and evaluate their spatial position. The formulation of the problem corresponds to the classical formulation of the problem of detecting 3D objects, formulated by the scientific community of computer vision.

The proposed solution is supposed to be implemented in mobile robotic devices with a computer device. The computer device of the robotic device contains a camera with depth sensors and an RGB camera, a central processor, internal memory, RAM, and a screen.

Also, the invention can be implemented in smartphones as part of a mobile application. To implement the proposed method can be used media, such as computer-readable media. At the same time, the computer-readable medium contains a program code that reproduces the proposed method when implemented in a computer device. In particular, the computer-readable medium must have sufficient RAM and computing resources. Resource requirements vary by device function, camera settings, and performance requirements.

It should be noted that 3D object detection traditionally uses a predefined set of 3D object bounding boxes, called anchors. Such a set can be considered as a set of a priori hypotheses about the position of objects in space (objects in the scene) and their sizes. The use of such a set of hypotheses makes it possible to detect objects in 3D space by selecting the most probable hypotheses and refining them. However, a priori hypotheses do not always describe real objects in a particular scene well, so the use of anchors limits the applicability of the object detection method. In the beginning, all object detection methods used anchors. Recently, a new approach has been described that allows not using anchors when solving the object detection problem; this allows a more versatile solution to be developed. Over the past few years, a whole class of methods has been formed that do not use anchors - they can be called "anchorless". Now they compete with traditional "anchor" methods.

The proposed convolutional anchorless method for detecting 3D objects indoors is a simple but effective method that uses a voxel representation of a point cloud (input data) and uses sparse convolution voxel processing.

In accordance with the proposed method, Npts points are captured in RGB colors and a set of bounding boxes of 3D objects is output. Points Npts in RGB colors are a set of N points, each of which is represented by its own coordinate and color in the proposed method, performed on a computer. Each point in three-dimensional space (point) is defined by three coordinates in space, as well as a color in the RGB palette (point in RGB colors). A set of points in 3D space is also called a point cloud. This cloud from a set of N points, each of which is represented by its own coordinate and color, can be obtained by processing the captured real scene with the objects in it. A cloud from a set of N points is entered into the neural network as data. Point coordinates are real numbers. Voxel is a common abbreviation for volume pixel, i.e. three-dimensional pixel. Typically a "2D" pixel in a 2D image is the base "cell" of the image. A pixel is an element of the image discretization: the image is divided into equal sections (elements) by a regular grid. Each such element has the shape of a square aligned with the sides of the image. By analogy with a pixel, a voxel is a part of a three-dimensional space bounded by a parallelepiped aligned along the coordinate axes and divided along the grid into grid elements. However, a voxel is defined more flexibly than a pixel. So, it is not necessary to divide space into elements by a regular three-dimensional grid: grid elements can be located in space in an arbitrary way. That is, the space is divided into grid elements arbitrarily. The center of a voxel is determined by averaging all point coordinates from the point cloud that fall within one grid element (i.e. x coordinates, y coordinates, and z coordinates are averaged). This means that each element of the grid corresponds to its own voxel. The result is a set of voxels that is as sparse and irregular as the original set of points in 3D space.

If the voxels are organized into a regular grid, then one speaks of a voxel volume, and in the absence of a regular structure, one speaks of a voxel representation. In addition, a voxel does not necessarily have the same spatial dimensions along all three axes: it may not be a cube, but an arbitrary parallelepiped, but cubic voxels are often used for the convenience of calculations.

The proposed FCAF3D method is capable of processing large-scale scenes using minimal runtime and memory in a single fully forward-propagated convolutional pass and does not require a heuristic post-processing step. Existing methods for detecting 3D objects use preliminary assumptions about the geometry of objects. Any geometric priors limit the generalizing ability of the method. Instead, the inventors propose a new Oriented Bounding Box (OBB) parameterization that allows better results without any priors. The proposed method provides the results of detection of 3D objects in units corresponding to the modern level. mAP@0.5 on ScanNet V2 (+4.5), SUN RGB-D (+3.5) and S3DIS (+20.5) datasets. mAP@0.5 is a standard metric for assessing the quality of 3D object detection. Possible values for mAP@0.5 range from 0 to 100. The higher the mAP@0.5 value, the higher the quality. On the S3DIS dataset, FCAF3D outperforms the competition by a huge margin.

Thus, the present invention provides the following contribution to the state of the art:

- the first in its class completely convolutional anchorless method for detecting 3D objects (FCAF3D) for indoor scenes is proposed;

- a new OBB parameterization is presented and proven to improve the accuracy of several existing 3D object detection methods on SUN RGB-D;

- the proposed method is significantly superior to the prior art on complex large-scale ScanNet, SUN RGB-D and S3DIS indoor datasets in mAP units, and at the same time it has a higher speed logical inference.

The task of detecting 3D objects (computer vision) is to detect and recognize three-dimensional objects and evaluate their spatial position in the scene. 3D objects have a complex, varied and sometimes changeable shape, which often cannot be described parametrically (by equations). Therefore, in the standard formulation of the object detection problem, the shape, size and position of objects are modeled by a simple three-dimensional figure - a parallelepiped, a "frame". Such boxes are called 3D bounding boxes. frames. They are defined by the 3D coordinates of the center of the frame, as well as the width, height, and length of the frame. For simplicity, it is assumed that all such parallelepipeds are oriented along the coordinate axes of three-dimensional space, i.e. all their edges and faces are directed along one of the axes. Sometimes they solve the problem in a more complex formulation, where the parallelepipeds are rotated in the horizontal plane, then the problem of object detection additionally includes determining the orientation of the object, i.e. angle of rotation. Each object also has a category label.

The category of an object is indicated in the markup (annotation) of the datasets. The annotation is a standard, reliable information contained in the original datasets. In this case, the annotation is a set of 3D bounding boxes of objects, with certain categories of objects for each point cloud contained in the dataset. An annotation is obtained with the involvement of evaluators when creating a dataset by its authors. An annotation is used to train a neural network: to learn how to make predictions based on input data, you need to look at a series of training examples of a given form (input data - the reference true output data contained in the markup) and find a pattern that allows you to establish a relationship between input and output data.

The categories are obtained as a result of peer review by an appraiser who is provided with 3D point clouds during the training data collection stage. Such 3D point clouds are derived from a set of RGB images and their respective depth maps, i.e. depth sensor measurements. For markup, special software is used that allows you to visualize point clouds on the screen, manipulate them without making changes to the source data (rotate, move and scale them to view from different sides), by clicking on the desired area of space or otherwise specifying the position of objects in space in the form of parallelepipeds covering them, i.e. 3D bounding boxes.

The proposed FCAF3D architecture consists of a skeleton part, a neck part, and a head part, these terms being generally accepted terms in the art. The designations of the parts of the object detection neural network as parts "skeleton", "neck" and "head" are used in articles describing such methods for detecting two-dimensional / three-dimensional objects as FCOS, ATSS, ImVoxelNet, FCOS3D. The "skeleton", "neck" and "head" parts are sparse 3D convolutional parts of the neural network.

The "skeleton" part of a neural network refers to a pre-trained neural network or a portion of a pre-trained neural network. Neural networks used as a "skeleton" are trained on large amounts of visual data, usually by solving an image classification problem. As a result of this training, they gain the ability to capture patterns in visual data. This ability can be used not only to solve the problem of image classification, but also to solve many other computer vision problems. The standard approach when designing a neural network for solving computer vision problems is to use a pretrained "skeleton", but with the replacement of some layers in it, designed to solve the classification problem, with other layers, designed to solve the target problem.

The "neck" part of the neural network takes as input the output of the "skeleton" - a four-dimensional tensor, and also returns four-dimensional tensors.

The "head" part of the neural network is the last, final layer of the neural network for obtaining predictions of the position, orientation, and categories of objects in the scene. The "head" part produces predictions for each of the objects in the scene. Each prediction contains the probability of classifying the object, the regression parameters of the object's bounding box, and the centering of the object within the object's bounding box.

The division into parts "neck" and "head" is conditional and formal, since each of these parts consists of neural network layers similar in type and purpose. In the case of the present invention, the neural network layers are sparse convolutional layers.

When developing the proposed FCAF3D, a sparse convolutional network of the GSDN type was chosen for scalability. To improve generalization, this network has reduced the number of hyperparameters that must be manually tuned; in particular, sparseness trimming in the neck is simplified. In addition, a simple multi-level position assignment has been introduced in the "head" part. Finally, the limitations of existing 3D bounding box parameterizations are discussed, and a new parameterization is proposed that improves both accuracy and generalization.

ResNet is a family of neural network architectures widely used to solve computer vision problems. The ResNet family has both lightweight architectures and more powerful architectures with more customizable options. Lightweight architectures are intended to be used when computing resources are limited and/or speed is important. More powerful architectures with a large number of customizable parameters show a better quality of solving the target problem compared to lightweight ones. Such powerful architectures are chosen when quality is a top priority and there is a significant amount of computing resources. All these architectures are arranged according to the same principle: they contain the same or similar computational units, interconnected in a certain way. As a result, the ResNet family is formed by neural network architectures with a different number of such computational units. Recently, a method has been described for modifying the architecture of the ResNet family of neural networks to adapt this architecture to process sparse 3D data (such as point clouds), although the original architectures of the ResNet family were designed to handle 2D data (images). Computing blocks of the neural network architecture of the ResNet family (like any other neural network architecture) consist of layers. In the neural network architecture of the ResNet family, there are two-dimensional convolutional layers in the computational blocks. The modification method is to replace all 2D convolutional layers with 3D convolutional layers. If such a modification is applied to all neural network architectures of the ResNet family, then a family of three-dimensional sparse neural network architectures of the ResNet family can be obtained. The proposed method is implemented in a modified neural network of the ResNet family.

To implement the computer vision method, a real scene with objects in that scene is captured using a depth sensor camera with an RGB camera. The captured real scene is represented by a computer device as a cloud of a set of N points, each of which is represented by its own coordinate and color (Npts point cloud in RGB colors). A cloud from a set of N points is entered into the neural network as data.

The depth sensor (depth camera) measures the distance to points in the scene and outputs the measurement results as a dense two-dimensional map, each pixel of which contains the distance from the depth camera to some point in the scene. Next, you need to determine how the coordinates of the pixels on the depth map correlate with the coordinates of the points in three-dimensional space, in other words, determine how the depth map is converted to three-dimensional space. To do this, you need to know the parameters of the depth camera, which determine the type of this display. The camera parameters are explicitly specified in the datasets used for the experiments in this work. In real conditions of application of the proposed method for detecting 3D objects, the camera parameters can be evaluated separately by any method for estimating camera parameters (this is a standard procedure, also known as camera calibration), or set directly in an explicit form, for example, as characteristics of a specific depth camera model.

A real scene with objects in it is captured by a camera with depth sensors and an RGB camera as a set of N points, each of which is additionally represented by its own coordinate and color. This requires that the RGB image pixels and the depth map pixels map to the same points in 3D space. Accordingly, it is necessary to match the pixels of the RGB image with the pixels of the depth map. To do this, the pixels of the depth map are displayed in 3D space (using depth camera settings) and then projected onto the image plane (using RGB camera settings). The result of this procedure is a depth map pixel-by-pixel aligned with the RGB image. Each point in 3D space mapped from a depth map pixel is assigned the RGB values that were recorded in the RGB image pixel corresponding to that depth map pixel.

The procedure for obtaining a point cloud from one RGB image and one depth map was described above. The measurements of any depth sensor are inaccurate. However, the measurement accuracy can be improved by having multiple measurements of a region of 3D space obtained from different points. In this case, it is possible to aggregate information from several measurements and thereby correct measurements on individual depth maps, or, for example, highlight some measurements as random outliers due to the imperfection of the measuring instrument and remove these outliers. Also, such aggregation of measurements allows you to increase the consistency of measurements in one 3D scene and get not a set of separate point clouds for each RGB image and depth map, but one point cloud that completely describes the entire scene. A number of RGB image aggregation and depth map methods have been developed, including simultaneous localization and mapping (SLAM) methods, signed truncated distance function (TSDF) integration methods, and others. The captured point cloud Npts in RGB colors is fed into the neural network.

A neural network consists of layers, each of which takes a tensor as input, evaluates a function on it, and returns the result, which is also a tensor. The layers may be sequential and/or parallel. The method of "assembling" layers into a neural network is also called neural network architecture.

In FIG. 1 shows the layers of the neural network. Point cloud Npts in RGB colors injected into the layers of the neural network; in fig. 1 convolutional layer is labeled "Conv0", this layer calculates a function called "convolution" (common mathematical term). The pool layer is labeled "Bundle" in FIG. 1, the pooling layer calculates the local maximum. The result of "Conv0" and "Union" is a sparse 3D tensor.

The sparse neural network for the proposed FCAF3D method is shown in FIG. 1. The following describes its operation.

Part "skeleton"

The "skeleton" part in FCAF3D is a sparse modification of ResNet [11], in which all 2D convolutions are replaced by 3D sparse convolutions. A family of sparse multidimensional versions of ResNet was first presented in [5], for brevity the authors call them HDResNet.

The "skeleton" part implements the following:

representation of the point cloud in the form of a volumetric pixel (voxel) representation of the input data, the Npts point cloud in RGB colors is represented as a voxel representation in the "Conv0" and "Unification" layers;

processing of the volumetric pixel representation of the input data with the receipt of four-dimensional tensors by the residual blocks.

As follows from FIG. 1, block 1, block 2, block 3, block 4 are the residual blocks. The residual block is a computational unit of the neural network architecture of the ResNet family. It consists of several layers of different types: convolutional layers, layers that perform normalization inside the minibatch, activation layer. A distinctive feature of this computing unit is a specially organized connection between the layers, called a through connection, or residual connection (which gave the name to both the computing unit - the residual block, and the ResNet family of neural network architectures - the residual networks).

Residual blocks are skip blocks that learn residual features based on layer inputs instead of learning unrelated features. They were introduced as part of the ResNet architecture. Formally, by denoting the desired underlying mapping as H(x), the authors allow stacked non-linear layers correspond to another mapping H(x) - x. The original mapping is converted to F(x) + x. F(x) acts as a remainder, hence the name "residual block". Intuition dictates that it is easier to optimize the residual mapping than the original unrelated mapping. In the extreme case, if the identity mapping were optimal, it would be easier to push the remainder to zero than it would be to match the identity mapping of a stack of non-linear layers. The presence of throughput connections makes it easier for the network to learn similar identity mappings.

Part of the neck

Features of the scene objects, expressed in numerical form, are extracted from different layers of the "skeleton" using the "neck" part. The "neck" part processes the 4D tensors from the "skeleton" part to extract the numeric features of the objects in the scene. Features are descriptions, representations, descriptors of objects in a format that a computer device can work with, i.e. in numerical form, as a set of numbers. These numbers can be organized as multidimensional matrices, i.e. tensors.

The meanings of these numbers are difficult to interpret, they are not visual: in general, it is impossible to specify a specific number and claim that it encodes a certain property of an object. According to this format, the features extracted by the "neck" are four-dimensional tensors.

The proposed neck part is a simplified GSDN decoder. The features on each layer of the neck part are processed with one 3D sparse transposed convolution operation and one 3D sparse sparse convolution operation.

Each operation of a transposed 3D sparse convolution with a kernel size of 2 can increase the number of non-zero values by a factor of 23. To prevent memory from growing too fast, GSDN uses a clipping layer that filters the input with a probability mask.

In GSDN, feature level probabilities are calculated using an additional convolutional scoring layer. This layer is trained with a custom loss to ensure consistency between predicted sparsity and anchors. In particular, voxel sparsity is set to positive if any of the subsequent anchors associated with the current voxel is positive. However, the use of this loss may not be optimal, as distant object voxels may be assigned with low probability.

For simplicity, the estimation layer with the associated loss has been removed and the probabilities from the classification layer in the "head" are used instead. A probability threshold is not configurable but is supported on most Nvox voxels to control the sparsity level, where Nvox is equal to the number of input points Npts. This is a simple yet elegant way to prevent sparsity from growing, since reusing the same hyperparameter makes the process more transparent and consistent.

As follows from FIG. 1, each of the layers Conv1, Conv2, Conv3, Conv4 is a convolutional layer. Convolutional neural network layers convolve the input and pass the result to the next layer. Each convolutional neuron processes data only for its receptive field. Convolutional neural networks are widely used to process data with a grid type topology (such as images), since convolution takes into account the spatial relationships between individual features.

The convolutional layers of the neural network convolve the input and pass the result to the next layer. Each convolutional layer processes data only for its receptive field. Convolutional neural networks are widely used to process data with a grid type topology (such as images), since convolution takes into account the spatial relationships between individual objects .

As follows from FIG. 1, TransConv1, TransConv2, TransConv3 are transposed convolutional layers. The transposed convolutional layer is the standard transposed convolutional layer. It takes a tensor as input, computes a convolution function on it, and returns the result of the convolution, which is also a tensor. It has parameters - the so-called convolution kernel, which is tuned during neural network training. It is essentially a convolutional layer, but it is able to increase the dimension of the input tensor by increasing its sparseness or duplicating values.

As follows from FIG. 1, "crop" is a crop layer. The clip layer is a non-standard layer used in GSDN. It takes a sparse 3D tensor and filters it with a probability mask. Feature-level probabilities are calculated using an additional convolutional scoring layer.

General part "head". The "head" part of an anchorless FCAF3D consists of three parallel sparse convolutional layers (see Fig. 1) with common weights for feature levels.

The extracted features obtained in the previous step are processed by a 3D sparse convolutional "head" with weights common to different feature levels. The "head" processes the extracted features of the scene objects to obtain predictions of the positions, orientations, and categories of objects in the scene, while the "head" outputs predictions for each of the objects in the scene. The prediction includes the probability of classifying the object, the regression parameters of the object's bounding box, and the centering of the object within the object's bounding box. The resulting predictions are filtered by comparing the predictions by the object classification probability to select the most likely prediction. The choice of the most probable prediction is considered as the final assessment of the position, orientation and category of the object in the scene and is characterized by the data of the position, orientation and category of the object in the scene. This numerical representation of the scene is used by the robotic device.

At any moment of computation, a set of coordinates of points in three-dimensional space is processed inside the neural network. The input is a set of point coordinates, at the input and output of each layer of the neural network there are data presented in the form of three-dimensional sparse tensors. All points are in the same 3D space. But the number of points and the coordinates of the points change during the calculation. Positions are the coordinates of the points that appear during the calculation. They are not exactly the same coordinates of the points that came in as input, but they are somewhere between the coordinates of the input points, in roughly the same region of space.

The coordinate system (coordinate grid) is defined through the coordinates of points in three-dimensional space and the marking of the bounding boxes of objects: all coordinates are defined in some coordinate system. In this coordinate system, the y-axis must have the same direction as the gravity vector, in which case the 3D bounding boxes of objects of the form OBB and AABB will be horizontal. AABB is short for Axis-Aligned Bounding Box (the bounding box is aligned to the coordinate axes). This is a general term for the 3D bounding box of some object, all edges and planes of which are directed along the coordinate axes of 3D space. OBB is short for Oriented Bounding Box (Oriented Bounding Box). This is a general term for the 3D bounding box of an object of a certain kind, located horizontally in space and arbitrarily rotated in the horizontal plane.

три параллельных разреженных сверточных слоя "головы" этой архитектуры выводят вероятности

классификации, параметры δ регрессии ограничивающей рамки и центрированность

, соответственно. Эта схема подобна простой и облегченной "голове" FCOS [26], но адаптирована для 3D данных.For every position

three parallel sparse convolutional "head" layers of this architecture output the probabilities

classification, bounding box regression parameters δ, and centrality

, respectively. This scheme is similar to the simple and lightweight FCOS "head" [26], but adapted for 3D data.

Classification probabilities mean that for each category of an object (table, chair, etc.) the probability that a given position is inside the three-dimensional bounding box of an object of that category ("given point belongs to an object of this category") is estimated.

As for the parameters of the bounding box regression, it should be clarified here that all existing methods for detecting 3D objects do not give a direct prediction of the 3D bounding box of an object in an explicit form. Typically, position-dependent 3D bounding box parameters are evaluated instead: for example, the distance from a given position to the 6 faces of the 3D bounding box. The bounding box is the 3D bounding box of an object.

описывает близость положения к центру эталонной (истинной) 3D ограничивающей рамки объекта, в которую попадает это положение. Центрированность - это относительная величина, которая может принимать значения от 0 до 1; чем ближе к центру, тем больше значение и тем ближе оно к 1.Centeredness

describes the proximity of a position to the center of the object's reference (true) 3D bounding box that the position falls within. Centering is a relative value that can take values from 0 to 1; the closer to the center, the larger the value and the closer it is to 1.

In other words, the head part returns the classification probabilities, the bounding box regression parameters, the centrality score for each position.

слой классификации выводит вероятности

классификации, слой регрессии выводит параметры δ регрессии ограничивающей рамки, а слой центрированности выводит оценку центрированности

, соответственно.As follows from FIG. 1, the "head" part refers to the convolutional layer. At the same time, the "head" part contains layers of regression, centering, and classification. For every position

classification layer outputs probabilities

classification, the regression layer outputs the bounding box regression parameters δ, and the centrality layer outputs the centrality score

, respectively.

As follows from FIG. 1, "Combining" refers to the combining layer. The pooling layer is the layer that computes the local maximum or local average standard layer. The pooling layer takes a tensor as input, returns a spatially ordered set of local maxima/local averages of that tensor, which is also a tensor.

Pooling layers reduce the dimensionality of data by merging the outputs of clusters of neurons on one layer into a single neuron on the next layer. Local pooling combines small clusters, usually 2x2 tiles are used. The global union acts on all feature map neurons. Two common types of pooling are widely used: maximum and average. Max pooling uses the maximum value of each local cluster of neurons in the feature map, while average pooling uses the average value.

During training, FCAF3D outputs positions for different feature levels to be assigned to true frames {b}.

The input is Npts points in RGB colors, and the output is the probability of object classification, the bounding box regression parameters, the centering index for each position (that is, for some set of points in 3D space). At the testing stage, 3D bounding boxes of objects are calculated from the classification probabilities, the bounding box regression parameters, and the centrality index. Examples of such 3D bounding boxes are shown in FIG. 3. Predictions of the position, orientation of an object in space, as well as its category are available in a numerical representation representing computer vision for a robotic device, and can be used by it in accordance with the task. A method for visualizing forecasts obtained using the proposed method is also implemented. The user can see an image of the scene with 3D object bounding boxes placed in it. These 3D bounding boxes are colored differently to encode different categories of objects. The color coding is fixed (the same color always corresponds to the same category) and arbitrary.

In ImVoxelNet's 3D object detection method, output positions (locations) are predicted at three levels, and the level has predefined maximum distances from the position to the edges of the object's 3D bounding box that can be assigned to that position. For three scales thresholds of 75 cm, from 75 cm to 1.5 m and more than 1.5 m, respectively, are set.

The present invention provides a simplified strategy for sparse data that does not require tuning of dataset-specific hyperparameters. For each bounding box (examples of 3D object bounding boxes are shown in FIG. 3), the last feature level at which the bounding box covers at least Nloc positions is selected. If there is no such attribute level, the first one is selected. Positions are filtered by center sampling [26], considering only points near the center of the bounding box as positive matches.

спариваются с ограничивающими прямоугольниками

.

- это эталонная (истинная) 3D ограничивающая рамка объекта, ассоциированная с положением

. Эталонные ограничивающие рамки объекта содержатся в разметке набора данных или могут быть получены непосредственно из этой разметки.By assigning some provisions

pair with bounding boxes

.

is the reference (true) 3D bounding box of the object associated with the position

. The object's reference bounding boxes are contained in the dataset's markup, or can be derived directly from that markup.

и значениями

3D центрированности.Accordingly, these positions are associated with true labels

and values

3D centering.

(истинные метки) означают эталонные категории объектов, которые известны или могут быть непосредственно получены для каждой ограничивающей рамки объекта из разметки набора данных.

(true labels) means reference categories of objects that are known or can be directly obtained for each object bounding box from the markup of the dataset.

(3D центрированность) означает центрированность. Центрированность описывает близость какого-либо положения к центру эталонной (истинной) 3D ограничивающей рамки объекта, в который попадает это положение. Центрированность - это относительная величина, которая может принимать значения от 0 до 1; чем ближе к центру, тем больше значение и тем ближе оно к 1.

(3D centered) means centered. Centerness describes the proximity of a position to the center of the reference (true) 3D bounding box of the object that the position falls within. Centering is a relative value that can take values from 0 to 1; the closer to the center, the larger the value and the closer it is to 1.

умножаются на 3D центрированность

непосредственно перед NMS, как предложено в [24].During evaluation inference

multiplied by 3D centering

immediately before NMS, as suggested in [24].

The overall loss function is formulated as follows:

. Потеря классификации Lcls представляет собой фокальную потерю, потеря регрессии Lreg представляет собой IoU, а потеря центрированности Lcntr представляет собой бинарную кросс-энтропию. Для каждой потери прогнозируемые значения обозначены крышечкой.In this case, the number of agreed positions Npos is equal to

. Classification loss Lcls represents focal loss, regression loss Lreg represents IoU, and centering loss Lcntr represents binary cross entropy. For each loss, the predicted values are indicated by a cap.

Classification loss Lcls represents focal loss, regression loss Lreg represents IoU, and centering loss Lcntr represents binary cross entropy. Focal loss, IoU, binary cross entropy are generic terms for various penalty functions used to train neural networks.

Parameterization of the bounding box (Fig. 3 shows an example of 3D object bounding boxes for clarity).

3D object bounding boxes can be axis-aligned (AABB) or oriented (OBB).

Therefore, AABB is horizontal and not rotated, while OBB is horizontal and randomly rotated. AABB can be defined by the center point (3 coordinates), length, width and height. For OBB, it is also necessary to set the angle of rotation in the horizontal plane - the ground angle θ.

, тогда как описание OBB включает в себя путевой угол θ: b^OBB=(x, y, z, w, l, h, θ). В обеих формулах x, y, z обозначают координаты центра ограничивающей рамки, а w, l, h - его ширину, длину и высоту, соответственно.AABB can be described as

, while the OBB description includes the track angle θ: b ^OBB =(x, y, z, w, l, h, θ). In both formulas, x, y, z denote the coordinates of the center of the bounding box, and w, l, h denote its width, length, and height, respectively.

δ можно сформулировать как набор из 6 элементов:AABB parametrization. The parametrization for AABB was proposed in [24]. In particular, for true AABB (x, y, z, w, l, h) and position

δ can be formulated as a set of 6 elements:

может быть тривиально получен из δ.Predicted AABB

can be trivially obtained from δ.

The track angle is the angle of rotation of the object's 3D bounding box in the horizontal plane, which determines the orientation of the object ("where the object is facing"). This is one of the parameters that the OBB oriented 3D bounding box defines (the OBB definition includes the track angle θ).

All modern methods for detecting 3D objects from point clouds solve the problem of estimating the path angle as a classification followed by regression. The track angle is divided into bins; then the exact track angle is regressed in the bin. For indoor scenes, the range from 0 to 2π is usually divided into 12 equal bins [22], [21], [33], [19]. For outdoor scenes, usually only two bins [30], [13] are used, since objects on the road can be either parallel or perpendicular to the road.

Estimation of the track angle value is carried out in two stages. First, a rough estimate is made: the range of values within which a given track angle falls is determined. Then, at the second stage, the value of the track angle in this interval is specified. These intervals are called a bin.

Once the track angle bin is selected, the track angle value is estimated by regression. VoteNet and other voting-based methods directly estimate the value of θ. Outdoor methods explore more sophisticated approaches, such as predicting the values of trigonometric functions. For example, SMOKE [17] estimates sin θ and cos θ and uses the predicted values to reconstruct the track angle.

In FIG. 2 shows objects in a room where the track angle has a precisely expressed value. In FIG. 2 shows examples of objects that look the same from several sides: a square table, a round table, another round table.

It should be noted that the track angle is the angle of rotation of the 3D bounding box of the 3D object in the horizontal plane, which determines the orientation of the object ("where the object is facing"). One of the parameters that the oriented 3D bounding box of the OBB true angle determines is the angle that characterizes the reference, true values of any of the object parameters estimated by this method. It is desirable that the values of the parameters estimated by this method (predicted, estimated, issued by the method) be as close as possible to the true values of the object's parameters. In this context, the true angle should be understood as the true track angle, i.e. the reference, true value of the track angle, known in advance from the labeling of the data set.

Accordingly, the true angle annotations can be chosen arbitrarily for these features, making the classification of track angle bins meaningless. To avoid culling correct predictions that do not match the annotations, the rotated IoU is used, since its value is the same for all possible track angle choices. Therefore, a parametrization of OBB is proposed that takes into account the ambiguity of rotation. It should be clarified that it is not always possible to unambiguously determine the orientation of an object, since some objects look the same from several sides: for example, a round stool, a round table, a square table, see fig. 2. Therefore, any value of the track angle taken as a reference will be chosen to some extent randomly. This is called pivot ambiguity.

The parametrization for OBB is based on the Möbius strip mapping, so the authors talk about the Möbius parameterization of OBB.

Considering OBB with parameters (x, y, z, w, l, h, θ), we denote q=w/l. If x, y, z, w+l, h are constant, then it turns out that OBB with

define the same bounding box. The set (q, θ), where θ ∈ (0, 2π], q ∈ (0,+inf), is topologically equivalent to the Möbius strip [20] up to this equivalence relation. Therefore, the authors can reformulate the problem of estimating (q, θ ) as a problem of predicting a point on a Möbius strip A natural way to embed a Möbius strip, which is a two-dimensional manifold, in Euclidean space is:

It is easy to check that the four points from Equation 3 are mapped to a single point in Euclidean space. However, experiments show that predicting only ln(q)sin(2θ) and ln(q)cos(2θ) gives better results than predicting all four values. Therefore, the authors opted for a pseudo-embedding of the Möbius strip in R ² . It is called "pseudo" because it represents the entire central circle of the Möbius strip, defined by ln(q)=0 to (0,0). Accordingly, it is impossible to distinguish points with In <7=0. However, In(q)=0 implies strict equality between w and I, which is rare in real scenarios. Also, the choice of angle has little effect on IoU if w=l; therefore, this rare case is ignored for the sake of accuracy of detection and simplicity of the method. As a result, a new parameterization of OBB is obtained:

тривиально выводится из δ. В предложенной параметризации w, l, θ нетривиальны и могут быть получены следующим образом:In the standard parametrization of Equation 2

is trivially derived from δ. In the proposed parametrization, w, l, θ are non-trivial and can be obtained as follows:

и размер

.where is the ratio

and size

.

Finally, the position, orientation, and category data of the object in the scene is output from the neural network as a numeric representation of the scene. The numeric representation of the scene is represented as a scene image.

Robotic devices can use the position, orientation, and category of objects in the numeric representation to plan paths within the scene to bypass those objects. Object categories can be used to manipulate only objects belonging to the required categories, for example, to load and transport the required objects according to the user's instructions, to clean furniture items of the specified categories, etc.

The position, orientation, and category of objects in the numeric representation can be used to automatically provide statistics about the objects present in a scene, such as keeping track of the number of items currently available in a retail area, or creating a textual description of a given scene in helper applications, such as helping blind people.

The numeric representation of the scene can be converted (using known techniques) into a scene image, wherein the computing device further comprises a screen, and the scene image is displayed on the screen to the user.

The scene image can be used to visually demonstrate the results of applying the proposed method to a person in artificial reality (AR) applications, in applications for monitoring or accounting for objects, etc. The consumer of the results in the form of images is a person.

The position, orientation and category of objects in an image representation can be used in AR to provide the user with information about the objects present in the scene and to supplement the captured image of the scene with a generated annotation of the objects present in the scene.

Experiments

The proposed method was evaluated on three benchmarks for detecting 3D objects: ScanNet V2 [7], SUN RGB-D [25], and S3DIS [1]. For all datasets, the named average accuracy (mAP) was used as a metric at IoU thresholds of 0.25 and 0.5.

The ScanNet dataset contains 1513 reconstructed 3D indoor scans with pointwise private and semantic labels for 18 object categories. In the presence of such an annotation, AABB is calculated according to the standard principle [22]. The training subset consists of 1201 scans, and 312 scans are left for validation.

SUN RGB-D is a dataset for understanding monocular 3D scenes containing over 10,000 indoor RGB-D images. The annotation consists of dotted semantic labels and OBB 37 object categories. As suggested in [22], the experiments were carried out with objects of the 10 most common categories. The parts allocated for training and validation contain 5285 and 5050 point clouds, respectively.

S3DIS. The Stanford Large-Scale 3D Indoor Spaces dataset contains 3D scans of 272 rooms from 6 buildings with 3D private and semantic annotations. Following [10], the proposed method was evaluated on furniture categories. AABB derived from 3D semantics. An official division was used, in which 68 rooms from zone 5 were designated for testing and the remaining 204 rooms constituted the training subset.

The same hyperparameters were used for all datasets, except for the following. First, the size of the output classification layer was equal to the number of object categories, which was 18, 10, and 5 for ScanNet, SUN RGB-D, and S3DIS.

Secondly, SUN RGB-D contains OBB, so additional targets δ7 and δ8 were predicted for this dataset; It should be noted that the loss function was not affected. Finally, ScanNet, SUN RGB-D, and S3DIS contain a different number of scenes, so each scene was repeated 10, 3, and 13 times per epoch, respectively.

Similar to GSDN [10], a sparse 3D modification of ResNet34 called HDResNet34 was used as a "skeleton". The "neck" part and the "head" part used the findings of the "skeleton" part at all feature levels. In the initial voxelization of the point cloud, the voxel size was set to 0.01 m and the number of points Npts to 100000. Accordingly, Nvox is 100000. Both ATSS [32] and FCOS [26] set Nloc to 32 to detect 2D objects. Accordingly, a feature level was chosen such that the bounding box covers at least Nloc=33 positions. 18 positions were obtained by sampling in the center. The NMS IoU threshold is 0.5.

Education. The FCAF3D method was implemented using the MMdetection3D framework [6]. The training procedure corresponded to the standard MMdetection scheme [3]: training took 12 epochs, and the learning rate decreased at the 8th and 11th epochs. An Adam optimizer was used with an initial learning rate of 0.001 and a weight reduction of 0.0001. All neural networks were trained on two NVidia V100s with batch size 8. Evaluation and performance tests were performed on one NVidia GTX1080Ti.

The present invention uses the evaluation protocol presented in [16]. Training and estimation are randomized since the input Npts are chosen randomly from the point cloud. To obtain statistically significant results, training is carried out 5 times and each trained neural network is tested independently 5 times.

Both best and average metrics for 5x5 tests are reported: this allows FCAF3D to be compared with 3D object detection methods that report either one best or average.

The FCAF3D method was implemented using the MMdetection3D framework [6].

The training procedure corresponded to the standard MMdetection scheme [3]: training took 12 epochs, and the learning rate decreased at the 8th and 11th epochs.

An Adam optimizer was used with an initial learning rate of 0.001 and a weight reduction of 0.0001. All neural networks were trained on two NVidia V100s with batch size 8. Evaluation and performance tests were performed on one NVidia GTX1080Ti.

results

Comparison with modern methods

Table 1 compares FCAF3D with known state-of-the-art methods on three indoor benchmarks.

Table 1 presents the results of FCAF3D and existing indoor 3D object detection methods that accept point clouds. The best metrics are in bold. FCAF3D outperforms previous modern methods: GroupFree (on ScanNet and SUN RGB-D) and GSDN (on S3DIS). The reported value of the metric is the best for 25 tests; in parentheses is the average value.

Table 1

The proposed method was evaluated on the ScanNet [7], SUN RGB-D [25] and S3DIS [1] datasets and demonstrated a convincing superiority over the known state of the art in all benchmarks (the average value is given in parentheses). On SUN RGB-D and ScanNet, the proposed method outperforms other methods by at least 3.5% mAP@0.5. On the ScanNet dataset, the proposed 3D object detection method is 4.5 points higher than the best competing 3D object detection method. On the SUN RGB-D dataset, it is 3.5 points higher. On the S3DIS dataset, it is 20.5 points higher. The same excellent results can be seen for the standard quality assessment metric mAP@0.25.

In FIG. 3 shows an example of ScanNet point clouds with predictable bounding boxes. In FIG. 3 shows a point cloud from ScanNet with AABB. The color of the bounding box indicates the category of the object. Left: evaluation by the proposed method (FCAF3D), right - true. Each category of objects has its own bounding box color. The categories are color-coded, namely: blue (marked as C) - chair, orange (marked as O) - table, green (marked as G) - door, red (marked as R) - cabinet. It can be seen that the 3D bounding boxes predicted by the proposed method (left) are similar to the true bounding boxes (right). This result clearly illustrates the proposed method.

Similar results were obtained for a point cloud from SUN RGB-D with OBB, as well as for a point cloud from S3DIS with AABB.

To study geometric priors, existing methods were trained and evaluated with the proposed modifications. Experiments were carried out with methods for detecting 3D objects that accept data of various modalities: point clouds, RGB images, or both, to see if in FCAF3D the above-mentioned losses were replaced by rotated IoU losses with the Möbius parameterization according to equation 5 . To obtain a complete picture, we used the sin-cos parameterization used in the SMOKE method for detecting 3D objects outdoors [17].

Rotated IoU loss reduces the number of trainable parameters and hyperparameters, including geometric priors and loss weights. This loss has already been used in the detection of 3D objects outdoors [34]. Recently, [6] reported the results of VoteNet training with axis-aligned IoU loss on ScanNet.

Table 2 shows that replacing the standard parameterization with the Möbius parameterization improves mAP@0.5 VoteNet and ImVoteNet by about 4%.

ImVoxelNet does not use a "classification + regression" scheme to estimate the track angle, but predicts its value directly in one step. Because the original ImVoxelNet uses rotated IoU losses, the authors only needed to change the parameterization without having to remove the excess loss. Once again, the Möbius parametrization helped to obtain better results, although this superiority is negligible.

table 2

Table 2 illustrates the results of several methods for detecting 3D objects, taking various modalities as input, with various OBB parameterizations on SUN RGB-D. The value of the FCAF3D metric is the best for 25 tests; the mean value is given in parentheses. For other methods, the results from the original papers are presented, as well as the results obtained from the proposed experiments with reimplementations (denoted as Reimpl) based on MMdetection3D. "RS" - a cloud of points. "RGB" - an RGB image or a set of RGB images. "RGB+PC" - an RGB image and a point cloud, or a set of RGB images and a point cloud. VoteNet is a voting-based 3D object detection method that accepts a point cloud in RGB colors. ImVoteNet is a voting-based 3D object detection method that accepts an RGB image and a point cloud, or a set of RGB images and a point cloud. ImVoxelNet is a 3D object detection method that accepts an RGB image or a set of RGB images. "Reimpl." means reintroduction. VoteNet and ImVoteNet were reintroduced for experimentation because the source code for these methods was not published. Both the results presented in the original papers and the results obtained using reimplemented methods are presented. These results prove the correctness of the re-implementation and provide comparable accuracy to the accuracy reported in the original papers.

“w/naive param.” means naive OBB parameterization, where each OBB parameter is evaluated directly. This parameterization is used in the original VoteNet.

“w/sin-cos param.” means sin-cos OBB parametrization. The sin-cos parameterization is formulated in the SMOKE method for detecting 3D objects outdoors.

“w/Mobius param.” stands for Mobius OBB parameterization.

As can be seen, all the methods studied, in particular VoteNet, ImVoteNet, ImVoxelNet, FCAF3D, benefit from the use of the proposed Möbius OBB parameterization. The results obtained with the Möbius parametrization are better than the results obtained with both the "naive" parametrization and the sin-cos parametrization described by the authors of the SMOKE method. The observed improvement is the same for various 3D object detection methods that accept different types of input.

GSDN anchors were then examined to prove that anchor-based layers have limited generalizability. According to Table 3, mAP@0.5 drops drastically by 12% if the GSDN is trained without anchors (equivalent to using a single anchor). In other words, GSDN performs poorly without domain-specific guidance in the form of anchors; therefore, this method is inflexible and non-generic. All FCAF3Ds presented in Table 3 correspond to the proposed method and have different ResNet modifications (HDResNet34, HDResNet34:3, HDResNet34:2) and different voxel sizes.

For comparison, FCAF3D with the same "skeleton" was evaluated, and it outperformed GSDN by a huge margin, reaching twice the mAP value.

Table 3

Table 3 presents the results of fully convolutional methods for detecting 3D objects that accept point clouds in ScanNet.

GSDN results obtained with 0.05 m voxels are reported. Although smaller voxels provide a more detailed representation of the data, the relationship between voxel size and accuracy is not straightforward. Changing the voxel size affects GSD and FCAF3D differently due to different assignment strategies. In FCAF3D, the authors tried to densely "cover" 3D space with positions. The spacing between positions is proportional to the voxel size, so the smaller the voxel size, the denser the coverage and hence the higher the response. GSDN "covers" 3D space with anchors. The linear dimensions of the anchors are proportional to the size of the voxels. The assignment ignored anchors with a small overlap with the bounding boxes. Decreasing voxel size reduces anchors; accordingly, some anchors will be ignored, resulting in a lower response.

So FCAF3D generally benefits from smaller voxels, while GSDN does not. In general, GSDN performance is expected to drop as the voxel size decreases from 0.05 to 0.01.

Next, the neural network design solution for the proposed method (FCAF3D) is discussed and its impact on metrics when applied independently in ablation studies is examined. The experiments were carried out with different voxel sizes, the number of points in the point cloud Npts, the number of positions selected by center sampling, as well as with and without centering. The results of ablation studies are summarized in Table 4 for all benchmarks.

Table 4

Table 4 presents the results of ablation studies on voxel size, number of points (which is equal to the number of Nvox voxels when trimmed), centering, and center sampling in FCAF3D. The best options are in bold (actually the default options used to generate the results in Table 1 above). The presented metric value is the best for 25 tests; in parentheses is the average value.

Voxel size. As you might expect, accuracy decreases with increasing voxel size. Voxels with sizes of 0.03, 0.02, and 0.01 µm were used. The noticeable gap in mAP between voxel sizes of 0.01 and 0.02 µm is explained by the presence of nearly flat objects such as doors, paintings, and boards. In particular, with a voxel size of 2 cm, the "head" will output positions with a tolerance of 16 cm, but almost flat objects can be less than 16 cm in one of the dimensions. There is a decrease in accuracy for large voxel sizes.

Amount of points. Similar to 2D images, thinned point clouds are sometimes referred to as low resolution clouds. Accordingly, they contain less information than their high resolution versions. As expected, the fewer points, the lower the detection accuracy. In this series of experiments, 20 thousand, 40 thousand and 100 thousand points were selected from the entire point cloud, and the obtained metric values revealed a clear relationship between the number of points and mAP. The authors do not believe that large values of Npts correspond to the level of existing methods (in particular, GSDN [10] uses all points in the point cloud, GroupFree [16] selects 50 thousand points, VoteNet [22] selects 40 thousand points for ScanNet and 20 thousand points for SUN RGB-D). Nvox=Npts is used for cutting direction in the "neck". When Nvox exceeds 100 thousand, the inference time increases due to the growing sparsity in the neck, while the accuracy improves slightly. Therefore, the grid search was limited to Npts 100 thousand and was used as the default value for the results obtained.

Centeredness. Using centering improves mAP for ScanNet and SUN RGB-D datasets. For S3DIS, the results are inconsistent: the better value of mAP@0.5 is offset by a slight decrease in mAP@0.25. However, the authors analyzed all the results and therefore can consider centering to be a useful feature with a small positive effect on mAP, almost reaching 1% of mAP@0.5 in ScanNet.

Center sampling. Finally, the authors examined the number of positions obtained in the centered sample. 9 positions are selected as proposed in FCOS [26], the entire set of 27 positions, as in ImVoxelNet [24], and 18 positions. The last choice turned out to be the best in terms of mAP on all benchmarks.

Centeredness. Using centering improves mAP for ScanNet and SUN RGB-D datasets. For S3DIS, the results are inconsistent: better mAP@0.5 is offset by a slight decrease in mAP@0.25. However, all results were analyzed and centering was found to be a useful feature with a small positive effect on mAP, almost reaching 1% of mAP@0.5 in ScanNet.

Center sampling. Finally, the number of positions selected in the center sample was examined. 9 positions were selected as proposed in FCOS [26], the entire set of 27 positions as in ImVoxelNet [24], and 18 positions. The last choice turned out to be the best in terms of mAP on all benchmarks.

Inference speed

Compared to standard folds, sparse folds are time and memory efficient. The GSDN authors claim that with sparse convolutions, they process a 78M-point scene spanning about 14,000m3 in one fully forward-propagated convolutional pass using as little as 5GB of GPU memory. FCAF3D uses the same sparse convolutions and the same "skeleton" as GSDN. However, as Table 3 shows, FCAF3D is slower than GSDN by default. This is due to the smaller voxel size: the authors use 0.01 m for proper layering, while GSDN uses 0.05 m.

To create the fastest method, the authors used HDResNet34:3 and HDResNet34:2 "skeletons" with only three and two feature levels, respectively. With these modifications, FCAF3D has a higher inference rate than GSDN (FIG. 4). In FIG. Figure 4 shows the dependence of detection accuracy on inference speed, measured in scenes per second, for the original and modified FCAF3D in comparison with existing methods for detecting 3D objects. In FIG. 4 shows mAP@0.5 on ScanNet relative to scenes per second. FCAF3D modifications have a different number of "skeleton" feature levels. For each existing method, there is a FCAF3D modification that surpasses this method both in detection accuracy and inference speed. According to the graph, the proposed FCAF3D method without accelerating modifications is the most accurate of all methods for detecting 3D objects; however, it is slower than some of the other methods. However, for each existing method, there is a FCAF3D modification that surpasses this method both in detection accuracy and inference speed.

In FIG. Figure 4 compares the key characteristics of the proposed FCAF3D method and its two modifications (FCAF3D with 3 levels and FCAF3D with 2 levels) with existing methods that solve a similar problem: GroupFree, H3DNet, BRNet, 3DETR, 3DETR-m, VoteNet, GSDN. The comparison was performed on the ScanNet dataset. Each point on the graph corresponds to some way of detecting 3D objects.

Y-axis: Performance (mAP) - mAP@0.5, a metric of 3D object detection accuracy. Higher values correspond to higher precision; accordingly, the higher the methods are located on the chart, the better they are.

X-axis: Scenes per second is the number of scenes processed per second. Higher values correspond to higher speed; accordingly, the more to the right the methods are located on the chart, the better they are.

For each of the existing methods, there is a modification of the proposed method, which simultaneously shows better accuracy and is faster (in other words, the point of which is located above and to the right on the graph). For GroupFree, H3DNet, BRNet, 3DETR, 3DETR-m, VoteNet, these are FCAF3D and FCAF3D with three levels. For GSDN - FCAF3D with two levels.

For an objective comparison, the authors re-measured the inference speed for GSDN methods and voting-based methods, since the point grouping operation and sparse convolutions have become much faster since the introduction of these methods. In performance tests, the authors chose implementations based on the MMdetection3D framework [6] to reduce differences in the code base. The reported inference speed for all methods was measured on the same GPU, so they can be compared directly.

FCAF3D is proposed, a first-in-class fully convolutional anchorless 3D object detection method for indoor scenes. The proposed method significantly outperforms the state of the art on the complex SUN RGB-D, ScanNet, and S3DIS indoor benchmarks in both mAP values and inference speed. A new oriented bounding box parameterization is also proposed and shown to improve accuracy for several 3D object detection methods. In addition, the proposed parameterization makes it possible to eliminate any preliminary assumptions about objects, thereby reducing the number of hyperparameters. In general, FCAF3D with the proposed bounding box parameterization is an accurate, scalable, and at the same time generalizable method. The proposed software solution is supposed to be executed in mobile robotic devices or run on smartphones as part of a mobile application. To implement the proposed method, the carrier device must meet the technical requirements. In particular, it must have sufficient RAM and computing resources. Resource requirements vary by device function, camera settings, and performance requirements.

The above embodiments of the invention are examples and should not be construed as limiting. In addition, the description of exemplary embodiments of the invention is intended to be illustrative and not to limit the scope of the claims, and many alternatives, modifications, and variations will be apparent to those skilled in the art.

Literature

1. Armeni, I., Sener, O., Zamir, A.R., Jiang, H., Brilakis, I., Fischer, M., Savarese, S.: 3d semantic parsing of large-scale indoor spaces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1534-1543 (2016).

2. Chen, J., Lei, B., Song, Q., Ying, H., Chen, D.Z., Wu, J.: A hierarchical graph network for 3d object detection on point clouds. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 392-401 (2020).

3. Chen, K., Wang, J., Pang, J., Cao, Y., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Xu, J ., et al.: Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155 (2019).

4. Cheng, B., Sheng, L., Shi, S., Yang, M., Xu, D.: Back-tracing representative points for voting-based 3d object detection in point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8963-8972 (2021).

5. Choy, C., Gwak, J., Savarese, S.: 4d spatio-temporal convnets: Minkowski convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 3075-3084(2019).

6. Contributors, M.: MMDetection3D: OpenMMLab next-generation platform for general 3D object detection. (2020)

https://github.com/open-mmlab/mmdetection3d.

7. Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Niesner, M.: Scannet: Richly-annotated 3d reconstructions of indoor scenes. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5828-5839 (2017).

8. Engelmann, F., Bokeloh, M., Fathi, A., Leibe, B., Niesner, M.: 3d-mpa: Multiproposal aggregation for 3d semantic instance segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9031-9040 (2020).

9. Ge, Z., Liu, S., Li, Z., Yoshie, O., Sun, J.: Ota: Optimal transport assignment for object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 303-312 (2021).

10. Gwak, J., Choy, C., Savarese, S.: Generative sparse detection networks for 3d single-shot object detection. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16. pp. 297-313. Springer (2020).

11. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778 (2016).

12. Hou, J., Dai, A., Niesner, M.: 3d-sis: 3d semantic instance segmentation of rgbd scans. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4421-4430 (2019).

13. Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., Beijbom, O.: Pointpillars: Fast encoders for object detection from point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12697-12705 (2019).

14. Li, B.: 3d fully convolutional network for vehicle detection in point cloud. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 1513-1518. IEEE (2017).

15. Lin, T.Y., Goyal, P., Girshick, R., He, K., Doll'ar, P.: Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 2980-2988 (2017).

16. Liu, Z., Zhang, Z., Cao, Y., Hu, H., Tong, X.: Group-free 3d object detection via transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 2949-2958 (2021).

17. Liu, Z., Wu, Z., T'oth, R.: Smoke: Single-stage monocular 3d object detection via keypoint estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 996-997 (2020).

18. Maturana, D., Scherer, S.: Voxnet: A 3d convolutional neural network for realtime object recognition. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 922-928. IEEE (2015).

19. Misra, I., Girdhar, R., Joulin, A.: An end-to-end transformer model for 3d object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2906-2917 (2021)

20. Munkres, J.R.: Topology (2000).

21. Qi, C.R., Chen, X., Litany, O., Guibas, L.J.: Imvotenet: Boosting 3d object detection in point clouds with image votes. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4404-4413 (2020).

22. Qi, C.R., Litany, O., He, K., Guibas, L.J.: Deep hough voting for 3d object detection in point clouds. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9277-9286 (2019).

23. Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 652-660 (2017).

24. Rukhovich, D., Vorontsova, A., Konushin, A.: Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpose 3d object detection. arXiv preprint arXiv:2106.01178 (2021).

25. Song, S., Lichtenberg, S.P., Xiao, J.: Sun rgb-d: A rgb-d scene understanding benchmark suite. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 567-576 (2015).

26. Tian, Z., Shen, C., Chen, H., He, T.: Fcos: Fully convolutional one-stage object detection. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9627-9636 (2019).

27. Wang, T., Zhu, X., Pang, J., Lin, D.: Fcos3d: Fully convolutional one-stage monocular 3d object detection. arXiv preprint arXiv:2104.10956 (2021).

28. Xie, Q., Lai, Y.K., Wu, J., Wang, Z., Lu, D., Wei, M., Wang, J.: Venet: Voting enhancement network for 3d object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3712-3721 (2021).

29. Xie, Q., Lai, Y.K., Wu, J., Wang, Z., Zhang, Y., Xu, K., Wang, J.: Mlcvnet: Multilevel context votenet for 3d object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10447-10456 (2020).

30. Yan, Y., Mao, Y., Li, B.: Second: Sparsely embedded convolutional detection. Sensors 18(10), 3337 (2018).

31. Yin, T., Zhou, X., Krahenbuhl, P.: Center-based 3d object detection and tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11784-11793 (2021).

32. Zhang, S., Chi, C., Yao, Y., Lei, Z., Li, S.Z.: Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9759-9768 (2020).

33. Zhang, Z., Sun, B., Yang, H., Huang, Q.: H3dnet: 3d object detection using hybrid geometric primitives. In: European Conference on Computer Vision. pp. 311-329. Springer (2020).

34. Zhou, D., Fang, J., Song, X., Guan, C., Yin, J., Dai, Y., Yang, R.: Iou loss for 2d/3d object detection. In: 2019 International Conference on 3D Vision (3DV). pp. 85-94. IEEE (2019).

35. Zhou, Y., Tuzel, O.: Voxelnet: End-to-end learning for point cloud based 3d object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4490-4499 (2018).

Claims (16)

1. A method for providing computer vision of a robotic device, implemented in a computer device of a robotic device, wherein the robotic device has a camera with depth sensors and an RGB camera, a central processor, internal memory, RAM, the method includes the steps of: capturing a real scene with objects in the scene using a depth sensor camera and an RGB camera; represent the captured real scene as a cloud of a set of N points, each of which is represented by its own coordinate and color, and a cloud of a set of N points is introduced as data into the neural network, and the neural network consists of a "skeleton" part, a "neck" part and a "head" part, the skeleton part, the neck part, and the head part are 3D sparse convolutional neural network parts; in a neural network, the following steps are performed: A) by means of the "skeleton" part, a point cloud is represented as a volumetric pixel representation of the input data; B) through the "skeleton" part, the volumetric pixel representation of the input data is processed to obtain four-dimensional tensors; C) by means of the "neck" part, four-dimensional tensors are processed to extract numeric features of objects in the scene; D) the head part processes the extracted features of the objects in the scene to obtain predictions of the positions, orientations, and categories of the objects in the scene, wherein for each object in the scene, the head part produces predictions, the prediction comprising: object classification probability, object bounding box regression parameters, and object centering within the object bounding box; E) through the "head" part, filter all received predictions by comparing the predictions by the probability of classifying the object to select the most likely prediction, while the most likely prediction is considered the final estimate of the position, orientation and category of the object in the scene and is characterized by data regarding the position, orientation and category object in the scene; output from the neural network data regarding the position, orientation and category of the object in the scene, in the form of a numerical representation of the scene representing computer vision for the robotic device. 2. The method of claim 1, further comprising the step of: converting the numeric representation of the scene into a scene image, wherein the computer device further comprises a screen, and the scene image is displayed on the screen for the user. 3. A computer-readable medium containing a program code that reproduces the method according to claim 1 when implemented in a computer device.

Images