RU2776814C1 - Method and electronic device for detecting three-dimensional objects using neural networks - Google Patents

Abstract

FIELD: computer vision technologies.

SUBSTANCE: present invention relates to computer vision technologies, and more particularly to methods and electronic devices for detecting three-dimensional objects or for understanding a scene, implemented at least in part with the help of neural networks. The effect is achieved by obtaining monocular images; extracting 2D feature maps from the monocular images by passing one or more monocular images through the 2D feature extraction portion; creating an average 3D voxel volume based on 2D feature maps; extracting a 2D representation of the 3D feature maps from the averaged 3D voxel volume by passing the averaged 3D voxel volume through the part encoder to extract the 3D features and performing 3D object detection as 2D object detection in the bird's eye view (BEV) plane, wherein the detection of two-dimensional objects in the BEV plane is implemented by passing a two-dimensional representation of three-dimensional feature maps through a part for detecting objects in street scenes containing parallel two-dimensional convolutional layers for classification and location.

EFFECT: improving the detection accuracy of three-dimensional objects in monocular images.

17 cl, 4 dwg

Description

FIELD OF TECHNOLOGY TO WHICH THE INVENTION RELATES

[0001] The present invention relates to computer vision technologies. More specifically, the present invention relates to methods and electronic devices for detecting three-dimensional objects or for understanding a scene, implemented using neural networks.

BACKGROUND OF THE INVENTION

[0002] RGB images are an accessible and versatile data source. Therefore, methods for detecting 3D objects based on RGB have been actively researched recently. Monocular RGB images provide visual clues to a scene and its objects, but they do not contain comprehensive information about the scene geometry and the absolute scale of the data. Therefore, it is extremely useful for a number of applications to be able to accurately and reliably detect 3D objects in such images. In the presence of a monocular image, deep learning methods for detecting 3D objects are only able to determine the scale of the data. Moreover, it is not possible to obtain consistent scene geometry from RGB images, as some areas may not be visible. However, using multiple images captured at known positions could provide more information about the scene than a monocular RGB image. Therefore, some well-known methods for detecting three-dimensional objects implement multi-view inference. These methods obtain predictions independently for each monocular RGB image, and then these predictions are combined.

[0003] All RGB-based 3D object detection methods are designed for use in indoor or outdoor scenes and operate under certain assumptions about the scene and objects. For example, methods for use in street scenes are typically evaluated on vehicles. Typically, vehicles in the same category are similar in size, positioned on the ground, and do not cross in a bird's eye view (BEV) under normal, non-emergency conditions. Accordingly, the projection of the BEV plane contains a lot of information about the three-dimensional location of the vehicle. Thus, the task of detecting three-dimensional objects for application in street scenes in a point cloud can be effectively reduced to the task of detecting two-dimensional objects in the BEV plane. At the same time, objects in indoor scenes can have different heights and be arbitrarily located in space, so their projections on the floor plane carry little information about their three-dimensional position. In general, the development of RGB-based 3D object detection methods tends to be region specific. In view of the foregoing, it is important to note that the object detection networks used in prior art methods typically have a complex and poorly designed structure because they are likely to have the most complex type of neural network architectures to build, debug and use.

The essence of the invention

[0004] The proposed technology and related technical solutions solve or at least minimize the problems encountered in known technical solutions for detecting objects in a scene or for understanding a scene based on at least one monocular image. At the same time, in contrast to the known methods, in the proposed technical solution, multiview input data are available immediately and are easy to obtain both during the training stage and during the output stage (for example, using a conventional digital camera of an electronic device). Therefore, multi-view input data is used not only for inference, but also for training. Both during training and during inference, the proposed solution accepts multiview inputs captured at known positions with an arbitrary number of views; this number can be unique for each multiview input. In this context, posed inputs/images captured mean that each RGB image has a corresponding known camera position . In addition, the proposed solution can accept monocular input data captured at known positions (considered as a special case of multiview input data). Also, the proposed technical solution works well on purely monocular references (that is, in the case of output based on a single monocular RGB image).

[0005] To accumulate information from a set of input data, the proposed technical solution builds a voxel representation of three-dimensional space. Such a unified approach (through the constructed 3D voxel representation) is used in the proposed technical solution for detecting objects in both indoor and outdoor scenes, as well as, at least in part, for understanding the scene. Understanding the scene is considered in this case as a subtask of the main task of object detection. Then, depending on whether a given scene is an indoor scene or an outdoor scene, or whether scene comprehension needs to be performed, the method can be configured to select and apply (perhaps based on one or more criteria) the appropriate head (term "head end" means one or more of the last convolutional layers applied in the main neural network) in the main neural network without changing the meta-architecture of the main neural network. "Meta-architecture" in the present invention may mean at least taking one or more images, extracting a 2D feature map from the images (using a 2D CNN), projecting the 2D feature map into 3D space, and aggregating the features in the 3D feature map. Examples of head parts to be described in detail below include, but are not limited to, an outdoor scene object detection part, an indoor scene object detection part, and a scene understanding part. Other point cloud based heads known in the prior art can also be used in the present invention due to the proposed core neural network meta-architecture.

[0006] In the proposed technical solution, the final predictions are obtained from three-dimensional feature maps or their two-dimensional representations (in the embodiment of the method for use in street scenes), which corresponds to the formulation of the detection problem based on a point cloud. Based on this, as briefly mentioned above, it is possible to use the standard intermediate part and head part of the point cloud-based object detector without any modifications. Deep learning models for 2D/3D object detection can usually be divided into 3 blocks: backbone, neck, and head. In this context, "reference part" means any neural network or at least one or more neural network layers that are responsible for feature extraction, "intermediate part" means any neural network or at least one or more neural network layers that are responsible for feature refinement (e.g., 2D to 2D in the proposed 2D CNN or 3D to 3D in the proposed 3D CNN. "Head" means any finite neural network or at least one or more finite neural network layers, e.g. for evaluation object frames (location, size) and categories (eg chair, table, car, etc.).

[0007] In summary, the following conclusion can be drawn: the proposed technical solution provides at least a triple result: (1) it uses end-to-end learning for multi-view object detection based only on RGB images captured at known positions; this greatly simplifies the collection of training data, as training requires only easily acquired/accessible conventional monocular RGB images; (2) a new full-convolutional object detection neural network is used, which works in both monocular and multi-angle modes, which increases the versatility of the network; (3) the proposed object detection neural network has a modular design, so that different domain-specific head parts and intermediate parts can be used without changing the meta-architecture of the main network; building a highway from building blocks helps to develop a reliable and efficient solution; (4) and finally, the accuracy of object detection by the proposed technical solutions exceeds all known technical solutions using neural network architectures trained on training data sets (ScanNet, SUN RGB-D, KITTI and nuScenes), which are most commonly used for object detection.

[0008] In a first aspect of the present invention, there is provided a method for detecting 3D objects using a neural network for object detection, comprising a 2D feature extraction part, a 3D feature extraction part, and an object detection part in street scenes, which are pretrained end-to-end based on only captured in known positions of monocular images. The method includes the steps of: obtaining one or more monocular images; extracting 2D feature maps from each of the one or more monocular images by passing the one or more monocular images through the 2D feature extraction portion; creating an average 3D voxel volume based on the 2D feature maps; extracting a 2D representation of 3D feature maps from the averaged 3D voxel volume by passing the averaged 3D voxel volume through a 3D feature extractor part encoder, and performing 3D object detection as 2D object detection in a bird's eye view (BEV) plane, wherein the 2D object detection objects in the BEV plane is implemented by passing a 2D representation of 3D feature maps through a part for detecting objects in street scenes containing parallel 2D convolutional layers for classification and location. The method according to the first aspect of the invention of the present invention is adapted to work in street scenes. Thus, the method according to the first aspect of the present invention can be performed, for example, by an electronic device provided with at least one camera for detecting zero or more three-dimensional objects in the surrounding area of the vehicle.

[0009] In a second aspect of the present invention, a method for detecting 3D objects using a neural network for object detection is provided, comprising a 2D feature extraction part, a 3D feature extraction part, and an object detection part in indoor scenes that are pretrained end-to-end based on only captured in known positions of monocular images. The method comprises the steps of: obtaining one or more monocular images; extracting 2D feature maps from each of the one or more monocular images by passing the one or more monocular images through the 2D feature extraction portion; creating an average 3D voxel volume based on the 2D feature maps; extracting refined 3D object maps from the averaged 3D voxel volume by passing the averaged 3D voxel volume through the 3D object extraction part, and performing 3D object detection by passing the refined 3D feature maps through the indoor scene object detection part using a dense voxel representation of intermediate features, and containing 3D convolutional layers for classification, centering, and location. The method according to the second aspect of the invention may be performed by an electronic device equipped with at least one camera for detecting zero or more three-dimensional objects in the surrounding area of the electronic device. Non-limiting examples of the electronic device used in the method according to the second aspect of the invention include, smartphone, tablet, smart glasses, mobile robot.

[0011] In a modification of the second aspect of the invention, a method for understanding a scene using an object detection neural network is provided. The object detection network in this modification contains a part for understanding the scene, which is pre-trained in an end-to-end manner, and the method further comprises the steps of: performing a global average pooling of the extracted 2D feature maps to obtain a tensor representing the extracted 2D feature maps, and estimating the configuration (layout) scene by passing a tensor through a scene comprehension portion configured to jointly evaluate camera rotation and a 3D scene configuration. The scene understanding part contains two parallel branches: two fully connected layers that output the 3D scene configuration, and two other fully connected layers that evaluate camera rotation.

[0012] In a third aspect of the invention, there is provided an electronic device mounted on a vehicle equipped with at least one camera. This electronic device contains a processor configured to implement the method according to the first aspect of the invention to detect zero or more three-dimensional objects in the surrounding area of the vehicle.

[0013] In a fourth aspect of the invention, an electronic device provided with at least one camera is provided. The electronic device comprises a processor configured to implement the method of the second aspect of the invention for detecting zero or more three-dimensional objects in the surrounding area of the electronic device. Non-limiting examples of an electronic device include a smartphone, tablet, smart glasses, mobile robot.

[0014] In a fifth aspect of the invention, an electronic device provided with at least one camera is provided. This electronic device includes a processor configured to implement a method according to a modification of the second aspect of the invention for detecting zero or more three-dimensional objects in the surrounding area of the electronic device. Non-limiting examples of an electronic device include a smartphone, tablet, smart glasses, mobile robot.

[0015] In a seventh aspect of the invention, there is provided a computer-readable medium storing computer-executable instructions that, when executed by a processor, cause the processor to perform the method of the first aspect, the second aspect, or modifications of the second aspect.

Brief description of the drawings

[0016] The above and other aspects, features and advantages of the present invention will become more apparent from the following detailed description, together with the accompanying drawings, which depict the following:

1 is a flowchart of a method for detecting 3D objects using a pre-trained object detection neural network having a part for detecting objects in street scenes according to the first embodiment of the present invention.

Figure 2 is a schematic representation of specific parts of the architecture a pre-trained neural network for object detection; and operations of the 3D object detection method according to the first embodiment of the present invention.

3 is a flowchart of a method for detecting 3D objects using a pre-trained object detection neural network having a part for detecting objects in indoor scenes according to a second embodiment of the present invention.

4 is a schematic representation of specific architectural parts of a pre-trained neural network for object detection and operations of a 3D object detection method according to a second embodiment of the present invention.

DESCRIPTION OF EMBODIMENTS

[0017] The technical solution disclosed in this application receives or otherwise receives a set of monocular images of an arbitrary size, along with camera positions for such images. Camera positions can be estimated from images by any known method, such as MVS (Multi-View Stereo), SLAM (Simultaneous Localization And Mapping), camera tracking, etc. Alternatively, estimated camera positions associated with corresponding images of the monocular image set may be taken or otherwise obtained as additional information in any suitable form.The image set may include one or more images.The images in this set may be, but are not limited to, RGB images or any other color images.

[0018] The proposed solution then extracts features (eg, in the form of 2D feature maps) from one or more images using the 2D feature extraction part of the main object detection neural network. Any 2D convolutional reference part such as VGGNet, ResNet, ResNet-50, MobileNet, or other Convolutional Neural Networks (CNNs) can be used as part of the 2D feature extraction. The extracted image features are then projected into a 3D voxel volume. For each voxel, the projected features from multiple images are aggregated in any known manner, such as by simple element-wise averaging. The voxel volume with projected and aggregated features is then passed to the 3D feature extraction part of the main object detection neural network. The 3D feature extraction part may be a 3D convolutional network, the so-called "intermediate part". The output (eg, in the form of 3D feature maps) of the 3D feature extraction part is input to the object detection part of the main object detection neural network. The object detection part may be formed by, but not limited to, the last few convolutional layers, the so-called "head" in the main object detection neural network. The object detection part is trained to perform object detection. For example, the object detection part may perform object detection by predicting the features of the bounding box for each anchor. An anchor is one or more location candidates in 3D space, possibly with additional a priori information such as the average size of an object or its aspect ratio. The resulting bounding boxes of detected objects can be parameterized as (x, y, z, w, h, l, and), where (x; y; z) - coordinates of the center of the bounding box;w, h, l - width, height and length, respectively, andand - angle of rotation around the axis z . The operations briefly described above will be described in detail below with reference to the accompanying figures. 2D feature projection and 3D intermediate network can be implemented, for example, as described in (1) Z. Murez, T. van As, J. Bartolozzi, A. Sinha, V. Badrinarayanan, and A. Rabinovich. Atlas: End-to-end 3D scene reconstruction from posed images (Through 3D Scene Reconstruction from Images Captured at Known Positions), 2020 or (2) J. Hou, A. Dai, and M. NieЯner. 3D-sis: 3D semantic instance segmentation of RGB-d scans,2019.

[0019] Next, the method according to the first embodiment of the present invention will be described in more detail with reference to Fig . 1 and Fig . 2 . 1 is a flowchart of a method for detecting 3D objects using a pre-trained object detection neural network (also referred to as "core neural network") having a part for detecting objects in street scenes according to an embodiment of the present invention. The method starts and proceeds to step S100 of receiving one or more monocular images. As briefly mentioned above, one or more monocular images can be RGB images or any other color images in one of various formats. Let I _t ∈ R ^{W × H × 3} be the t -th image in the set T of images. Here T >1 in case of multiview input and T =1 for single input.

,

,

, и

. Конкретное количество (т.е. четыре) двумерны карт признаков, выводимых частью для извлечения двумерных признаков, а также конкретные заданные формы выходных двумерны карт признаков не следует рассматривать в качестве ограничений настоящей технологии, поскольку также возможны другие количества двумерных карт признаков и другие формы двумерных карт признаков. В качестве альтернативы, могут быть выведены и использованы впоследствии всего три или всего две двумерные карты признаков.[0020] The method then proceeds to step S105 of extracting 2D feature maps from each of the one or more monocular images by passing the one or more monocular images through the 2D feature extraction portion shown as a "2D CNN" in FIG . 2. The 2D feature extraction part outputs four 2D shape feature maps

,

,

, and

. The specific number (i.e., four) of 2D feature maps output by the 2D feature extraction part, as well as the specific desired shapes of the output 2D feature maps, should not be construed as limitations of the present technology, as other numbers of 2D feature maps and other shapes of 2D features are also possible. feature maps. Alternatively, as few as three or as few as two 2D feature maps can be derived and used subsequently.

.

и

являются зависящими от опорной части; конкретные заранее определенные значения

и

будут представлены ниже в качестве примера. Карты признаков F ₁-F _T, имеющие агрегированные признаки, показаны на фиг.2 для соответствующих входных изображений I ₁-I _T.[0021] The features in the resulting 2D feature maps can then be aggregated, for example, through a Feature Pyramid Network (FPN) that produces a single tensor F _t of the shape

.

and

are dependent on the reference part; specific predefined values

and

will be presented below as an example. Feature maps F ₁ - F _T having aggregated features are shown in FIG . 2 for the respective input images I ₁ - I _T .

. Ось z задана перпендикулярной плоскости пола, ось x направлена вперед, а ось y ортогональна обеим осям x и z. Для каждого набора данных (например, ScanNet, SUN RGB-D, KITTI и nuScenes) существуют известные пространственные пределы, оцененные эмпирически для всех трех осей. Обозначим эти пределы как x _min, x _max, y _min, y _max, z _min, z _max. Для фиксированного размера вокселя s пространственные ограничения можно сформулировать как N _x s=x _max−x _min, N _y s=y _max−y _min, N _z s=z _max−z _min. Затем можно использовать модель камеры с точечной диафрагмой, но без ограничения, для определения соответствия между 2D координатами (u, v) в двумерной карте признаков F _t и 3D координатами (x, y, z) в объеме V _t:[0022] The method then proceeds to step S110 of generating an average 3D voxel volume based on the 2D feature maps. For t -x inputs, the extracted 2D feature map F _t is then projected into a 3D voxel volume

. The z -axis is set perpendicular to the floor plane, the x -axis is directed forward, and the y -axis is orthogonal to both the x and z axes. For each data set (eg ScanNet, SUN RGB-D, KITTI and nuScenes) there are known spatial limits, empirically estimated for all three axes. Let's denote these limits as x _min , x _max , y _min , y _max , z _min , z _max . For a fixed voxel size s , the spatial constraints can be formulated as N _x s = x _max − x _min , N _y s = y _max − y _min , N _z s = z _max − z _min . You can then use the camera model with, but not limited to, a pinhole to determine the correspondence between the 2D coordinates ( u, v ) in the 2D feature map F _t and the 3D coordinates ( x, y, z ) in the volume V _t :

,

,

[0023] K and R _t are matrices of internal and external parameters of the camera, respectively, and P is a perspective display. The internal parameters of the matrix of internal parameters of the camera determine the position and orientation of the camera in relation to the world coordinate system. Camera extrinsics matrix extrinsic parameters allow conversions between camera coordinates and pixel coordinates in an image frame. Specific values for internal and external parameters can be set for the camera as a result of the camera calibration process.

[0024] After the 2D feature maps are projected, all voxels along the camera beam are filled with the same features as the corresponding 2D feature map(s). A binary mask M _t may be defined if necessary. The shape of the binary mask M _t may correspond to the shape of the volume V _t . This binary mask indicates whether each voxel is inside the truncated camera pyramid, i.e. whether the given voxel is an "internal (visible) voxel" or not. Thus, for each image I _t , the mask M _t can be defined as:

[0025] Then a two-dimensional feature map F _t is projected for each internal (i.e., actual) voxel in volume V _t :

[0026] Figure 2 shows internal volumes V ₁ - V _T with projected features corresponding to feature maps F ₁ - F _T having aggregated features for the respective input images I ₁ - I _T .

[0027] The aggregated binary mask M is the sum of M ₁ ,..., M _t :

[0028] Finally, an averaged 3D voxel volume V is generated by averaging the projected features in the volumes V ₁ ,..., V _t over the internal voxels:

, сгенерированный на основе внутренних объемов V ₁-V _T со спроецированными признаками, соответствующими картам признаков F ₁-F _T, имеющим агрегированные признаки, для соответствующих входных изображений I ₁-I _T.[0029] Figure 2 shows the averaged 3D voxel volume of a V shape

, generated on the basis of internal volumes V ₁ - V _T with projected features corresponding to feature maps F ₁ - F _T having aggregated features, for the corresponding input images I ₁ - I _T .

[0030] Since the part for detecting objects in street scenes receives two-dimensional feature maps, it is necessary to obtain a two-dimensional representation of the generated three-dimensional voxel volume for use in the proposed method. Therefore, returning to FIG . 1, after the average 3D voxel volume V has been generated, the method proceeds to step S115 of extracting a 2D representation of 3D feature maps from the average 3D voxel volume. This step is performed by passing the averaged 3D voxel volume only through the encoder of the 3D feature extraction part shown as "3D CNN" in FIG . 2. The 3D feature extraction part used in this step may correspond to the encoder-decoder architecture disclosed in the aforementioned work (1).

преобразуется в тензор формы

, являющийся двумерным представлением трехмерных карт признаков усредненного трехмерного воксельного объема. Полученный тензор формы

представляет собой выходные данные (показан на фиг.2 как "3D признаки") части для извлечения трехмерных признаков. Благодаря этапу S115 задачу обнаружения трехмерных объектов в трехмерном пространстве можно свести к задаче обнаружения двумерных объектов в плоскости BEV. Обе части - часть для извлечения трехмерных признаков, показанная на фиг.2 как "3D CNN" и часть для обнаружения объектов в уличных сценах, показанная на фиг.2 как "2D Conv" для данного варианта осуществления способа, содержат 2D свертки.[0031] After passing through several 3D convolutional and downsampling layers of the encoder part for extracting 3D features, the averaged 3D voxel volume of V shape

converted to shape tensor

, which is a 2D representation of 3D feature maps of the averaged 3D voxel volume. The resulting shape tensor

is the output (shown as "3D features" in FIG . 2) of the 3D feature extraction part. Through step S115, the task of detecting three-dimensional objects in three-dimensional space can be reduced to the task of detecting two-dimensional objects in the BEV plane. Both the 3D feature extraction part shown as "3D CNN" in FIG . 2 and the street scene detection part shown as "2D Conv" in FIG . 2 for this embodiment of the method contain 2D convolutions.

, являющегося представлением усредненного трехмерного воксельного объема V в плоскости BEV, способ переходит на этап S120 выполнения обнаружения трехмерных объектов в форме обнаружения двумерных объектов в плоскости BEV. Обнаружение двумерных объектов в плоскости BEV реализуется путем передачи двумерного представления трехмерных карт признаков через часть для обнаружения объектов в уличных сценах, содержащую параллельные двумерные сверточные слои для классификации и определения местоположения. "Параллельность" по отношению к слоям в данном контексте означает, что выходные данные предпоследнего сверточного слоя подаются на 2 конечных слоя (один для классификации, другой для определения местоположения), которые не зависят друг от друга, т.е. обработка в этих слоях может выполняться одновременно. Таким образом, задача обнаружения трехмерных объектов в уличных сценах переформулируется в способе согласно первому варианту осуществления в задачу обнаружения двумерных объектов в плоскости BEV.[0032] Finally, after getting the shape tensor

, which is a representation of the average 3D voxel volume V in the BEV plane, the method proceeds to step S120 of performing 3D object detection in the form of 2D object detection in the BEV plane. Detection of 2D objects in the BEV plane is implemented by passing a 2D representation of 3D feature maps through a part for detecting objects in outdoor scenes containing parallel 2D convolutional layers for classification and location. "Parallelism" in relation to layers in this context means that the output of the penultimate convolutional layer is fed to 2 final layers (one for classification, the other for location), which are independent of each other, i.e. processing in these layers can be performed simultaneously. Thus, the task of detecting 3D objects in street scenes is reformulated in the method according to the first embodiment as a task of detecting 2D objects in the BEV plane.

[0033] As part of detecting objects in street scenes in this embodiment of the method may be used, but not limited, working with 2D snaps warhead shown to be effective in (3) A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom. Pointpillars: Fast encoders for object detection from point clouds, 2019, and (4) Y. Yan, Y. Mao, and B. Li. Second: Sparsely embedded convolutional detection, 2018, on the KITTI and nuScenes datasets. Any other point cloud head can be used with the proposed core object detection neural network.

[0034] Because 3D detection methods for use in street scenes are typically evaluated on vehicles or other objects on or near the road (e.g., pedestrians, road infrastructure such as road markings and traffic signs), all such objects have a similar given scale and belong to fixed categories. For single scale and The single-class detection part for detecting objects in street scenes contains two parallel convolutional 2D layers. One layer evaluates the probability of the class, and the other regresses the seven parameters of the object's bounding box if it is found.

. Конкретное значение

предопределено и будет проиллюстрировано ниже.[0035] Input to the part for detecting objects in street scenes. This input, as already briefly noted above, is the shape tensor

. specific meaning

predetermined and will be illustrated below.

[0036] The output of the part for detecting objects in street scenes. For each anchor in the 2D BEV, the part for detecting objects in street scenes returns a class probability p and a 3D bounding box, for example, as the following set of seven elements:

,

,

,

,

,

,

.

,

,

,

,

,

,

.

ограничивающей рамки постоянна для всех привязок, поскольку они расположены в плоскости BEV и вычисляются для одного класса объектов (например, автомобилей).[0037] In this case, ^.gt and ^.a , respectively, denote the true bounding boxes (ground truth) and bounding boxes of the anchor. Diagonal length

The bounding box is constant for all anchors because they are in the BEV plane and are calculated for the same feature class (for example, cars).

[0038]Losses. To train the part to detect objects in street scenes, the following loss function can be used, but without limitation. The total loss in street scenes may include several loss terms, namely the smoothed mean absolute deviation as position lossL _loc, focal loss for classificationL _cls and cross-entropy loss for directionL _dir. In general, the total losses in street scenes can be formulated, for example, as follows:

,

,

- это количество положительных привязок; привязка считается положительной, если хотя бы один истинный объект соответствует этой привязке.

,

,

, но без ограничения указанными значениями.[0039]

is the number of positive bindings; an anchor is considered positive if at least one true object matches that anchor.

,

,

, but not limited to the specified values.

[0040] As described above with reference to FIGS . 1 and 2 , the object detection core neural network used in the 3D object detection method according to the first embodiment of the present invention creates a 3D voxel representation of space. Thus, it can use a head or intermediate part from any of the known methods for detecting three-dimensional objects based on a point cloud. Therefore, instead of the time-consuming implementation of a non-standard and complex architecture, modern warheads can be used with appropriate methods and without additional modifications. Thus, the modular design of the object detection neural network used in the object detection method according to the first embodiment of the present invention provides increased flexibility and simplifies the development process. However, it should be understood that head designs for applications in outdoor scenes and indoor scenes differ significantly.

[0041] An embodiment of the method described with reference to FIGS . 1 and 2 is adapted for use in street scenes and can be executed by a vehicle electronic device equipped with at least one camera to detect zero or more three-dimensional objects in the environment. vehicle area. Such an electronic device may be a vehicle autonomous driving system or at least a part of it. Such a system may include at least one or more cameras mounted on a vehicle and configured to capture images of the vehicle's environment around the vehicle, and a processor configured to perform or initiate various vehicle-related operations. It should be understood that the invention disclosed in this application does not claim to be a technical solution for autonomous driving as such, but it may be part of such a technical solution for a vehicle responsible for detecting three-dimensional objects.

[0042] Let us now turn to a more detailed description of the method according to the second variant of the present invention with reference to Fig.3 and Fig.4 . 3 is a flowchart of a method for detecting 3D objects using a pre-trained object detection neural network having a part for detecting objects in indoor scenes according to the second embodiment of the present invention. Since steps S200, S205, S210 of the method of the second embodiment (in indoor scenes) correspond to steps S100, S105, S110 of the first embodiment of the method (in outdoor scenes), repeated descriptions of steps S200, S205, S210 will be omitted for brevity. In addition, since the meta-architecture (i.e. 2D CNN, 3D CNN plus additional operations between them, as shown in Fig.2 and 4 ) of the main object detection neural network in both versions of the proposed and described method remains the same, the description of the meta- the architecture shown in Figure 4 will also be omitted for brevity.

[0043] Referring now to FIGS. 3 and 4 , after the average 3D voxel volume V has been generated, the method proceeds to step S215 to extract refined features (e.g., if 3D features are input, refined 3D features corresponding to the inputted 3D features are output ) 3D feature maps from the averaged 3D voxel volume. This step is performed by passing the averaged 3D voxel volume through the entire 3D convolutional encoder-decoder architecture of the 3D feature extraction portion shown in FIG . 4 as "3D CNN". The 3D feature extraction part may correspond to the encoder-decoder architecture disclosed in the above document (1).

[0044] Each layer of the encoder-decoder architecture used at this stage contains residual blocks with a predetermined number of 3D convolutional layers with a kernel of a predetermined size. In one example, the specified number of 3D convolution layers is two and the specified kernel size is three. For downsampling, convolutions are used with a kernel of a given size and a given convolution step (stride). In one example, the specified kernel size of downsampling-related convolutions is three and the specified convolution step is two. Upsampling may be performed, for example, by trilinear interpolation followed by convolution with a kernel of a given size. In one example, the upsampling-related convolution kernel size is one. However, the present invention is not limited to the values of the parameters indicated in the above examples, since other specific values of these parameters are possible, which can be determined empirically based on factors such as, for example, the scope of the proposed method and the computing/hardware/other capabilities of the device intended to implement the proposed method. As a result of step S215, the decoder of the 3D feature extraction portion outputs refined 3D feature maps (in a non-limiting example, the number of such output maps is three) of the following forms:

,

,

.

,

,

.

. В одном примере

, чтобы не увеличивать занимаемую память в процессе вычислений в части для извлечения трехмерных признаков. Уточненные трехмерные карты признаков показаны на фиг.4 как "3D признаки".The specific meaning will be illustrated next.

. In one example

, so as not to increase the occupied memory in the process of computing in the part for extracting three-dimensional features. The refined 3D feature maps are shown in FIG . 4 as "3D features".

[0045] As described above with reference tofigure 3 andfigure 4, the main object detection neural network used in the 3D object detection method according to the second embodiment of the present invention also (similarly to the first embodiment) builds a 3D voxel space representation. Therefore, it can also use an intermediate part or a head part from any of the known methods for detecting three-dimensional objects based on a point cloud. Therefore, instead of a lengthy implementation of custom and complex architecture, modern intermediates and/or heads can be used with appropriate methods and without further modifications. Thus, the modular structure of the object detection neural network used in the 3D object detection method according to the second embodiment of the present invention provides, similarly to the first embodiment of the present invention, increased flexibility and simplifies the development process. However, it should be borne in mind that head designs for applications in outdoor scenes and in indoor scenes differ significantly.

[0046] Finally, after extracting the refined 3D feature maps, the method according to the second embodiment proceeds to step S220 of performing 3D object detection. This step is implemented by passing refined 3D feature maps through the indoor scene object detection part, which uses a dense voxel representation of intermediate features and contains 3D convolutional layers for classification, centering, and location. All current methods for detecting 3D objects in indoor scenes perform deep Hough voting to represent sparse point clouds. In contrast, the method according to the second embodiment follows work (1) and uses a dense voxel representation of intermediate features. To the knowledge of the authors, there is no part for detecting objects in room scenes for detecting 3D objects that is dense, 3D, and multi-scale.

[0047] The design of the part for detecting objects in indoor scenes may follow, for example, the 2D FCOS detection method described in (5) - Z. Tian, C. Shen, H. Chen, and T. He. FCOS: Fully convolutional one-stage object detection, 2019. The original FCOS head described in (5) takes 2D features from FPN and evaluates 2D bounding boxes using 2D convolutional layers. In order to adapt the FCOS for 3D detection, the 2D convolutions of the original FCOS are replaced in the present invention with 3D convolutions so that the part for detecting objects in indoor scenes can process 3D input data. According to FCOS, a central sample can be used to select the locations of potential objects. In a non-limiting example, 9 (3×3) candidates may be selected. The number of candidates may be more or less than specified. Because the proposed method operates in 3D space, the limit of candidate locations for each object can be set to a predetermined number (eg, 27.3×3×3). The resulting part for detecting objects in indoor scenes contains three 3D convolutional layers for classification, location, and centering, respectively, with common weights for all object scales.

,

, и

.[0048] Input to the part for detecting objects in indoor scenes . Multiscale input data to the part for detecting objects in room scenes can contain, as already briefly noted above, three shape tensors

,

, and

.

[0049] The output of the part for detecting objects in indoor scenes . For each 3D location ( x ^a ; y ^a ; z ^a ) and each of the three axes, the head part evaluates class probability p , centeredness c, and a 3D bounding box, such as the following seven-tuple:

,

,

,

,

,

,

,

,

,

, и.

,

, and .

,

,

,

,

,

, соответственно, обозначают минимальные и максимальные координаты вдоль соответствующих осей истинной ограничивающей рамки.[0050] While

,

,

,

,

,

, respectively, denote the minimum and maximum coordinates along the respective axes of the true bounding box.

[0051] Losses . The loss function described in document (5) is modified and used to train the part for detecting objects in indoor scenes used in the second embodiment of the method proposed in this document. The modified total loss function includes the focal loss for classification L _cls , the cross entropy loss for centering L _cntr , and the loss of the degree of intersection between two bounding boxes (Intersection-over-Union , IoU) for location L _loc . Since the method according to the second embodiment (for operation in indoor scenes) solves a 3D detection problem instead of a 2D detection problem, the 2D IoU loss is replaced by a rotated 3D IoU loss. In addition, true centering is updated with the third dimension. The resulting total loss for room scenes can be formulated, for example, as follows:

,

,

[0052] where n _pos is the number of positive 3D locations.

[0053] An embodiment of the method described with reference to FIGS. 3 and 4 is adapted for use in indoor scenes and can be performed by an electronic device equipped with at least one camera to detect zero or more 3D objects in the surrounding area of the electronic device . The electronic device in the second embodiment of the method may be one of, for example, a smartphone, tablet, smart glasses or a mobile robot. If the electronic device is a smartphone, tablet or smart glasses, users can point the cameras of such devices in different directions to detect objects that fall within the field of view of these cameras. In this case, such devices can, for example, serve as devices that help visually impaired users to move around the room and find various objects, for example, through synthesized voice prompts giving information about zero or more detected objects or information about their location.

[0054] If the electronic device is a mobile robot, such as a mobile cleaning robot, the electronic device may be the robot itself or at least a part of it. Such an electronic device may include at least one or more cameras mounted on the body of the robot and configured to capture images of the environment around the robot, and a processor configured to perform or initiate various operations associated with the robot. It should be understood that the proposed invention does not claim to be a technical solution for robot navigation and control of the movements of the robot as such, but it may be part of such a technical solution for a robot responsible for detecting three-dimensional objects.

[0055] A method according to a modification of the second embodiment (not shown in the figures) is adapted for understanding a scene, including estimating a camera position and determining a room configuration. In the method according to this modification of the second embodiment, the main object detection neural network further comprises a pre-trained part for understanding the scene. In some room benchmarking criteria, the 3D object detection problem can be formulated as a sub-problem of scene understanding. Following (6) Y. Nie, X. Han, S. Guo, Y. Zheng, J. Chang and JJ Zhang. Total 3D understanding: Joint layout, object pose and mesh reconstruction for indoor scenes from a single image, 2020, camera rotations and indoor configurations can be predicted in the present modification of the second embodiment of the method. The scene comprehension portion in this modification of the second embodiment of the method is trained and configured to jointly estimate the camera rotation R _t and the 3D scene configuration. This part for understanding the scene consists of two parallel branches: two fully connected layers infer the three-dimensional configuration of the scene, and two other fully connected layers evaluate camera rotations.

[0056] Input to the scene understanding part . The input is a single tensor of shape 8 _{c 0} , obtained by global average pooling of the part output to extract the 2D features.

[0057] Output from the Scene Understanding Part . The scene comprehension part outputs the camera position, for example, as a tuple of pitch at and roll , z , and a 3D configuration box (which is a box that approximates the room), for example, as a 7-tuple of ( x , y , z , w , l , h , and ). The yaw angle and offset can be set to zero.

используется по существу аналогично L_loc в первом варианте осуществления способа. В целом, общие потери для обучения части для понимания сцены можно сформулировать, например, следующим образом:[0058] Losses. The losses used in (6) are modified to bring them in line with the subtask that must be solved in this modification of the second variant of the method. Accordingly, the loss of L _layout configuration as a rotated loss of 3D IoU between the frames of the predicted and true configuration; this is the same loss that is used for the part for detecting objects in indoor scenes according to the second embodiment of the method described above. To evaluate camera rotation,

is used essentially in the same way as L _loc in the first embodiment of the method. In general, the total loss for learning the scene understanding part can be formulated, for example, as follows:

,

,

[0059] where l _layout =0.1 and l _pose =1.0, but not limited to the specified values.

[0060] Similar to the second embodiment of the method, a modification of the second embodiment of the method described above is adapted for understanding indoor scenes and can be performed by an electronic device equipped with at least one camera to detect zero or more three-dimensional objects in the surrounding area of the electronic device, evaluate position of the camera and determine the configuration of the room. The electronic device in the second embodiment of the method may be one of, for example, a smartphone, tablet, smart glasses or a mobile robot. If the electronic device is a smartphone, tablet, or smart glasses, users can point the cameras of such devices in different directions in the room to understand the characteristics of the scene and the content of the scene that falls into the field of view of the cameras. In this case, such devices can, for example, serve as devices that help visually impaired users to move around the room and find various objects, for example, using synthesized voice prompts that provide information about the scene.

[0061] If the electronic device is a mobile robot, such as a mobile cleaning robot, the electronic device may be the robot itself or at least a part of it. Such an electronic device may include at least one or more cameras mounted on the body of the robot and configured to capture images of the scene around the robot, and a processor configured to perform or initiate various operations associated with the robot. It should be understood that the invention disclosed in this application does not claim to be a technical solution for robot navigation and control of robot movements per se, but it may be part of such a technical solution for a robot responsible for understanding the scene.

[0062] In addition, a third embodiment of the method is provided, in which the method is adapted to automatically select the pontic and/or head depending on operating conditions or tasks to be solved. As an example, if the speed of the electronic device implementing the method exceeds a predetermined threshold (for example, 4 km per hour), or the GPS coordinates of the electronic device implementing the method indicate that the current position of the electronic device is outdoors, and/or the corresponding function is activated on the electronic device (for example, the object detection function in the camera settings), the method can switch to the processing branch and architecture described above with reference to Fig.1 and Fig.2 .

[0063] In another example, if the speed of the electronic device implementing the method is below a predetermined threshold (for example, 4 km per hour), or the GPS coordinates of the electronic device implementing the method indicate that the current position of the electronic device is indoors, and /or the corresponding function is activated on the electronic device (for example, the object detection function in the camera settings), the method can switch to the processing branch and architecture described above with reference to figures 2 and 3. Other exemplary implementations of such automatic switching will be clear to a person skilled in this areas based on the examples above.

[0064] Next, datasets that can be used to train and evaluate the proposed object detection neural network according to the first method embodiment (in street scenes), the second method embodiment (in indoor scenes), the modification of the second method embodiment (scene understanding) will be described. or a third variant of the method (capable of automatically switching between processing branches and architectures or parts thereof, depending on operating conditions or tasks).

[0065] In the present invention, four datasets are used to train neural networks or evaluate neural networks: ScanNet, SUN RGB-D, KITTI, and nuScenes. SUN RGB-D and KITTI are focused on monocular mode, and for ScanNet and nuScenes the problem of detection is solved in multiview mode.

[0066] KITTI . The KITTI Object Detection Dataset is the most significant benchmark for street scene applications for detecting monocular 3D objects. It consists of 3711 training, 3768 verification and 7518 test images. All markup for 3D objects in this dataset has some level of difficulty: easy, medium, and hard. The proposed method for detecting three-dimensional objects was evaluated based on the results on moderate objects from the test sample. The proposed method was evaluated on objects of the category of vehicles.

[0067] nuScenes. The nuScenes dataset provides data for the development of algorithms for solving problems related to autonomous driving. It contains LiDAR point clouds, RGB images captured by six cameras, accompanied by IMU and GPS measurements. This dataset covers 1,000 video sequences, each recorded for 20 seconds, with a total of 1.4 million images and 390,000 point clouds. The training group covers 28130 scenes and the test group contains 6019 scenes. The markup contains 1.4 million objects divided into 23 categories. The 3D detection accuracy was measured on vehicle categories. This test calculates not only the Average Accuracy (AP) metric, but also the Average Linear Error (ATE), Average Scale Error (ASE), and Average Orientation Error (AOE).

[0068] SUN RGB-D . The SUN RGB-D dataset is one of the first and best known 3D indoor scene datasets. It contains 10,335 images taken at various indoor locations, as well as corresponding depth maps from four different sensors and camera positions. The training group consists of 5285 frames, and the remaining 5050 frames make up the test subset. The markup contains 58657 objects. For each frame, a room configuration is provided.

[0069] ScanNet . The ScanNet dataset contains 1513 scans covering over 700 unique indoor scenes, of which 1201 scans are for the training set and 312 scans are used for validation. Overall, this dataset contains over 2.5 million images with associated depth maps and camera positions, as well as reconstructed point clouds with 3D semantic labeling. 3D bounding boxes are evaluated from semantic point clouds according to a standard protocol. The resulting bounding boxes of the object are aligned along the axes, so the angle of rotation was not predicted for ScanNet either.

[0070] Implementation details (the following information is not intended to be limiting, but only for the sufficiency of the disclosure ). Three-dimensional volume . As a feature extractor, that is, a 2D feature extractor part, ResNet-50 can be used. Accordingly, the number of convolutions in the first convolution block c ₀ can be set to 256. The size c ₁ of the 3D volume features can be preset to 64. For datasets with indoor scenes, the size c _{2 of} the output features can be set to c ₁ so as not to increase the amount of memory used during refinement. For datasets with street scenes, c ₂ can be set to 256.

[0071] Indoor scenes and outdoor scenes have different absolute scales. Therefore, the spatial dimensions of the volume of features for each data set can be set taking into account the subject area. Thus, for example, the voxel size s can be set to 0.32 meters for datasets with street scenes. The minimum and maximum values for all three axes for the street scene datasets also correspond to the point cloud ranges for the vehicle class. For the selection of room scene dataset constraints, the size of the room can be considered as 6.4×6.4×256 meters. To improve memory efficiency, the voxel size s can be doubled from 0.04 to 0.08.

[0072] Training . During training, L _indoor / L _extra is optimized for datasets with indoor scenes, and L _outdoor is optimized for datasets with outdoor scenes. An Adam optimizer can be used with an initial learning rate set to 0.0001 and a weight reduction of 0.0001. This implementation can be based on the MMDetection framework and uses the default learning settings. The network can be trained in 12 epochs, and the learning rate can be reduced by a factor of ten after the 8th and 11th epochs. For ScanNet, SUN RGB-D, and KITTI, each scene can be shown with networks three times in each training epoch. For training, distribution of one scene (multiview scenario) or four images (monocular scenario), 8 Nvidia Tesla P40 GPUs per GPU were used. Horizontal mapping can be arbitrarily applied during training, and the size of the input data in monocular experiments can be changed by no more than 25% of their original resolution. In addition, in indoor scenes, 3D voxel representations can be expanded similarly to point cloud based methods, so the center of the voxel grid can be shifted by no more than 1 m along each axis.

[0073] Conclusion . During the inference stage, the output can be filtered using the Rotated NMS algorithm, which is applied to projections of objects on the ground plane.

[0074] Other implementations and details . The disclosed method can be performed/implemented on any user/electronic device (eg, smartphone, tablet, navigation system, on-board equipment, laptop, smart watch, etc.). Such user/electronic devices or any other computing devices may be provided with computing means, including, without limitation, a processor, a neural processing unit (NPU), a graphics processing unit (GPU). Without limitation, the computing means may be implemented as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a system on a chip (SoC). Such user devices, or any other user computing devices, may also include memory (RAM, ROM), a (touch) screen, input/output media, a camera, communication media, and so on.

[0075] The proposed solution can also be implemented on a computer-readable medium that stores computer-executable instructions that, when executed by processing or computing means of the device, cause the device to perform any step (s) of the proposed technical solution. Any type of data is processed by intelligent systems trained using the methods described above. The learning phase can be performed online or offline. The trained neural networks (in the form of weights and other parameters/computer executable instructions) can be transferred to the user device and stored there for use during the inference stage (in use).

[0076] At least one of the many parts/modules may be implemented using an artificial intelligence (AI) model. The AI related function can be executed via non-volatile memory, volatile memory and processor. The processor may include one or more processors. Here, the one or more processors may be a general purpose processor such as a central processing unit (CPU), an application processor (AP) or the like, a graphics-only processing unit such as a graphics processing unit (GPU), an image processing unit (VPU), and /or a dedicated AI processor such as a Neural Processing Unit (NPU).

[0077] One or more processors control the processing of input data in accordance with a predetermined operating rule or artificial intelligence (AI) model stored in nonvolatile memory and nonvolatile memory. A predetermined operating rule or artificial intelligence model is provided through education or training. In this context, provision by training means that by applying a training algorithm to a set of training data, a predetermined operating rule or AI model with a desired performance is created. The training may be performed on the same device on which the AI is running according to the embodiment and/or may be implemented via a separate server/system.

[0078] An AI model may comprise multiple neural network layers. Each layer has a set of weights and performs a layer operation by computing the previous layer and operating on the set of weights. Examples of neural networks include, but are not limited to, Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network ( BRDNN), Generative Adversarial Networks (GANs), and Deep Q-Nets.

[0079] The learning algorithm is a method for training a predetermined target device using a plurality of training data to induce, allow, or instruct the target device to perform a determination or prediction. Examples of learning algorithms include, without limitation, supervised learning, unsupervised learning, partially supervised learning, or reinforcement learning.

[0080] The present invention may be implemented in mobile robot/vehicle navigation systems or as mobile applications for scene understanding and object recognition. The disclosed invention is intended for scene analysis and object recognition. The results obtained can be used in a wide range of tasks where decisions are made based on the scene and its objects. For example, the proposed method, when implemented as software, can provide mobile robotic device systems and vehicle navigation systems with spatial information for trajectory planning, capturing, and manipulating objects. In addition, in a mobile application, the proposed solution can be used to automatically generate scene hints for the user.

[0081] The disclosed invention is for detecting and recognizing three-dimensional objects and estimating their spatial position. The formulation of the problem follows the classical formulation of the problem of detecting three-dimensional objects, formulated by the community of scientists involved in computer vision.

[0082] The proposed technical solution can be implemented as a software solution in mobile robotic devices/vehicle navigation systems or run on smartphones as part of a mobile application. To implement the proposed technical solution, the device in question must meet the technical requirements, namely, have a sufficient amount of RAM and computing resources. Resource requirements vary by device function, camera settings, and performance requirements.

[0083] It should be understood that not all of the technical effects mentioned herein need to be used in any and every embodiment of the present technology. For example, embodiments may be implemented without the user using some of these technical effects, while other embodiments may be implemented with the user using other technical effects or none at all.

[0084] Modifications and improvements to the above-described implementations of the present technology may be apparent to those skilled in the art. The above description is intended to be illustrative rather than limiting. Therefore, the scope of the present technology is limited solely by the scope of the appended claims.

[0085] Although the implementations disclosed above have been described and shown with reference to specific steps performed in a specific order, it should be understood that these steps may be combined, subdivided, or performed in another order without departing from the scope of the present invention. Accordingly, the order and grouping of steps is not a limitation of the present technology. The use of the singular form with respect to any element disclosed in this application does not preclude that there may be two or more such elements in an actual implementation, and vice versa.

Claims (40)

1. A method for detecting three-dimensional objects using a neural network for detecting objects, containing a part for extracting two-dimensional features, a part for extracting three-dimensional features, a part for detecting objects in street scenes, which are pre-trained in an end-to-end method based on only monocular images captured at known positions, including includes steps in which: receive (S100) one or more monocular images; extracting (S105) 2D feature maps from each of the one or more monocular images by passing the one or more monocular images through the 2D feature extraction portion; creating (S110) an average 3D voxel volume based on the 2D feature maps; extracting (S115) a 2D representation of the 3D feature maps from the averaged 3D voxel volume by passing the averaged 3D voxel volume through a 3D feature extraction portion encoder, and performing (S120) detection of three-dimensional objects as detection of two-dimensional objects in a bird's-eye view (BEV) plane, and detection of two-dimensional objects in the BEV plane is realized by passing a two-dimensional representation of three-dimensional feature maps through a part for detecting objects in street scenes containing parallel 2D convolutional layers for classification and location. 2. A method according to claim 1, characterized in that the method is suitable for use in street scenes and is performed by a vehicle electronic device equipped with at least one camera for detecting zero or more three-dimensional objects in the surrounding area of the vehicle. 3. The method of claim 1, further comprising aggregating features in two-dimensional feature maps corresponding to the image for each image from one or more monocular images via a Feature Pyramid Network (FPN). 4. The method of claim 1, wherein the step of generating (S110) the averaged 3D voxel volume further performs the following steps: for each image from one or more monocular images create a three-dimensional voxel volume corresponding to the image using a camera model with a pinhole, which determines the correspondence between two-dimensional coordinates in the respective two-dimensional feature maps and three-dimensional coordinates in the three-dimensional voxel volume; for each 3D voxel volume, a binary mask corresponding to the 3D voxel volume is determined, the binary mask indicating, for each voxel in the 3D voxel volume, whether the voxel is within the frustum of the camera of the corresponding image; for each 3D voxel volume corresponding to the binary mask and corresponding 2D feature maps, projecting features of the 2D feature maps for each internal voxel onto the 3D voxel volume as determined by the binary mask; generating an aggregated binary mask by aggregating the binary masks defined for the 3D voxel volumes of all of the one or more monocular images; and generating an average 3D voxel volume by averaging the features projected into the 3D voxel volumes of all of the one or more monocular images for each internal voxel in the average 3D voxel volume as determined by the aggregated binary mask. 5. The method of claim 1, wherein the object detection neural network is trained on one or more street scene training datasets by optimizing the overall loss function for the street scenes based on at least location loss, focal loss for classification, and cross entropy. direction loss. 6. A method for detecting three-dimensional objects using a neural network for detecting objects, containing a part for extracting two-dimensional features, a part for extracting three-dimensional features, a part for detecting objects in scenes of rooms that are pre-trained in an end-to-end method based on only monocular images captured at known positions, containing stages where: receive (S200) one or more monocular images; extracting (S205) 2D feature maps from each of the one or more monocular images by passing the one or more monocular images through the 2D feature extraction portion; creating (S210) an average 3D voxel volume based on the 2D feature maps; extracting (S215) refined 3D object maps from the averaged 3D voxel volume by passing the averaged 3D voxel volume through the 3D object extraction part, and performing (S220) 3D object detection by passing the refined 3D feature maps through an indoor scene object detection portion using a dense voxel representation of intermediate features and containing 3D convolutional layers for classification, centering, and location. 7. The method according to claim 6, characterized in that this method is suitable for use in indoor scenes and is performed by an electronic device equipped with at least one camera for detecting zero or more three-dimensional objects in the surrounding area of this electronic device, and the electronic device represents one of the smartphone, tablet, smart glasses, mobile robot. 8. The method of claim 6, further comprising, for each image from one or more monocular feature images, features are aggregated in 2D feature maps corresponding to the image via a Feature Pyramid Network (FPN). 9. The method of claim 6, wherein the step of generating (S210) the averaged 3D voxel volume further performs the following steps: for each image from one or more monocular images create a three-dimensional voxel volume corresponding to the image using a camera model with a pinhole, which determines the correspondence between two-dimensional coordinates in the respective two-dimensional feature maps and three-dimensional coordinates in the three-dimensional voxel volume; for each 3D voxel volume, a binary mask corresponding to the 3D voxel volume is determined, the binary mask indicating, for each voxel in the 3D voxel volume, whether the voxel is within the frustum of the camera of the corresponding image; for each 3D voxel volume corresponding to the binary mask and corresponding 2D feature maps, projecting features of the 2D feature maps for each internal voxel onto the 3D voxel volume as determined by the binary mask; generating an aggregated binary mask by aggregating the binary masks defined for the 3D voxel volumes of all of the one or more monocular images; and generating an average 3D voxel volume by averaging the features projected into the 3D voxel volumes of all of the one or more monocular images for each internal voxel in the average 3D voxel volume as determined by the aggregated binary mask. 10. The method of claim 6, wherein the object detection neural network is trained on one or more indoor scene training datasets by optimizing the total loss function for the indoor scenes based on at least focal loss for classification, cross entropy loss for centrality. and loss of the metric of the degree of intersection between the two bounding boxes (IoU) for positioning. 11. The method of claim 6, wherein the object detection neural network further comprises a scene understanding portion that is pre-trained in an end-to-end manner, the method further comprising: performing a global average pooling of the extracted 2D feature maps to obtain a tensor representing the extracted 2D feature maps, and estimating a scene configuration by passing a tensor through a scene understanding portion configured to jointly estimate the camera rotation and the 3D scene configuration; the scene understanding part contains two parallel branches: two fully connected layers that output the 3D scene configuration, and two other fully connected layers that evaluate camera rotation. 12. The method according to claim 11, characterized in that the method is suitable for use in indoor scenes and is performed by an electronic device equipped with at least one camera for understanding the scene in the surrounding area of the electronic device, the electronic device being one of a smartphone, tablet, smart glasses, mobile robot. 13. The method of claim 11, wherein the scene understanding part of the object detection neural network is trained on one or more indoor scene training datasets by optimizing the overall scene understanding loss function based on at least a configuration loss and a camera rotation estimation loss. . 14. An electronic device mounted on a vehicle equipped with at least one camera, characterized in that this electronic device contains a processor configured to implement the method according to any one of claims 1 to 5 for detecting zero or more three-dimensional objects in the surrounding area vehicle. 15. An electronic device provided with at least one camera, characterized in that the electronic device comprises a processor configured to implement a method according to any one of claims 6-13 for detecting zero or more three-dimensional objects in the surrounding area of the electronic device. 16. The electronic device of claim 15, wherein the electronic device is a smartphone, tablet, smart glasses, or mobile robot. 17. A computer-readable medium that stores computer-executable instructions that, when executed by the processor, cause the processor to perform the method of any one of claims. 1-5 or 6-13.

Images