CN115169448A - A unified approach to 3D description generation and visual localization based on deep learning - Google Patents

Abstract

The invention discloses a three-dimensional description generation and visual positioning unified method based on deep learning, which uses a united frame to realize the three-dimensional description generation and the united information output of point cloud visual positioning in a complex scene; the combi-frame comprises: the system comprises a target detection module, a feature perception enhancement module, a description generation module and a visual positioning module. According to the method, through the joint training of the three-dimensional point cloud visual positioning and the description generation task, the model can fully learn the relation characteristics among objects from the visual positioning task and can also learn the fine-grained characteristics of the objects from the description generation task; the method comprises the following steps of enabling a scene needing three-dimensional point cloud visual positioning and description generation tasks to adopt a trained combined frame to realize description generation and visual positioning of objects in the scene; the method can be beneficial to the development of AR/VR industry, and the combined application of the AR/VR industry and the model can adapt to more practical application scenes, so that the life of people is facilitated.

Description

A unified approach to 3D description generation and visual localization based on deep learning

technical field

The invention relates to the fields of point cloud description generation, visual positioning and deep learning, in particular to a unified method for three-dimensional description generation and visual positioning based on deep learning.

Background technique

For indoor robot navigation technology, sensing the complex indoor environment is an essential step. Robots need to have the ability to understand the scene, find the object we need or the location to navigate from the complex environment. In AR/VR, generating a corresponding description for each object in the 3D model space is a very new technology, which has applications in the field of VR games, AR furniture selection, virtual navigation and other scenarios. To further improve the interaction ability between 3D scenes and natural language, researchers propose visual localization and description generation tasks.

The visual localization task refers to inputting a text description and a scene, so that the model can locate the location of the object described by the text in the scene. The description generation task refers to inputting a scene, which can Object, generate a corresponding text description. In the field of 3D point cloud visual localization and description generation, previous methods have implemented these two tasks in different networks.

Most of the current 3D point cloud visual localization and description generation methods consist of two stages. In the first stage, a 3D object detector or panoramic segmentation model is used to generate object proposals and candidate boxes from the input scene; in the second stage, the visual localization model uses the input text description to locate the object proposal that best matches it, and the description generates The model generates a corresponding description statement for each object suggestion. However, the existing 3D visual positioning technology does not take into account the relationship between different objects well, and the 3D description generation technology ignores the appearance representation information of the object itself.

Therefore, the current 3D point cloud visual positioning and description generation methods do not have joint applications and cannot adapt to more AR/VR practical application scenarios; and the previous methods use different networks, and can only achieve one task at a time. There are also flaws. For example, the patent document publication number: CN113657478A enhances the relationship between different objects through relational modeling, but the method in this document is strongly dependent on the visual localization problem itself and cannot be extended to description generation tasks.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a unified method of 3D description generation and visual positioning based on deep learning, which can solve the above deficiencies, and adopts a joint framework to realize the description generation and visual positioning of objects in the scene. , to achieve the effect that the two tasks help each other.

To achieve the above object, the technical scheme adopted in the present invention is:

The present invention provides a unified method for 3D description generation and visual positioning based on deep learning, using a joint framework to realize joint information output of 3D description generation and point cloud visual positioning in complex scenes; the joint framework includes: target detection module, feature perception enhancement module, description generation module and visual localization module; the method includes:

Obtain the relevant point cloud data and the corresponding text data of the preset scene, and divide it into a training set, a cross-validation set and a test set according to the preset ratio;

Input the relevant point cloud data in the training set into the target detection module, locate objects in the scene and generate initial object suggestions;

Inputting the initial object suggestion into the feature-aware enhancement module to generate a corresponding enhanced target suggestion;

The enhanced target suggestion is input into the description generation module and the visual positioning module respectively; the description generation module converts the enhanced target suggestion into text features, and generates a description sentence corresponding to each object suggestion; the visual positioning module will The corresponding text data and the enhanced target suggestion are fused to generate the position where the described object is located;

Iteratively train the joint framework, and use the cross-validation set and the test set for verification and testing;

Scenes that require 3D point cloud visual localization and description generation tasks will use the trained joint framework to achieve description generation and visual localization of objects in the scene.

Further, the target detection module adopts the VoteNet object detection module to predict the distance from the center point to each measurement, encode the point cloud, locate objects in the scene and generate initial object suggestions.

Further, the feature-aware enhancement module is composed of stacked two-layer multi-head self-attention layers, which include additional attribute encoding modules and relation encoding modules;

Both the attribute encoding module and the relation encoding module are composed of several fully connected layers; the attribute encoding module is used to encode the characteristics of the object itself, and the characteristics include: color, size and shape information; the relation encoding module It is used to encode the distance information between every two objects; attribute encoding and relation encoding constitute the object's enhanced object proposal.

Further, the description generation module uses a layer of multi-head cross-attention module to fuse the enhanced target suggestions of the target object and other target objects, and then uses a fully connected layer and a word prediction module to generate each text in the description sentence one by one. .

Further, the description generation module adopts the K nearest neighbor strategy, and selects the top K objects closest to the target object as other target objects other than the target object.

Further, the visual localization module consists of a layer of multi-head cross-attention, and finally uses a classifier to generate the confidence score suggested by each object, and uses the object with the highest predicted score as the final result.

Compared with the prior art, the present invention has the following beneficial effects:

Through the joint training of 3D point cloud visual localization and description generation tasks, the method enables the model not only to fully learn the relationship features between objects from the visual localization task, but also to learn the fine-grained features of the objects themselves from the description generation task. ;For scenes that require 3D point cloud visual positioning and description generation tasks, a joint framework after training is used to achieve description generation and visual positioning of objects in the scene; this method can help the development of AR/VR industry, the combination of the two The application can make the model adapt to more practical application scenarios, which is convenient for people's life.

Description of drawings

Figure 1 is a flow chart of a unified method for 3D description generation and visual positioning based on deep learning;

Fig. 2 is the joint framework structure and flow chart;

FIG. 3 is a structure and flow chart describing the generation module and the visual positioning module.

Detailed ways

In order to make the technical means, creative features, achievement goals and effects realized by the present invention easy to understand, the present invention will be further described below with reference to the specific embodiments.

In the description of the present invention, it should be noted that the terms "upper", "lower", "inner", "outer", "front end", "rear end", "two ends", "one end" and "the other end" The orientation or positional relationship indicated by etc. is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the indicated device or element must have a specific orientation, with a specific orientation. The orientation configuration and operation are therefore not to be construed as limitations of the present invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed to indicate or imply relative importance.

In the description of the present invention, it should be noted that, unless otherwise expressly specified and limited, the terms "installed", "provided with", "connected", etc. should be understood in a broad sense, for example, "connected" may be a fixed connection It can also be a detachable connection or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or an indirect connection through an intermediate medium, or the internal communication between the two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood in specific situations.

The 3D description generation task is more object-oriented, and it tends to learn more attribute information of the target object (that is, the object of interest) in the scene; while the 3D vision basic task is more relation-oriented, and it pays more attention to the relationship between objects. Based on this, the present invention provides a unified method for 3D description generation and visual positioning based on deep learning. Under a unified framework, a simple and powerful network structure is used to jointly solve these two (3D description generation and visual positioning). ) different but closely related tasks. By using a task-independent 3D object detector, i.e. object detection module, attribute and relation feature-aware enhancement module, and two lightweight task-specific modules (i.e. description generation module and visual localization module), the two tasks are combined train.

Referring to Figure 1, the present invention provides a unified method for deep learning-based three-dimensional description generation and visual positioning, including:

S10. Obtain relevant point cloud data and corresponding text data of the preset scene, and divide them into a training set, a cross-validation set, and a test set according to a preset ratio;

S20, input the relevant point cloud data in the training set into the target detection module, locate objects in the scene and generate initial object suggestions;

S30, inputting the initial object suggestion into the feature-aware enhancement module to generate a corresponding enhanced target suggestion;

S40. Input the enhanced target suggestion into a description generation module and a visual positioning module respectively; the description generation module converts the enhanced target suggestion into text features, and generates a description sentence corresponding to each object suggestion; the visual positioning The module fuses the corresponding text data and the enhanced target suggestion to generate the position where the described object is located;

S50, performing iterative training on the joint framework, and using the cross-validation set and the test set for verification and testing;

S60. In a scene that requires three-dimensional point cloud visual positioning and description generation tasks, the trained joint framework is used to realize description generation and visual positioning of objects in the scene.

Wherein, in step S10, the preset scene, for example, refers to an AR/VR scene, specifically including: VR game field, AR furniture selection, indoor virtual navigation and other fields, and other applications based on 3D scenes are acceptable. The point cloud data in the scene can be obtained through a laser scanner, and the corresponding text data can be obtained. Taking AR interior decoration preview as an example, it is necessary to obtain indoor point cloud data, indoor decorations, furniture, home appliances, etc. related text data, etc., For example, it is divided into training set, cross-validation set and test set according to 7:2:1.

In step S20, the point cloud data in the training set is input into the target detection module to detect the initial object suggestion: the target detection module uses the existing high-efficiency method VoteNet to cluster the point cloud, and then draws on the idea of FCOS to predict the The distance between the center point and each side of the target object to generate initial object proposals.

In step S30, the initial object proposal is input into a feature-aware enhancement module of attributes and relationships, and an enhanced target proposal is obtained: the feature-aware enhancement module is composed of stacked two-layer multi-head self-attention layers, wherein Contains additional attribute encoding modules and relational encoding modules, wherein the attribute encoding module and the relational encoding module are composed of several fully connected layers. The function of the attribute encoding module is to convert some characteristics of the object itself (such as color, size, shape and other information). ) to encode, and the relational encoding module encodes information such as the distance between each two objects.

In step S40, the enhanced target suggestion features obtained from the feature-aware enhancement module are converted into text features by the description generation module, and finally a description sentence corresponding to each object suggestion is generated. The description generation module uses a layer of multi-head cross-attention to fuse target object proposals with other local object proposals, and then uses a fully connected layer and word prediction module to generate each word in the description sentence one by one. or text.

Using the visual positioning module, the text data corresponding to the input obtained in step S10 and the enhanced target suggestion feature obtained from the feature perception enhancement module are fused to find the position of the described object. The visual localization module also consists of a layer of multi-head cross-attention, and finally uses a classifier to generate a confidence score for each object proposal, and uses the object with the highest predicted score as the final result.

Steps S50-S60 are to iteratively train the above joint framework, and use the cross-validation set and the test set for verification and testing; and then use the trained joint framework for scenes that require 3D point cloud visual positioning and description generation tasks. Realize description generation and visual positioning of objects in the scene.

The visual positioning technology allows the model to find the corresponding position of the object described in the language from the complex scene to assist the robot in navigation and positioning; the description generation task allows the model to generate a corresponding description for each object in the scene, helping the development of the AR/VR industry , the combined application of the two can make the model adapt to more practical application scenarios and facilitate people's lives.

In this example, the VoteNet object detection module is used, and the point cloud is encoded using an improved bounding box modeling method to more accurately locate objects and generate initial object proposals. Then, the proposed features are augmented by a task-agnostic feature-aware augmentation module to generate enhanced object proposals. Finally, the augmented object proposals are input into the description generation module and visual localization module of dense description generation task and visual localization task, respectively, and the final results are generated for each task.

Through the joint training of 3D point cloud visual localization and description generation tasks, the method enables the model not only to fully learn the relationship features between objects from the visual localization task, but also to learn the fine-grained features of the objects themselves from the description generation task. ; The scene that requires 3D point cloud visual localization and description generation task, adopts the joint framework after training to realize the description generation and visual localization of objects in the scene.

Below in conjunction with the accompanying drawings, the lower joint framework is specifically described:

As shown in Fig. 2(a), the joint framework consists of three modules: 1) an object detection module; 2) a feature-aware enhancement module for attributes and relationships; and 3) a task-specific description generation module and a visual localization module. Both the object detection module and the feature-aware enhancement module are task-agnostic modules because they have no specific association with subsequent tasks and can be shared by the two tasks. The description generation module and the visual localization module are task-specific lightweight Transformer-based network structures for description generation and visual localization tasks, respectively.

The 3D description generation task enables the model to learn more sufficient fine-grained features, and the 3D visual localization task enables the model to learn more sufficient relational features. Through the joint training strategy, the two tasks can help each other. The description generation task can help the visual localization task learn more of its own attributes (size, color, shape, etc.), and the visual localization task can help the description generation task learn. to more complex relationship information between objects.

As shown in Fig. 2(b), the feature-aware enhancement module for attributes and relations consists of a stacked two-layer multi-head self-attention layer, which contains additional attribute encoding modules and relation encoding modules , among them, the attribute encoding module and the relation encoding module are composed of several fully connected layers. The function of the attribute encoding module is to encode some characteristics of the object itself (such as color, size and other information), while the relation encoding module is to encode Information such as the distance between each two objects is encoded.

As shown in Fig. 2(c), the description generation module uses a layer of multi-head cross-attention module to fuse target object proposals and local other object proposals. Each word in the description sentence is then generated one by one using a fully connected layer and a word prediction module.

As shown in Fig. 2(d), the visual localization module is also composed of a layer of multi-head cross-attention, and finally a classifier is used to generate the confidence score of each object proposal, and the predicted score The tallest object is the final result.

Specifically, as shown in (a) and (b) in Figure 3, for the description generation module, for the Key and Value input by the cross attention module, by using the K nearest neighbor strategy, according to its center distance in the three-dimensional coordinate space, The top K objects closest to the target object are selected, thereby filtering out the less relevant objects in the scene. The features proposed by the selected objects are used as the Key and Value of the multi-head cross-attention module. In a specific implementation, for example, k is set to 20 according to experience. This strategy is specially designed for description generation tasks, as it mainly focuses on the most obvious (or dominant) relationships between the target object and its surrounding objects, and the rest of the relational information may be less important for description generation tasks.

Among them, the above concepts of query, key & value come from the recommendation system. The basic principle is: given a query, calculate the correlation between the query and the key, and then find the most suitable value according to the correlation between the query and the key. For example: in AR interior decoration recommendation. The query is a person's preference information for decoration (such as decoration style, age, gender, etc.), the key is the type of decoration (European style, Chinese style, etc.), and the value is the decoration suggestion to be recommended. In this example, although each attribute of query, key and value is in a different space, they actually have a certain potential relationship, that is to say, through some transformation, the attributes of the three can be made in a similar space .

As shown in Figure 3(c): For the visual localization module, the input Key and Value of the cross-attention module are generated based on the input text language description. Specifically, a pre-trained GloVE (Global Vectors for Word Representation) model and a Gated Recurrent Unit (GRU, Gated Recurrent Unit) model can be used to extract text features. The word features output by the GRU form Key and Value. In addition, GRU also generates a global language feature to predict the class of the object described by each sentence. The features proposed by the object are used as Query input. Finally, a basic classifier is used to generate a confidence score for each object proposal, and the object proposal with the highest predicted score is used as the final visual localization result.

The unified method for generating 3D descriptions and visual positioning based on deep learning provided by the present invention will be described through a specific embodiment:

1. Taking various scenarios (VR game field, AR furniture selection, virtual navigation) as an example, the point cloud and corresponding text data in various scenarios can be divided into training set and cross-validation according to the ratio of 7:2:1 set and test set.

2. Input the point cloud data P∈N×(3+K) in the training set into the target detection module, where the points in the N input point clouds not only contain 3D XYZ point cloud coordinate information, but also 1D object height information , 3-dimensional normal vector information and feature information obtained from 128-dimensional 2D semantic segmentation together form K=132-dimensional auxiliary attribute feature. By using PointNet++ to perform feature extraction on the point cloud, and then according to VoteNet, use the voting and grouping modules to cluster the point cloud, and get the object suggestion (object) center. Then, drawing on the idea of FCOS, the initial object proposal bounding box and 128-dimensional initial object proposal features are obtained by predicting the distance from the center point to each side of the object bounding box.

3. Input the initial object proposal into the feature-aware enhancement module of attributes and relationships, and obtain the enhanced object proposal. The following is the detailed structure and working principle of the feature-aware enhancement module:

In order to allow each object feature to learn its own characteristics more clearly and fully model the complex relationship between objects, the feature-aware enhancement module can be designed as a structure similar to Transformer-encoder, which is mainly composed of two layers of multi-head self-attention layers. , which contains the attribute encoding module and the relation encoding module. The input features Query, Key, and Value of the self-attention module are the features suggested by the object.

Attribute encoding module: In order to aggregate attribute features and initial object features, the relevant features of the bounding box, that is, the 27-dimensional bounding box and the center point coordinate features (3-dimensional XYZ coordinates of 8 bounding box points and 1 center point) are combined by using a fully connected layer. , the previously input 132-dimensional auxiliary attribute features are spliced, and then encoded into 128-dimensional attribute features. This attribute feature is added to the 128-dimensional initial object proposal feature to enhance the initial object proposal feature.

Relation encoding module: can encode the relative distance between any two object proposals to capture complex object relationships, not only encoding the Euclidean distance between any two centers of object proposals (i.e. distance Dist ∈ M × M ×1), and encodes three pairs of distances along the x, y, z directions between any two centers proposed by the object (i.e. distance [Dx, Dy, Dz] ∈ M × M × 3) to better capture different directions object relation on , where M is the number of initial object proposals. Then, it is spliced into a spatial proximity matrix (Dx, Dy, Dz, and Dist) and aggregated along the channel dimension, input into the fully connected layer for encoding, to generate a spatial relationship matrix, and the output dimension H is the same as the attention head in the multi-head attention module. The number of matches (in the implementation, say H = 4), and then the spatial relationship matrix is summed with the similarity matrix (the so-called attention map) generated by each head of the multi-head self-attention module.

4. Input the enhanced object suggestion into the description generation module to generate the corresponding description. The following is the detailed structure and working principle of the description generation module.

The description generation head mainly consists of one cross-attention layer. First, you need to select the object proposals that need to generate description sentences. In the testing phase, you can use all the object proposals in the scene (after the non-maximization suppression (NMS) process) as input one by one, and then use the recurrent network structure to gradually generate the description generated. each word. Then for each object, the hidden features output by the multi-head cross-attention module and the word features of the previous word (the real ground-truth word in the training phase and the previous predicted word in the testing phase, here in order to avoid the training process with The big difference in the testing process is that an autoregressive strategy is used during training, specifically, 10% of the true words are replaced with predicted words) and the current object suggestion features are fused as the Query input by the cross-attention module. .

As shown in Figure 3, for the Key and Value input by the cross-attention module, the K nearest neighbor strategy is used, and the top K objects closest to the target object are selected according to their center distance in the three-dimensional coordinate space, thereby filtering out the scene objects that are less relevant. The features proposed by the selected objects are used as the Key and Value of the multi-head cross-attention module.

Finally, the multi-head cross-attention module is followed by a fully connected layer and a simple word prediction module to predict each word in the title in a one-by-one manner.

5. Input the enhanced object suggestion and the text data corresponding to the input in step 1 into the visual positioning module, and find the description corresponding to the object suggestion. The following is the detailed structure and working principle of the visual positioning module:

The 3D visual localization task is to locate objects of interest based on language descriptions, and the visual localization head mainly focuses on matching between a given language description and detected object proposals.

As shown in Fig. 3, the object proposals described in the localization language are located using a cross-attention module. The input Key and Value of the cross-attention module are generated using the input text language description. Specifically, a pre-trained GloVE (Global Vectors for Word Representation) model and a Gated Recurrent Unit (GRU, Gated Recurrent Unit) model can be used to extract text features. The word features of the output of the GRU form Key and Value. In addition, GRU also generates a global language feature to predict the class of the object described by each sentence. The features proposed by the object are used as Query input. Finally, a basic classifier is used to generate a confidence score for each object proposal, and the object proposal with the highest predicted score is used as the final visual localization result.

The following is the effect of the method (Ours) provided by the present invention in multiple data sets (ScanRefer, Scan2Cap). Compared with other methods, the method provided by the present invention has achieved the current best performance (bold):

Table 1 Visual localization results on ScanRefer dataset

Table 2 describes the generation results on the Scan2Cap dataset

The method provided by the present invention enables the model not only to fully learn the relationship features between objects from the visual positioning task, but also to learn the object itself from the description generation task through the joint training of the three-dimensional point cloud visual positioning and the description generation task. The fine-grained features of the scene; the scene that will require 3D point cloud visual positioning and description generation tasks, using a joint framework after training to achieve description generation and visual positioning of objects in the scene; this method can help the development of the AR/VR industry, which The combined application of the two can make the model adapt to more practical application scenarios and facilitate people's lives.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied therein, including but not limited to disk storage, optical storage, and the like.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flows of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the invention. Thus, provided that these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims (6)

1. A three-dimensional description generation and visual positioning unified method based on deep learning is characterized in that a joint framework is used for realizing three-dimensional description generation and joint information output of point cloud visual positioning in a complex scene; the joint frame includes: the system comprises a target detection module, a feature perception enhancement module, a description generation module and a visual positioning module; the method comprises the following steps:

acquiring related point cloud data and corresponding text data of a preset scene, and dividing the point cloud data and the corresponding text data into a training set, a cross validation set and a test set according to a preset proportion;

inputting the related point cloud data in the training set into the target detection module, positioning objects in a scene and generating an initial object suggestion;

inputting the initial object suggestion into the feature perception enhancement module to generate a corresponding enhancement target suggestion;

inputting the enhancement target suggestions into a description generation module and a visual positioning module respectively; the description generation module converts the enhanced target suggestions into text features and generates description sentences corresponding to the object suggestions; the visual positioning module fuses the corresponding text data and the enhancement target suggestion to generate the position of the described object;

performing iterative training on the combined framework, and performing verification and test by adopting the cross verification set and the test set;

and (3) adopting a trained combined frame to realize the description generation and the visual positioning of objects in the scene for the scene needing the three-dimensional point cloud visual positioning and description generation task.

2. The method of claim 1, wherein the target detection module employs a VoteNet object detection module, predicts distances from a center point to each measurement, encodes point clouds, locates objects in a scene, and generates an initial object suggestion.

3. The unified method for three-dimensional description generation and visual localization based on deep learning of claim 1, wherein the feature perception enhancement module is composed of two stacked multi-headed self-attention layers, which contain additional attribute coding module and relationship coding module;

the attribute coding module and the relation coding module are both composed of a plurality of full connection layers; the attribute coding module is used for coding the characteristics of the object, wherein the characteristics comprise: color, size and shape information; the relation coding module is used for coding the distance information between every two objects; the attribute codes and the relationship codes constitute enhanced target suggestions for the object.

4. The unified method for three-dimensional description generation and visual localization based on deep learning of claim 1, wherein the description generation module uses a layer-multi-head cross attention module to fuse the enhanced target suggestions of the target object and other target objects, and then uses a fully-connected layer and word prediction module to generate each text in the description sentence one by one.

5. The method as claimed in claim 4, wherein the description generation module selects the top K objects nearest to the target object of the object as other target objects except the target object by using a K nearest neighbor strategy.

6. The unified method for deep learning based three-dimensional description generation and visual localization as claimed in claim 1, wherein the visual localization module is composed of a layer of multi-head cross attention, and finally uses a classifier to generate confidence score of each object suggestion, and takes the object with highest predicted score as the final result.

Images