CN118350464A - Conversational target positioning method and device based on arbitrary granularity text input - Google Patents

Abstract

The invention relates to the technical field of artificial intelligence, and provides a dialogue type target positioning method and device based on text input with arbitrary granularity, wherein the method comprises the following steps: inputting the image to be positioned into a visual encoder to extract image features, and projecting the image features into a word embedding space to obtain image word vectors; word segmentation is carried out on the input text to obtain word segmentation vectors, and the word segmentation vectors are mapped to word embedding spaces to obtain text word vectors; taking the image word vector and the text word vector as image text word vector pairs, and inputting the image text word vector pairs into a large language model to obtain an answer sequence; the answer sequence comprises a target category and a target position in the image to be positioned; the large-scale language model is obtained by training based on a sample input text, a sample to-be-positioned image, a label target type in the sample to-be-positioned image and a label target position, and the large-scale language model has the capability of positioning text input with any granularity by utilizing extensive knowledge obtained by large-scale pre-training, so that the accuracy of the conversational target positioning method is improved.

Description

Conversational target positioning method and device based on arbitrary granularity text input

Technical Field

The present invention relates to the field of artificial intelligence technology, and in particular to a conversational target positioning method and device based on arbitrary granularity text input.

Background technique

With the development of command adjustment technology, large vision-language models (LVLMs) have shown superiority in various visual understanding tasks. These models are increasingly used in real-world scenarios, such as personal assistants in mobile devices. However, their application in specific areas that require fine-grained object perception and spatial localization is still limited. In addition, there are concerns about the occurrence of hallucinations due to the lack of fine-grained supervision.

On the one hand, this limitation mainly stems from the weak understanding of multiple objects by large image-text models. Although the current popular large image-text models, such as Shikra, Qwen-VL, and KOSMOS-2, have basic object perception capabilities, they are limited to locating a single existing object and are unable to cope with more complex scenarios. Despite integrating millions of data from the Referring Expression Comprehension (REC) task, or reformatting other task data to adapt to the instruction expression comprehension format, they only contain a single reference scenario. In essence, the root of the problem lies not in the amount of data, but in the lack of a comprehensive and orderly training benchmark.

On the other hand, some studies have attempted to call offline visual expert models through large language models, or integrate specific visual detection heads to achieve the localization of various fine-grained objects. The former is a scheduler-like approach that manages each visual expert model through a specific application programming interface (API). However, this approach is insufficient in fully utilizing the comprehensive analysis capabilities of large language models (LLMs), and the visual expert models themselves are also limited by insufficient generalization capabilities. The latter generates task-specific outputs through specialized head structures such as detection heads and segmentation heads. This allows for more customized responses, but the number of parameters increases significantly as the number of tasks increases. The increase in interfaces and task heads will burden large language models and cannot share a unified structure with large image and text models. In addition, both methods rely on external assistance, and neither fully utilizes the extensive knowledge acquired by large language models through large-scale pre-training, reducing the accuracy of conversational object localization methods.

Summary of the invention

The present invention provides a conversational target localization method and device based on arbitrary granularity text input, which are used to solve the defects of the two methods in the prior art that both methods rely on external assistance and do not fully utilize the extensive knowledge obtained by large language models through large-scale pre-training, thereby reducing the accuracy of the conversational target localization method.

The present invention provides a conversational target positioning method based on arbitrary granularity text input, comprising:

Get the image to be located and the input text;

Inputting the image to be located into a visual encoder to extract image features of the image to be located, and projecting the image features into a word embedding space to obtain an image word vector;

Segmenting the input text to obtain a segmentation vector, and mapping the segmentation vector to the word embedding space to obtain a text word vector;

The image word vector and the text word vector are used as an image-text word vector pair, and the image-text word vector pair is input into a large language model to obtain an answer sequence output by the large language model; the answer sequence includes the target category and target position in the image to be located;

The large language model is trained based on sample input text, sample images to be located, and label target categories and label target positions in the sample images to be located.

According to a conversational target localization method based on arbitrary granularity text input provided by the present invention, the image-text word vector pair is input into a large language model to obtain an answer sequence output by the large language model, and then the method further includes:

Based on the image to be located, the input text, and a preset answer sequence before word segmentation, determining each first conditional probability of each word segmentation in the answer sequence;

Based on the first conditional probabilities, a generation probability of the answer sequence is determined.

According to a conversational target localization method based on arbitrary granularity text input provided by the present invention, the image-text word vector pair is input into a large language model to obtain an answer sequence output by the large language model, and then the method further includes:

Determine, based on the image to be located, the input text, and a preset answer sequence before word segmentation, each second conditional probability of each target category in the image to be located in the answer sequence;

Based on the second conditional probabilities, a generation probability of each target category in the image to be located in the answer sequence is determined.

According to a conversational target localization method based on arbitrary granularity text input provided by the present invention, the generation probability of each target category in the image to be located in the answer sequence is determined based on each second conditional probability, and then the method further includes:

Determine each third conditional probability of the target position in the image to be located in the answer sequence based on the image to be located, the input text, and the answer sequence before the preset position;

Based on the third conditional probabilities, a generation probability of each target position in the image to be located in the answer sequence is determined.

According to a conversational target location method based on arbitrary granularity text input provided by the present invention, the generation probability of each target position in the image to be located in the answer sequence is determined based on each third conditional probability, and then the method further includes:

Based on the generation probability of each target position and the generation probability of each target category, a confidence score of each target in the image to be located is determined.

According to a conversational target localization method based on arbitrary granularity text input provided by the present invention, the training step of the large language model includes:

Determine an initial large language model, and obtain the sample input text, the sample image to be located, and the label target category and label target position in the sample image to be located;

Input the sample image to be located into the initial visual encoder to extract sample image features of the sample image to be located, and project the sample image features into the sample word embedding space based on the initial projection layer to obtain a sample image word vector;

Segmenting the sample input text to obtain a sample segmentation vector, and mapping the sample segmentation vector to the sample word embedding space to obtain a sample text word vector;

The sample image word vector and the sample text word vector are used as a sample image text word vector pair, and the sample image text word vector pair is input into an initial large language model to obtain a predicted answer sequence output by the initial large language model;

Based on the predicted target category and the label target category in the predicted answer sequence, as well as the predicted target position and the label target position in the predicted answer sequence, a target loss is determined, and parameters of the initial large language model are iterated based on the target loss to obtain the large language model.

According to a conversational target localization method based on arbitrary granularity text input provided by the present invention, the parameter iteration of the initial large language model based on the target loss to obtain the large language model further includes:

Performing gradient back propagation on the initial visual encoder and the initial projection layer based on the target loss to obtain the visual encoder and the projection layer.

The present invention also provides a conversational target positioning device based on arbitrary granularity text input, comprising:

An acquisition unit, used for acquiring an image to be located and input text;

A projection unit, configured to input the image to be located into a visual encoder to extract image features of the image to be located, and project the image features into a word embedding space to obtain an image word vector;

A mapping unit, configured to segment the input text to obtain a segmentation vector, and map the segmentation vector to the word embedding space to obtain a text word vector;

An input unit, used to use the image word vector and the text word vector as an image-text word vector pair, and input the image-text word vector pair into a large language model to obtain an answer sequence output by the large language model; the answer sequence includes the target category and target position in the image to be located;

The large language model is trained based on sample input text, sample images to be located, and label target categories and label target positions in the sample images to be located.

The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for conversational target localization based on text input of arbitrary granularity as described above is implemented.

The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described conversational target localization methods based on arbitrary granularity text input.

The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for conversational target localization based on text input of arbitrary granularity.

The method and device for conversational target localization based on arbitrary granularity text input provided by the present invention inputs the image to be located into the visual encoder to extract the image features of the image to be located, and projects the image features into the word embedding space to obtain the image word vector, segment the input text to obtain the word segmentation vector, and map the word segmentation vector into the word embedding space to obtain the text word vector. Finally, the image word vector and the text word vector are used as an image-text word vector pair, and the image-text word vector pair is input into the large language model to obtain the answer sequence output by the large language model, and the answer sequence includes the target category and target position in the image to be located. The large language model is trained based on sample input text, sample image to be located, and the label target category and label target position in the sample image to be located. On the one hand, it does not rely on external assistance, and the sample input text and the sample image to be located have a unified structure with the large picture-text model; on the other hand, it makes full use of the extensive knowledge obtained by the large language model through large-scale pre-training, so as to have the ability to locate text input of arbitrary granularity, and further improves the accuracy and reliability of the conversational target localization method.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a flow chart of a method for conversational target location based on arbitrary granularity text input provided by the present invention;

FIG2 is a second flow chart of the method for conversational target location based on arbitrary granularity text input provided by the present invention;

FIG3 is a schematic diagram of a flow chart of a progressive two-stage training method provided by the present invention;

FIG4 is a schematic diagram of the structure of a conversational target location device based on arbitrary granularity text input provided by the present invention;

FIG. 5 is a schematic diagram of the structure of an electronic device provided by the present invention.

Detailed ways

In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

The terms "first", "second", etc. in the specification and claims of the present invention are used to distinguish similar objects, rather than to describe a specific order or precedence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type.

The present invention provides a method for conversational target location based on arbitrary granularity text input. FIG. 1 is a flow chart of the method for conversational target location based on arbitrary granularity text input provided by the present invention. FIG. 2 is a flow chart of the method for conversational target location based on arbitrary granularity text input provided by the present invention. As shown in FIG. 1 and FIG. 2, the method includes:

Step 110: Obtain the image to be located and the input text.

Specifically, an image to be located and an input text may be obtained, where the image to be located is an image that needs to be subsequently located, and the image to be located may be represented by Xv. The image to be located may be pre-collected by an image acquisition device, may be captured in real time, or may be downloaded or scanned from the Internet, which is not specifically limited in the embodiment of the present invention.

Furthermore, after the image to be positioned is acquired, a preprocessing operation may be performed on the image to be positioned. The preprocessing operation includes but is not limited to normalization processing, sharpening processing, denoising processing, etc., which is not specifically limited in the embodiment of the present invention.

The input text can be represented by _Xins , and the input text can be an instruction or a question. The input text can be directly input by the user, or can be obtained by transcribing the collected audio, or can be obtained by collecting images through image collection devices such as scanners, mobile phones, cameras, tablets, and performing OCR (Optical Character Recognition) on the images. The embodiments of the present invention do not specifically limit this.

The input text can be "Please locate the person wearing a red cotton coat in the picture", or "Please detect all the cars in the picture", or "Please locate the person wearing a red cotton coat in the picture", "Please detect all the cars in the picture", "Is there an elephant in the picture?".

Step 120, inputting the image to be located into a visual encoder to extract image features of the image to be located, and projecting the image features into a word embedding space to obtain an image word vector;

Step 130, segment the input text to obtain a segmentation vector, map the segmentation vector to the word embedding space, and obtain a text word vector.

Specifically, after obtaining the image to be located and the input text, the image to be located can be input into the visual encoder to extract the image features of the image to be located. Here, the image features of the image to be located can be represented by Z _v . After the image features are extracted, the image features can be projected into the word embedding space to obtain the image word vector. The image word vector can be represented by H _v .

It should be noted that the dimension of the image word vector H _v is the same as that of the word embedding space.

Word embedding space is a technique in the field of Natural Language Processing (NLP) that aims to map semantics to geometric space. This technique associates a digital vector with each word in the dictionary so that the distance between any two vectors can reflect the partial semantic relationship between the two associated words. The geometric space formed by these vectors is called the embedding space. The word embedding space has certain structured semantic information, in which semantically similar words or phrases are embedded in close vectors, while semantically dissimilar words or phrases are embedded in vectors that are farther away.

It should be noted that the framework of the target positioning method of the embodiment of the present invention can be constructed based on a graphic model. The graphic model can be an LLaVA (Large Language and Vision Assistant) graphic model or an InstructBLIP (Instruct Bootstraping Language Image Pre-training) model, etc. The embodiment of the present invention does not specifically limit this.

The LlaVA graphic model consists of three modules, including a visual encoder, a projection layer and a large language model, wherein the visual encoder can be the ViT-L/14 visual encoder in the CLIP (Contrastive Language-Image Pre-training) model, the projection layer is an arbitrarily initialized linear layer, and the large language model can be a Llama2-13B model, which is not specifically limited in the embodiment of the present invention.

Among them, ViT-L/14 visual encoder: ViT (Vision Transformer) is a model for image classification that divides an image into a series of small blocks (called patches), which are then input into the Transformer architecture as a sequence. ViT-L/14 refers to a large version of ViT, where "L" stands for "Large" and "14" refers to the size of the image block (patch) in the Transformer encoder. This large model has more parameters and stronger representation capabilities, and can capture complex features in images.

Llama2-13B Large Language Model: Llama (Large Language Model Family) is a series of large language models. Llama2-13B refers to a specific model in this series, where "2" indicates that this is the second-generation model, and "13B" indicates that the model has 13 billion parameters. Models of this size have strong text generation and understanding capabilities and can handle complex natural language tasks.

Then, the input text is segmented to obtain a segmentation vector, and the segmentation vector is mapped to the word embedding space to obtain a text word vector, where the text word vector can be represented by _Hins .

Here, the input text is segmented. The input text can be input into a word segmenter for segmentation.

Step 140, using the image word vector and the text word vector as an image-text word vector pair, and inputting the image-text word vector pair into a large language model to obtain an answer sequence output by the large language model; the answer sequence includes the target category and target position in the image to be located;

The large language model is trained based on sample input text, sample images to be located, and label target categories and label target positions in the sample images to be located.

Specifically, after the image word vector and the text word vector are obtained, the image word vector and the text word vector can be used as an image-text word vector pair. Here, the image-text word vector pair can be represented by (H _v ,H _ins ).

After obtaining the image-text word vector pair, the image-text word vector pair can be input into a large language model to obtain an answer sequence output by the large language model, where the answer sequence can be represented by _Xa , and its sequence length is represented by L, where each output word segment is represented by _xa,j , and j represents the index of the word segment in the answer sequence.

Here, the answer sequence includes the target category and target position in the image to be located. The target category can be represented by R, and the target position is the candidate box representation of any object in the image to be located, that is, the coordinates of the upper left and lower right points, that is, [x1, y1, x2, y2].

For example, the answer sequence _Xa may be _Xa : person wearing red cotton clothes - [0.465, 0.785, 0.658, 0.998], or _Xa : car - [0.123, 0.325, 0.498, 0.567] & car - [0.321, 0.465, 0.785, 0.892], or _Xa : there is no elephant in the picture, etc. This embodiment of the present invention does not make any specific limitation to this.

It should be noted that if there are multiple targets in the image to be located, multiple targets in the answer sequence are connected with "&" to seamlessly adapt to the scene of multiple objects without introducing other special placeholders.

In addition, there are currently two main ways to represent the coordinates in the target position: one is to create a new word segmentation to represent each coordinate on the coordinate axis, and the other is to directly use numeric characters in natural language to represent each bit in the coordinate. Based on Shikra's comparative experiments on the REC (Recommendation Systems) task, the numerical method can achieve just the right performance, and the introduction of additional word segmentations outside the original word segmentation table requires training from scratch, resulting in a slower convergence speed. Therefore, the embodiment of the present invention uses numerical characters to represent the target position, and represents a single target in the answer sequence as a target category and textualized normalized coordinates (target position), and sets the accuracy of the normalized coordinates to 0.001. This accuracy strikes a balance between reducing the positioning error of small objects (pixel area less than 32×32) and sequence length.

Here, the large language model is trained based on sample input text, sample images to be located, label target categories and label target positions in the sample images to be located.

Specifically, the sample image to be located is input into the initial visual encoder to extract the sample image features of the sample image to be located, and the sample image features are projected into the sample word embedding space based on the initial projection layer to obtain the sample image word vector.

Then, the sample input text is segmented to obtain a sample segmentation vector, and the sample segmentation vector is mapped to the sample word embedding space to obtain the sample text word vector.

Furthermore, the sample image word vector and the sample text word vector are used as a sample image-text word vector pair, and the sample image-text word vector pair is input into an initial large language model to obtain a predicted answer sequence output by the initial large language model.

Finally, based on the difference between the predicted target category and the labeled target category in the predicted answer sequence, as well as the difference between the predicted target position and the labeled target position in the predicted answer sequence, the target loss is determined, and the parameters of the initial large language model are iterated based on the target loss, and the initial large language model after parameter iteration is used as the large language model.

It can be understood that the large language model trained based on the sample input text, the sample image to be located, the labeled target categories and the labeled target positions in the sample image to be located, on the one hand, does not rely on external assistance, and the sample input text and the sample image to be located have a unified structure with the large picture and text model; on the other hand, it makes full use of the extensive knowledge acquired by the large language model through large-scale pre-training, so that it has the ability to locate text input of any granularity, further improving the accuracy and reliability of the conversational target localization method.

The method provided by the embodiment of the present invention inputs the image to be located into the visual encoder to extract the image features of the image to be located, and projects the image features into the word embedding space to obtain the image word vector, segment the input text to obtain the word segmentation vector, and map the word segmentation vector into the word embedding space to obtain the text word vector. Finally, the image word vector and the text word vector are used as an image-text word vector pair, and the image-text word vector pair is input into the large language model to obtain the answer sequence output by the large language model, and the answer sequence includes the target category and target position in the image to be located. The large language model is trained based on sample input text, sample image to be located, and the label target category and label target position in the sample image to be located. On the one hand, it does not rely on external assistance, and the sample input text and the sample image to be located have a unified structure with the large image-text model; on the other hand, it makes full use of the extensive knowledge obtained by the large language model through large-scale pre-training, so as to have the ability to locate text input of any granularity, further improving the accuracy and reliability of the conversational target location method.

Based on the above embodiment, in step 140, the image-text word vector pair is input into a large language model to obtain an answer sequence output by the large language model, and then the following steps are further included:

Step 141, based on the image to be located, the input text, and a preset answer sequence before word segmentation, determining first conditional probabilities of each word segmentation in the answer sequence;

Step 142: Determine the generation probability of the answer sequence based on the first conditional probabilities.

Specifically, the first conditional probabilities of each word segmentation in the answer sequence may be determined based on the image to be located, the input text, and a preset answer sequence before word segmentation.

After obtaining each first conditional probability, the generation probability of the answer sequence can be determined based on each first conditional probability. The formula is as follows:

Among them, p( _Xa | _Xv , _Xins ) represents the generation probability of the answer sequence, _Xa represents the answer sequence with an output length of _La , _Xv represents the image to be located, _Xins represents the input text, _{Xa, <j} represents the answer sequence before the preset segmentation j, and θ is the parameter of all training in the entire framework.

That is, the generation probability of the answer sequence is the product of the conditional probability of each word segmentation in the sequence based on the input (the image to be located and the input text) and the preset answer sequence Xa _{, <j} before the word segmentation.

Therefore, in each step of generation before the end symbol, the embodiment of the present invention can obtain the conditional probability p _θ (xa _,j | _Xv , _Xins ,Xa _,<j ) of each word segmentation.

In the related technology, the existing conversational positioning framework with positioning capabilities is limited to a single target scenario. Users cannot locate all targets of the same category at the same time. When locating targets of multiple categories, multiple inputs or calls to expert models are required, which reduces the accuracy and efficiency of the conversational target positioning method.

Based on the above problems, an embodiment of the present invention proposes a multi-object confidence scoring mechanism that does not require special training, which is used to represent the credibility of each located object and highlight the highest quality objects among all located objects. The framework uses an autoregressive method to generate output.

In step 140, the image-text word vector pair is input into a large language model to obtain an answer sequence output by the large language model, and then the following steps are further included:

Step 210, based on the image to be located, the input text, and a preset answer sequence before word segmentation, determining each second conditional probability of each target category in the image to be located in the answer sequence;

Step 220: Determine the generation probability of each target category in the image to be located in the answer sequence based on the second conditional probabilities.

Specifically, the second conditional probabilities of each target category in the image to be located in the answer sequence may be determined based on the image to be located, the input text, and a preset answer sequence before word segmentation.

After obtaining the second conditional probabilities of each target category, the generation probability of each target category in the image to be located in the answer sequence can be determined based on the second conditional probabilities. The formula is as follows:

Among them, p(R| _Xv , _Xins , _Xa,＜j ) represents the generation probability of each target category in the image to be located in the answer sequence, _Xa represents the answer sequence with an output length of L _a , _Xv represents the image to be located, _Xins represents the input text, _Xa,＜j represents the answer sequence before the preset word segmentation j, _Lr represents the length of each target category, k represents the kth position of the answer sequence, and θ is the parameter of all training in the entire framework.

That is, for each category description of the located object, its target category description R is assumed to be of length L _r and starts from the kth position of the entire answer sequence. According to the Bayesian probability formula, the generation probability of each target category in the image to be located in the answer sequence is obtained.

Based on the above embodiment, step 220 further includes:

Step 221, based on the image to be located, the input text, and the answer sequence before the preset position, determine each third conditional probability of the target position in the image to be located in the answer sequence;

Step 222: Determine the generation probability of each target position in the image to be located in the answer sequence based on the third conditional probabilities.

Specifically, each third conditional probability of the target position in the image to be located in the answer sequence may be determined based on the image to be located, the input text, and the answer sequence before the preset position.

After obtaining the third conditional probabilities of the target positions in the image to be located, the generation probability of each target position in the image to be located in the answer sequence can be determined based on the third conditional probabilities. The formula is as follows:

Among them, p _loc represents the generation probability of each target position in the image to be located in the answer sequence, X _v represents the image to be located, _Xins represents the input text, X _{a, <k} represents the answer sequence before the preset position k, coord represents each target position in the image to be located, that is, the candidate box predicted by each target, (x1, y1) represents the upper left corner coordinate of the candidate box predicted by each target, and (x2, y2) represents the lower right corner coordinate of the candidate box predicted by each target.

Based on the above embodiment, step 222 further includes:

Step 2221, based on the generation probability of each target position and the generation probability of each target category, determine the confidence score of each target in the image to be located.

Specifically, after obtaining the generation probability of each target position and the generation probability of each target category, the confidence score of each target in the image to be located can be determined based on the generation probability of each target position and the generation probability of each target category. The formula is as follows:

Wherein, score represents the confidence score of each target in the image to be located, p _loc represents the generation probability of each target position in the image to be located in the answer sequence, and p(R|X _v ,X _ins ,X _a,＜j ) represents the generation probability of each target category in the image to be located in the answer sequence.

It should be noted that the confidence score of each target in the image to be located is the geometric mean of the generation probability of each target position and the generation probability of each target category.

The method provided by the embodiment of the present invention proposes a target confidence scoring mechanism that does not require training in order to achieve more accurate and user-friendly positioning for positioning scenarios of multiple categories and multiple objects. When outputting multiple targets of different categories, the mechanism simultaneously outputs the confidence score of each target, and the user can filter all located targets by threshold according to actual needs. This framework breaks through the single reference limitation of existing graphic models with positioning capabilities, allowing users to customize the input of descriptions of objects of interest (input text) of different granularities, improving the flexibility of the model, and being able to simultaneously output all detected targets of interest or reject non-existent targets based on user input, reducing the deployment difficulty and resource consumption of introducing multiple models, and has stronger versatility and broader application prospects.

Based on the above embodiment, the training step of the large language model includes:

Step 310, determining an initial large language model, and obtaining the sample input text, the sample image to be located, and the label target category and label target position in the sample image to be located;

Step 320, inputting the sample image to be located into the initial visual encoder to extract sample image features of the sample image to be located, and projecting the sample image features into a sample word embedding space based on an initial projection layer to obtain a sample image word vector;

Step 330, segmenting the sample input text to obtain a sample segmentation vector, and mapping the sample segmentation vector to the sample word embedding space to obtain a sample text word vector;

Step 340, taking the sample image word vector and the sample text word vector as a sample image text word vector pair, and inputting the sample image text word vector pair into an initial large language model to obtain a predicted answer sequence output by the initial large language model;

Step 350, based on the predicted target category and the label target category in the predicted answer sequence, as well as the predicted target position and the label target position in the predicted answer sequence, determine the target loss, and iterate the parameters of the initial large language model based on the target loss to obtain the large language model.

Specifically, in order to better train a large language model, the following steps can be used for training:

First, an initial large language model is determined, and sample input texts, sample images to be located, label target categories and label target positions in sample images to be located are collected in advance.

The parameters of the initial large language model here can be randomly generated or preset.

Then, the sample image to be located can be input into the initial visual encoder to extract the sample image features of the sample image to be located, and the sample image features can be projected into the sample word embedding space based on the initial projection layer to obtain the sample image word vector.

The sample input text is then segmented to obtain a sample segmentation vector, and the sample segmentation vector is mapped to the sample word embedding space to obtain the sample text word vector.

After obtaining the sample image word vector and the sample text word vector, the sample image word vector and the sample text word vector can be used as a sample image-text word vector pair, and the sample image-text word vector pair can be input into the initial large language model to obtain a predicted answer sequence output by the initial large language model.

After obtaining the predicted answer sequence based on the initial large language model, the predicted answer sequence is compared with the labeled answer sequence, and the target loss is determined based on the difference between the predicted answer sequence and the labeled answer sequence.

It can be understood that the greater the difference between the predicted target category in the predicted answer sequence and the labeled target category in the labeled answer sequence, and the predicted target position in the predicted answer sequence and the labeled target position in the labeled answer sequence, the greater the target loss value.

After obtaining the target loss, parameters of the initial large language model can be iterated based on the target loss, and the initial large language model after the parameter iteration is used as the large language model.

Based on the above embodiment, the step 350 of performing parameter iteration on the initial large language model based on the target loss to obtain the large language model further includes:

Step 351, performing gradient back propagation on the initial visual encoder and the initial projection layer based on the target loss to obtain the visual encoder and the projection layer.

Specifically, the initial visual encoder and the initial projection layer may be gradient back-propagated based on the target loss to obtain a visual encoder with updated parameters and a projection layer with updated parameters.

Based on the above embodiment, Figure 3 is a flow chart of the progressive two-stage training method provided by the present invention. As shown in Figure 3, the framework of the embodiment of the present invention is trained using the designed progressive two-stage training method. Previously, the large picture-text model with positioning capability added the referential expression understanding data to the picture-text alignment stage, and then fine-tuned the image input text pairs related to the designed positioning. By increasing the amount of data, this method is feasible for the reference positioning of a single object. However, when dealing with multiple objects of different granularities, the model needs to have a basic understanding of the entire image scene and context. For this reason, the pre-training and input text fine-tuning paradigm of large language models is borrowed, as shown in Figure 3, and the following progressive two-stage training process is proposed:

The first stage is the pre-training of basic scenes. The first stage aims to enhance the multi-object perception capability of the framework and realize positioning in basic scenes. The weights of the inherited LLaVA image-text model are initialized to give the framework the ability to align images and instructions and the ability to understand instructions. 6 million data containing various positioning scenes are used for training. In order to construct the input text _Xins , for each image to be positioned _Xv , a template is randomly selected from all generated task templates and combined with the label to form a training sample. At this stage, the embodiment of the present invention trains the entire network framework, retains the details in the visual features, and is more conducive to small target positioning and fine-grained discrimination.

The second stage is full-scenario instruction fine-tuning. In order to better adapt to the full positioning scenario, instruction fine-tuning (input text fine-tuning) is performed on the basic framework trained in the first stage. Specifically, with the ability to locate text input of any granularity, the framework's understanding of user intent in specific scenarios is further optimized. The embodiment of the present invention further fine-tunes the framework's visual encoder, projection layer, and large language model with 500,000 input text data (instruction data) to enhance the framework's ability to understand user intent.

Based on any of the above embodiments, the present invention provides a conversational target location method based on arbitrary granularity text input, the steps are as follows:

The first step is to obtain the image to be located and the input text.

In the second step, the image to be located is input into the visual encoder to extract the image features of the image to be located, and the image features are projected into the word embedding space to obtain the image word vector.

The third step is to segment the input text to obtain a segmentation vector, and map the segmentation vector to the word embedding space to obtain a text word vector.

In the fourth step, the image word vector and the text word vector are used as an image-text word vector pair, and the image-text word vector pair is input into a large language model to obtain an answer sequence output by the large language model, wherein the answer sequence includes the target category and target position in the image to be located.

The fifth step is to determine the first conditional probabilities of each word segmentation in the answer sequence based on the image to be located, the input text, and the preset answer sequence before word segmentation, and then determine the generation probability of the answer sequence based on each first conditional probability.

The sixth step is to determine the second conditional probabilities of each target category in the image to be located in the answer sequence based on the image to be located, the input text, and the preset answer sequence before word segmentation, and then determine the generation probability of each target category in the image to be located in the answer sequence based on each second conditional probability.

The seventh step is to determine the third conditional probabilities of the target positions in the image to be located in the answer sequence based on the image to be located, the input text, and the answer sequence before the preset position, and then determine the generation probability of each target position in the image to be located in the answer sequence based on each third conditional probability.

In the eighth step, based on the generation probability of each target position and the generation probability of each target category, the confidence score of each target in the image to be located is determined.

Among them, the training steps of the large language model include:

S1. Determine an initial large language model and obtain sample input text, sample images to be located, label target categories and label target positions in the sample images to be located.

S2. Input the sample image to be located into the initial visual encoder to extract the sample image features of the sample image to be located, and project the sample image features into the sample word embedding space based on the initial projection layer to obtain the sample image word vector.

S3. Segment the sample input text to obtain a sample segmentation vector, map the sample segmentation vector to the sample word embedding space, and obtain the sample text word vector.

S4. Use the sample image word vector and the sample text word vector as a sample image-text word vector pair, and input the sample image-text word vector pair into an initial large language model to obtain a predicted answer sequence output by the initial large language model.

S5. Determine the target loss based on the predicted target category and the label target category in the predicted answer sequence, as well as the predicted target position and the label target position in the predicted answer sequence, and iterate the parameters of the initial large language model based on the target loss to obtain the large language model.

S6. Perform gradient backpropagation on the initial visual encoder and the initial projection layer based on the target loss to obtain the visual encoder and the projection layer.

The method provided in the embodiment of the present invention provides a conversational target positioning framework for text input of arbitrary granularity, so that the model has the ability to flexibly understand the object reference descriptions of different granularities input by the user and accurately locate. The embodiment of the present invention refers to the sequence modeling method in natural language processing, and redefines the positioning tasks of different granularities as the problem of generating the location description of the object of interest based on the input text. The conversational target positioning framework unifies all target positioning scenarios into a conversational sequence generation task of input text-all object coordinates without introducing any special placeholders. At the same time, it constructs a confidence scoring mechanism for multi-category targets. Through formal unification, positioning tasks of different granularities under arbitrary text input can be processed by the same model.

The following is a description of the conversational target positioning device based on arbitrary granularity text input provided by the present invention. The conversational target positioning device based on arbitrary granularity text input described below and the conversational target positioning method based on arbitrary granularity text input described above can be referenced to each other.

Based on any of the above embodiments, the present invention provides a conversational target positioning device based on arbitrary granularity text input. FIG4 is a structural schematic diagram of the conversational target positioning device based on arbitrary granularity text input provided by the present invention. As shown in FIG4, the device includes:

An acquisition unit 410 is used to acquire an image to be located and input text;

A projection unit 420 is used to input the image to be located into a visual encoder to extract image features of the image to be located, and project the image features into a word embedding space to obtain an image word vector;

A mapping unit 430 is used to segment the input text to obtain a segmentation vector, and map the segmentation vector to the word embedding space to obtain a text word vector;

An input unit 440 is used to use the image word vector and the text word vector as an image-text word vector pair, and input the image-text word vector pair into a large language model to obtain an answer sequence output by the large language model; the answer sequence includes the target category and target position in the image to be located;

The large language model is trained based on sample input text, sample images to be located, and label target categories and label target positions in the sample images to be located.

The device provided by the embodiment of the present invention inputs the image to be located into the visual encoder to extract the image features of the image to be located, and projects the image features into the word embedding space to obtain the image word vector, segment the input text to obtain the word segmentation vector, and map the word segmentation vector into the word embedding space to obtain the text word vector. Finally, the image word vector and the text word vector are used as an image-text word vector pair, and the image-text word vector pair is input into the large language model to obtain the answer sequence output by the large language model, and the answer sequence includes the target category and target position in the image to be located. The large language model is trained based on sample input text, sample image to be located, and the label target category and label target position in the sample image to be located. On the one hand, it does not rely on external assistance, and the sample input text and the sample image to be located have a unified structure with the large image-text model; on the other hand, it makes full use of the extensive knowledge obtained by the large language model through large-scale pre-training, so as to have the ability to locate text input of any granularity, further improving the accuracy and reliability of the conversational target location method.

Based on any of the above embodiments, the method further includes a unit for determining a generation probability of an answer sequence, wherein the unit for determining a generation probability of an answer sequence is specifically configured to:

Based on the image to be located, the input text, and a preset answer sequence before word segmentation, determining each first conditional probability of each word segmentation in the answer sequence;

Based on the first conditional probabilities, a generation probability of the answer sequence is determined.

Based on any of the above embodiments, it further includes a generation probability unit for determining each target category, and the generation probability unit for determining each target category specifically includes:

Determine each second conditional probability unit, which is used to determine each second conditional probability of each target category in the image to be located in the answer sequence based on the image to be located, the input text, and the preset answer sequence before word segmentation;

A second generation probability determination unit is used to determine the generation probability of each target category in the image to be located in the answer sequence based on the second conditional probabilities.

Based on any of the above embodiments, it further includes a unit for determining a generation probability of each target position, and the unit for determining a generation probability of each target position specifically includes:

Determine each third conditional probability unit, used for determining each third conditional probability of the target position in the image to be located in the answer sequence based on the image to be located, the input text, and the answer sequence before the preset position;

The subunit for determining the generation probability of each target position is used to determine the generation probability of each target position in the image to be located in the answer sequence based on the third conditional probabilities.

Based on any of the above embodiments, it further includes a confidence scoring unit for determining each target, wherein the confidence scoring unit for determining each target is specifically used to:

Based on the generation probability of each target position and the generation probability of each target category, a confidence score of each target in the image to be located is determined.

Based on any of the above embodiments, a training unit is further included, and the training unit is specifically used for:

Determine an initial large language model, and obtain the sample input text, the sample image to be located, and the label target category and label target position in the sample image to be located;

Input the sample image to be located into the initial visual encoder to extract sample image features of the sample image to be located, and project the sample image features into the sample word embedding space based on the initial projection layer to obtain a sample image word vector;

Segmenting the sample input text to obtain a sample segmentation vector, and mapping the sample segmentation vector to the sample word embedding space to obtain a sample text word vector;

The sample image word vector and the sample text word vector are used as a sample image text word vector pair, and the sample image text word vector pair is input into an initial large language model to obtain a predicted answer sequence output by the initial large language model;

Based on the predicted target category and the label target category in the predicted answer sequence, as well as the predicted target position and the label target position in the predicted answer sequence, a target loss is determined, and parameters of the initial large language model are iterated based on the target loss to obtain the large language model.

Based on any of the above embodiments, a gradient back propagation unit is further included, and the gradient back propagation unit is specifically used for:

Performing gradient back propagation on the initial visual encoder and the initial projection layer based on the target loss to obtain the visual encoder and the projection layer.

Figure 5 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 5, the electronic device may include: a processor (processor) 510, a communication interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call the logic instructions in the memory 530 to execute a conversational target localization method based on arbitrary granularity text input, the method comprising: obtaining an image to be located and an input text; inputting the image to be located into a visual encoder to extract image features of the image to be located, and projecting the image features into a word embedding space to obtain an image word vector; segmenting the input text to obtain a segmentation vector, mapping the segmentation vector to the word embedding space to obtain a text word vector; using the image word vector and the text word vector as an image-text word vector pair, and inputting the image-text word vector pair into a large language model to obtain an answer sequence output by the large language model; the answer sequence includes a target category and a target position in the image to be located; the large language model is trained based on sample input text, sample image to be located, and labeled target categories and labeled target positions in the sample image to be located.

In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the conversational target localization method based on arbitrary granularity text input provided by the above methods, the method including: obtaining an image to be located and an input text; inputting the image to be located into a visual encoder to extract image features of the image to be located, and projecting the image features into a word embedding space to obtain an image word vector; segmenting the input text to obtain a segmentation vector, mapping the segmentation vector to the word embedding space to obtain a text word vector; using the image word vector and the text word vector as an image-text word vector pair, and inputting the image-text word vector pair into a large language model to obtain an answer sequence output by the large language model; the answer sequence includes a target category and a target position in the image to be located; the large language model is trained based on sample input text, sample image to be located, and labeled target categories and labeled target positions in the sample image to be located.

On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the conversational target localization method based on arbitrary granularity text input provided by the above-mentioned methods, the method comprising: obtaining an image to be located and an input text; inputting the image to be located into a visual encoder to extract image features of the image to be located, and projecting the image features into a word embedding space to obtain an image word vector; segmenting the input text to obtain a segmentation vector, mapping the segmentation vector to the word embedding space to obtain a text word vector; using the image word vector and the text word vector as an image-text word vector pair, and inputting the image-text word vector pair into a large language model to obtain an answer sequence output by the large language model; the answer sequence includes a target category and a target position in the image to be located; the large language model is trained based on sample input text, sample image to be located, and labeled target categories and labeled target positions in the sample image to be located.

The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A conversational object localization method based on arbitrary granularity text input, comprising:

Acquiring an image to be positioned and an input text;

Inputting the image to be positioned into a visual encoder, extracting image features of the image to be positioned, and projecting the image features into a word embedding space to obtain an image word vector;

Word segmentation is carried out on the input text to obtain word segmentation vectors, and the word segmentation vectors are mapped into the word embedding space to obtain text word vectors;

taking the image word vector and the text word vector as an image text word vector pair, and inputting the image text word vector pair into a large language model to obtain an answer sequence output by the large language model; the answer sequence comprises a target category and a target position in the image to be positioned;

the large language model is obtained through training based on sample input text, sample images to be positioned, label target categories and label target positions in the sample images to be positioned.

2. The conversational object localization method of claim 1, wherein the inputting the image text word vector pairs into a large language model results in a sequence of answers output by the large language model, further comprising:

Determining each first conditional probability generated by each word segmentation in an answer sequence based on the image to be positioned, the input text and the answer sequence before preset word segmentation;

And determining the generation probability of the answer sequence based on the first conditional probabilities.

3. The conversational object localization method of claim 1, wherein the inputting the image text word vector pairs into a large language model results in a sequence of answers output by the large language model, further comprising:

Determining second conditional probabilities of target categories in the image to be positioned in the answer sequence based on the image to be positioned, the input text and the answer sequence before preset word segmentation;

And determining the generation probability of each target category in the image to be positioned in the answer sequence based on each second conditional probability.

4. A conversational object localization method based on any granularity of text input according to claim 3, wherein determining a probability of generation of each object class in the image to be localized in the answer sequence based on each second conditional probability, further comprises:

Determining third conditional probabilities of target positions in the images to be positioned in the answer sequence based on the images to be positioned, the input text and the answer sequence before the preset positions;

and determining the generation probability of each target position in the image to be positioned in the answer sequence based on each third conditional probability.

5. The method for conversational object localization based on any granularity of text input of claim 4, wherein determining a probability of generating each object position in the image to be localized in the answer sequence based on the third conditional probabilities, further comprises:

and determining the confidence scores of the targets in the image to be positioned based on the generation probabilities of the target positions and the generation probabilities of the target categories.

6. The method for conversational object localization based on any one of claims 1 to 5, wherein the training step of the large language model includes:

determining an initial large language model, and acquiring the sample input text, the sample to-be-positioned image, a label target category and a label target position in the sample to-be-positioned image;

Inputting the sample image to be positioned into an initial visual encoder to extract sample image features of the sample image to be positioned, and projecting the sample image features into a sample word embedding space based on an initial projection layer to obtain a sample image word vector;

the sample input text is subjected to word segmentation to obtain a sample word segmentation vector, and the sample word segmentation vector is mapped into the sample word embedding space to obtain a sample text word vector;

the sample image word vector and the sample text word vector are used as sample image text word vector pairs, and the sample image text word vector pairs are input into an initial large language model to obtain a predicted answer sequence output by the initial large language model;

And determining target loss based on the predicted target category and the tag target category in the predicted answer sequence, and the predicted target position and the tag target position in the predicted answer sequence, and carrying out parameter iteration on the initial large language model based on the target loss to obtain the large language model.

7. The conversational object localization method of claim 6, wherein the iterating parameters over the initial large language model based on the object loss to obtain the large language model, further comprising:

and carrying out gradient back transmission on the initial visual encoder and the initial projection layer based on the target loss to obtain the visual encoder and the projection layer.

8. A conversational object-locating device based on arbitrary granularity text input, comprising:

The acquisition unit is used for acquiring the image to be positioned and the input text;

the projection unit is used for inputting the image to be positioned into the visual encoder to extract the image characteristics of the image to be positioned, and projecting the image characteristics into the word embedding space to obtain an image word vector;

The mapping unit is used for word segmentation of the input text to obtain word segmentation vectors, and mapping the word segmentation vectors into the word embedding space to obtain text word vectors;

the input unit is used for taking the image word vector and the text word vector as an image text word vector pair, inputting the image text word vector pair into a large language model, and obtaining an answer sequence output by the large language model; the answer sequence comprises a target category and a target position in the image to be positioned;

the large language model is obtained through training based on sample input text, sample images to be positioned, label target categories and label target positions in the sample images to be positioned.

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the conversational object localization method of any of claims 1 to 7 based on arbitrary granularity text input when the program is executed.

10. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor implements the conversational object localization method of any one of claims 1 to 7 based on arbitrary granularity text input.