CN111813978A - An image description model generation method, generation device and storage medium - Google Patents

Abstract

The present application discloses a method for generating an image description model, a generating device and a storage medium, in particular, acquiring an image sample and a description sentence corresponding to the image sample; inputting the image sample and the description sentence into a first prediction model to generate a first distribution probability set; obtain the second distribution probability set corresponding to the target vocabulary corresponding to each entity object in the image sample; assign weight values to the first distribution probability set and the second distribution probability set respectively, so as to obtain the corresponding target vocabulary at each decoding time. The third distribution probability set of the third distribution probability, and output the predicted sentence; based on the first distribution probability set, the second distribution probability set and the weight value, respectively determine the sentence entity object coverage loss function and the language sequence loss function, and determine the model Loss function to optimize the image description model until the image description model is generated. The embodiment of the present application improves the accuracy of image description by optimizing the image description model.

Description

An image description model generation method, generation device and storage medium

technical field

The present application relates to the field of computer technology, and in particular, to a method for generating an image description model, a generating device and a storage medium.

Background technique

Image description mainly describes the visual content of the image through natural language expression. In recent years, image description technology has been widely used, and automatic image description generation can be applied to image retrieval. The existing image description technology mainly extracts the feature vector of the input image data through Convolutional Neural Networks (CNN), and secondly, uses Recurrent Neural Network (RNN) or Long Short-Term Memory (LongShort-Term Memory, LSTM) decodes the extracted features, obtains the output sequence, and generates image descriptions.

When the existing image description technology trains the image description generation model, the image and the corresponding description sentence are usually used as the sample data set for training, so the vocabulary is fixed, and the generated model is generally limited to the image data that has been trained. , the application in scenarios where new image data is used as input is limited. In addition, the recognition accuracy of the object to be recognized in the description sentence output by the above model is not high.

SUMMARY OF THE INVENTION

An embodiment of the present application provides a method for generating an image description model, which improves the accuracy of image description of an input image to be recognized by establishing an image description model.

The method includes:

obtaining an image sample and a description sentence corresponding to the image sample;

Inputting the image sample and each target word in the description sentence into a pre-trained first prediction model in turn, and generating a first distribution probability corresponding to each target word contained in the description sentence at each decoding moment The first distribution probability set of ;

obtaining a second distribution probability set of the second distribution probability corresponding to the target vocabulary corresponding to each entity object in the image sample at each decoding moment;

Weights are respectively assigned to the first distribution probability corresponding to the target vocabulary in the first distribution probability set and the second distribution probability corresponding to the second distribution probability set, so as to obtain the target vocabulary in each The third distribution probability set of the third distribution probability corresponding to each decoding moment, and output the prediction sentence;

Based on the first distribution probability set, the second distribution probability set and the weight value, determine the sentence entity object coverage loss function and the language sequence loss function respectively;

According to the sentence entity object coverage loss function and the language sequence loss function, a model loss function is determined, and the image description model is optimized according to the model loss function until the image description model is generated.

Optionally, the generating method further includes:

Inputting the image sample into the pre-trained convolutional neural network model, and extracting the feature vector of the image sample;

The feature vector and the target vocabulary corresponding to the description sentence at the last decoding time are input into the long short-term memory network model.

Optionally, the generating method includes:

Inputting the image sample into an object detector pre-trained based on a preset image data set, and generating a prediction score set in which each entity object included in the image sample corresponds to the prediction score of the target vocabulary;

Based on the prediction score set and the hidden state of the long short-term memory network model at the current decoding moment, the target vocabulary is determined as the correct semantic word of the entity object, and the target vocabulary at each decoding moment is generated. A second set of distribution probabilities for the second distribution probability.

Optionally, the generating method includes:

Based on the first weight value assigned to the first distribution probability of the target word and the second weight value assigned to the second distribution probability, a third distribution corresponding to the target word containing the target word at each decoding time is calculated A third distribution probability set of probabilities, wherein the sum of the first weight value and the second weight value is 1.

Optionally, the generating method includes:

Based on the sum of the second distribution probability corresponding to each of the target words at each of the decoding moments in the second distribution probability set and the assigned weight value, the target word in the description The actual distribution probability in the sentence, to determine the sentence entity object coverage loss function;

A language sequence loss function is determined based on the prediction sentence for the image sample and the description sentence output by the first prediction model.

In another embodiment of the present invention, a method for acquiring image description is provided, the method comprising:

Get the image to be recognized;

The to-be-recognized image is input into the image description model in each step of the above-mentioned method for generating an image description model, so as to generate a description sentence of the to-be-recognized image.

In another embodiment of the present invention, an apparatus for generating an image description model is provided, and the generating apparatus includes:

an acquisition module for acquiring an image sample and a description sentence corresponding to the image sample;

The first generation module is used to sequentially input the image sample and each target word in the description sentence into the pre-trained first prediction model, and generate each target word contained in the description sentence in each decoding. the first distribution probability set of the first distribution probability corresponding to the moment;

A second generation module, configured to obtain a second distribution probability set of the second distribution probability corresponding to the target vocabulary corresponding to each entity object in the image sample at each decoding moment;

The third generation module is configured to assign weight values to the first distribution probability corresponding to the target word in the first distribution probability set and the second distribution probability corresponding to the second distribution probability set, to obtain A third distribution probability set containing the third distribution probability corresponding to the target vocabulary at each decoding moment, and outputting a predicted sentence;

a first determination module, configured to determine a sentence entity object coverage loss function and a language sequence loss function based on the first distribution probability set, the second distribution probability set and the weight value, respectively;

The second determination module is configured to determine a model loss function according to the sentence entity object coverage loss function and the language sequence loss function, and optimize the image description model according to the model loss function until the generation of the Image description model.

Optionally, the generating device also includes:

an extraction module for inputting the image sample into the pre-trained convolutional neural network model, and extracting the feature vector of the image sample;

The input module is configured to input the feature vector and the target vocabulary corresponding to the description sentence at the last decoding moment into the long short-term memory network model.

Optionally, the second generation module includes:

an input unit, configured to input the image sample into an object detector pre-trained based on a preset image data set, and generate a predicted score corresponding to the predicted score of the target vocabulary for each entity object contained in the image sample gather;

The generating unit is used to determine the target vocabulary as the correct semantic word of the entity object based on the prediction score set and the hidden state of the long short-term memory network model at the current decoding moment, and generate the target vocabulary in The second distribution probability set of the second distribution probability for each decoding moment.

Optionally, the third generation module is further used for:

Based on the first weight value assigned to the first distribution probability of the target word and the second weight value assigned to the second distribution probability, a third distribution corresponding to the target word containing the target word at each decoding time is calculated A third distribution probability set of probabilities, wherein the sum of the first weight value and the second weight value is 1.

Optionally, the first determining module includes:

The first determination unit is configured to, based on the sum of the second distribution probability corresponding to each of the target words at each of the decoding moments in the second distribution probability set and the assigned weight value, and the Determine the actual distribution probability of the target vocabulary in the description sentence, and determine the sentence entity object coverage loss function;

A second determining unit, configured to determine a language sequence loss function based on the predicted sentence for the image sample and the description sentence output by the first prediction model.

In another embodiment of the present invention, there is provided a device for acquiring image description, the device comprising:

an acquisition module, used to acquire the image to be recognized;

The generating module is configured to input the image to be recognized into the image description model in each step of the above-mentioned method for generating an image description model, so as to generate a description sentence of the image to be recognized.

In another embodiment of the present invention, a non-transitory computer-readable storage medium is provided, the non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the above-described Steps in a method for generating an image description model.

In another embodiment of the present invention, a terminal device is provided, including a processor, where the processor is configured to execute each step in the above-mentioned method for generating an image description model.

As can be seen from the above, based on the above-mentioned embodiment, an image sample and a description sentence corresponding to the image sample are obtained first. Secondly, input the image samples and each target word in the description sentence into the pre-trained first prediction model in turn, and generate a first distribution of the probability of the first distribution corresponding to each target word contained in the description sentence at each decoding moment. set of probabilities. At the same time, a second distribution probability set of the second distribution probability corresponding to the target vocabulary corresponding to each entity object in the image sample at each decoding moment is obtained. Further, assign a weight value to the first distribution probability corresponding to the target word in the first distribution probability set and the second distribution probability corresponding to the second distribution probability set, so as to obtain the corresponding value of the target word at each decoding time. A set of third distribution probabilities of the third distribution probability, and outputting a prediction sentence. Then, based on the first distribution probability set, the second distribution probability set and the weight value, the sentence entity object coverage loss function and the language sequence loss function are respectively determined. Finally, according to the sentence entity object coverage loss function and the language sequence loss function, the model loss function is determined, and according to the model loss function, the image description model is optimized until the image description model is generated. The embodiment of the present application optimizes the image description model by determining the model loss function, and improves the accuracy of the image description.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present application more clearly, the following drawings will briefly introduce the drawings that need to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore do not It should be regarded as a limitation of the scope, and for those of ordinary skill in the art, other related drawings can also be obtained according to these drawings without any creative effort.

FIG. 1 shows a schematic flowchart of a method for generating an image description model provided in Embodiment 10 of the present application;

2 shows a schematic diagram of generating a sentence entity object coverage loss function and a language sequence loss function in a method for generating an image description model provided in Embodiment 20 of the present application;

3 shows a schematic diagram of a specific process in a method for generating an image description model in Embodiment 30 provided by the present application;

FIG. 4 shows a schematic flowchart of a method for acquiring an image description provided by Embodiment 40 of the present application;

FIG. 5 shows a schematic diagram of an apparatus for generating an image description model provided by Embodiment 50 of the present application;

FIG. 6 shows a schematic diagram of an apparatus for acquiring image description provided by Embodiment 60 of the present application;

FIG. 7 shows a schematic diagram of a terminal device provided by Embodiment 70 of the present application.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present application clearer, the present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

Based on the problems in the prior art, the embodiment of the present application provides a method for generating an image description model. By proposing an LSTM-based pointing mechanism LSTM-P for the object to be recognized, the Visual features, and secondly, these visual features are fed into the LSTM, which outputs the first distribution probabilities of all target words contained in the vocabulary. In addition, the pre-trained object detector is used to identify the entity objects in the input image, obtain the predicted scores of each entity object, and use the predicted scores of each entity object and the hidden state of the current decoding moment of the LSTM as input in the replication layer. , output the second distribution probability that the target vocabulary is directly used as the description of the entity object. Finally, the first distribution probability and the second distribution probability output by the two models are input to the pointing mechanism, the weights are dynamically allocated by the pointing mechanism, the final probability distribution of each target vocabulary is output, and the corresponding sentence entity object coverage loss is calculated. function and language sequence loss function to determine the model loss function of the entire image description model and optimize the image description model. The image description model based on LSTM-P expands the range of the target vocabulary that can be recognized, and at the same time improves the accuracy of the description sentences for image description.

The application field of this application is mainly in the field of computer technology, and is applicable to the field of computer vision technology and natural language processing. As shown in FIG. 1 , it is a schematic diagram of Embodiment 10 in a method for generating an image description model provided by an embodiment of the present application. The detailed steps are as follows:

S11, acquire an image sample and a description sentence corresponding to the image sample.

In this step, multiple image samples and description sentences of the image samples are obtained as multiple pairs of input data. For the image samples shown in Figure 2, the corresponding description sentences are "a large dog laying on a blanket on a couch", where, The description sentence corresponding to the image sample needs to be preset. At the same time, the descriptive sentence is a correct description of the content meaning of the image sample.

S12: Input the image sample and the description sentence into the pre-trained first prediction model, and generate a first distribution probability set of the first distribution probability corresponding to each target word included in the description sentence at each decoding moment.

为经过包含CNN和LSTM的第一预测模型对图像样本和对应的描述语句input sentence进行训练后得到的预测的各个第一分布概率组成的第一分布概率集合。In this step, the first prediction model is mainly composed of CNN and LSTM. The CNN model is mainly used to extract the feature vector of the image sample, input the image sample into the CNN model, and output the feature vector of the image sample. Specifically, the image samples are input into the CNN, the pre-trained network VGG-16 or ResNet is used, and the feature vectors of the image samples are extracted through network layers other than the softmax classifier that outputs the class scores. CNN is usually called an encoder, which encodes a large amount of information contained in image samples into feature vectors, processes them, and uses the extracted feature vectors as the initial input for subsequent LSTMs. The role of LSTM is mainly to process feature vectors through decoding and convert them into natural language. Among them, the image samples and the corresponding description sentences are input into the LSTM together. Specifically, the features extracted by the CNN and the target vocabulary in the description sentence at the last decoding time are input into the LSTM, and the LSTM generates the predicted sentences one by one according to the order of the words in the sentence. Here, LSTM finally outputs the first distribution probability set of the first distribution probability of the target vocabulary at each decoding moment according to the order of the words in the sentence. As shown in Figure 2, where

is a first distribution probability set composed of each predicted first distribution probability obtained after the image sample and the corresponding description sentence input sentence are trained by the first prediction model including CNN and LSTM.

S13: Acquire a second distribution probability set of the second distribution probability corresponding to the target vocabulary corresponding to each entity object in the image sample at each decoding moment.

In this step, the image sample is firstly input into the pre-trained object detector, and the object detector identifies each entity object that may be included in the image sample. Among them, the object detector is pre-trained on the image dataset. The image dataset contains a large amount of image data and corresponding descriptors. The object detector is used to identify each entity object in the image sample, and each entity object may be the predicted score set of the corresponding target vocabulary. As shown in Figure 2, the object detector is represented as Object Leaner. After inputting the image sample into Object Leaner, each entity object contained in the output image sample may be the predicted score of the target vocabulary. When the image sample shown in Figure 2 is input into Object Leaner, Object Leaner outputs the predicted probability that each entity object may be related to the target vocabulary, that is, the predicted score, such as dog: 1.00, couch; 0.21, bed: 0.13, blank: 0.12.

Secondly, after obtaining the prediction score set of the prediction scores for each entity object, the hidden state of the LSTM at the current decoding moment and the obtained prediction score set are input into the Copying Layer at the same time. In the replication layer, the target vocabulary of the recognized entity object is determined as the correct semantic word of each entity object in the input image sample, and the second distribution probability of the second distribution probability corresponding to the target vocabulary at each decoding moment is generated gather

S14, respectively assigning weights to the first distribution probability corresponding to the target word in the first distribution probability set and the second distribution probability corresponding to the second distribution probability set, so as to obtain the third distribution probability corresponding to the target word at each decoding time. A third set of distribution probabilities for distribution probabilities.

In this step, after obtaining the first distribution probability set generated by the first prediction model and the second distribution probability set generated in the Copying Layer, the above two distribution probability sets are input into the created Pointing Mechanism model. Pointing Mechanism dynamically assigns weights _pt and 1- _pt to the two probability distribution sets, and calculates the weighted sum to determine the finally generated third distribution probability set Pr that contains the third distribution probability of each target vocabulary in the predicted sentence ^t .

S15: Determine the sentence entity object coverage loss function and the language sequence loss function based on the first distribution probability set, the second distribution probability set, and the weight value.

In this step, because the sentence finally generated by the first prediction model is composed of each vocabulary generated one by one according to the order of the target vocabulary in the description sentence. Therefore, the first distribution rate of the sentence finally generated by the first prediction model is equal to the sum of the logarithmic probabilities generated by each word in the sentence to represent the language sequence loss function. By minimizing the language sequence loss function, the final output sentence is guaranteed. The contextual grammatical dependencies of each target vocabulary in the paper make the generated sentences grammatically fluent.

In addition, for the final semantic consistency of predicted sentences, multi-label classification is performed on the target vocabulary. Specifically, in the second distribution probability set generated by the above-mentioned replication layer, the sum of the products of the second distribution probability at each decoding time and the assigned corresponding weight value is calculated, and the probability of each target vocabulary at each decoding time is obtained. word probability. Finally, according to the probability of word occurrence, the coverage loss function of the sentence entity object is obtained. At the same time, by minimizing the sentence entity object coverage loss function, it is ensured that each target word in the final output sentence is consistent with the correct meaning of the entity object in each corresponding image sample, and the accuracy of each target word in the generated sentence is improved.

S16: Determine the model loss function according to the sentence entity object coverage loss function and the language sequence loss function, and optimize the image description model according to the model loss function until the image description model is generated.

In this step, after the above-mentioned semantic consistency loss function and language sequence loss function are determined, optimization parameters are selected to determine the model loss function. At the same time, the LSTM-P image description model is continuously revised according to the model loss function until the optimal image description model is generated.

Based on the above-mentioned embodiment of the present application, firstly obtain an image sample and a description sentence corresponding to the image sample; The first distribution probability set of the first distribution probability corresponding to each target word included at each decoding moment, and at the same time, the second distribution probability corresponding to the target word corresponding to each entity object in the image sample at each decoding moment is obtained. the second distribution probability set, and then assign weight values to the first distribution probability corresponding to the target word in the first distribution probability set and the second distribution probability corresponding to the second distribution probability set, so as to obtain the target word in each The third distribution probability set of the third distribution probability corresponding to each decoding moment, and output the predicted sentence. Based on the first distribution probability set, the second distribution probability set and the weight value, the sentence entity object coverage loss function and the language sequence loss function are determined respectively. , and finally, according to the sentence entity object coverage loss function and the language sequence loss function, the model loss function is determined, and according to the model loss function, the image description model is optimized until the image description model is generated. In the embodiment of the present application, an image description model based on LSTM-P is designed to realize the image description of the image to be recognized. Through the directivity mechanism, the correct target vocabulary corresponding to the identified entity object is dynamically integrated into the output prediction sentence. The image description model based on LSTM-P first fully exploits the contextual relevance between the generated target words through the conventional CNN&RNN language model. At the same time, an object trainer trained on the identified dataset recognizes solid objects in the input image samples. In addition, the vocabulary is directly copied from the target vocabulary corresponding to the recognized entity object through the Copying Layer. Further, the pointing mechanism is to generate predicted sentences through the first prediction model composed of CNN and LSTM, and directly copy words from the recognized target words, and dynamically assign weight values to the target words through these two methods. Pointing Mechanism can identify the best time to copy object samples to corresponding descriptive sentences in the current context. And by reducing the error rate of sentence entity object coverage loss and language sequence loss in the output prediction sentence, the image description model based on LSTM-P is trained globally. In addition, the lack of sentence-level coverage is used as feedback to further improve the coverage of each entity object in the image sample by the output predicted sentence.

As shown in FIG. 3 , it is a schematic diagram of a specific flow of a method for generating an image description model in Embodiment 30 provided for this application. The detailed process of the specific process is as follows:

S301: Obtain an image sample and a description sentence corresponding to the image sample.

S302: Input the image sample and each target word in the corresponding description sentence into the pre-trained first prediction model in turn.

Here, the first prediction training model includes a convolutional neural network CNN and a long short-term memory network model LSTM. Specifically, the image samples are input into a pre-trained convolutional neural network model, and feature vectors of the image samples are extracted. Further, the feature vector and the target vocabulary corresponding to the description sentence at the last decoding moment are input into the long short-term memory network model.

S303: Generate a first distribution probability set.

In this step, the output of the first prediction model is a first distribution probability set describing the first distribution probability corresponding to each target word included in the sentence at each decoding moment. For an image sample I, a sentence S is used to describe the image sample I. where S={ω ₁ , ω ₂ , . . . , ω _Ns } consists of N _s words. I ∈ R ^Dv and Wt ∈ R ^Dw represent Dv-dimensional visual features and Dw-dimensional textual features of the t-th word in sentence S, respectively. In the first prediction model stage, the formula 1 of the first distribution probability of the target vocabulary ωt ₊₁ is expressed as follows:

Among them, h ^t is the output state of the LSTM at the decoding time t.

S304, input the image samples into the object detector, and generate a set of prediction scores.

Here, the image samples are input into an object detector pre-trained based on a preset image data set, and a prediction score set in which each entity object included in the image samples corresponds to the prediction score of the target vocabulary is generated.

S305, according to the prediction score set and the hidden state of the LSTM at the current moment, generate a second distribution probability set.

In this step, the image samples are first input into an object detector pre-trained based on a preset image data set, and a prediction score set in which each entity object included in the image samples corresponds to the prediction score of the target vocabulary is generated. Further, based on the prediction score set and the hidden state of the long short-term memory network model at the current decoding moment, the target vocabulary is determined as the correct semantic word of the entity object, and the second distribution probability of the target vocabulary at each decoding moment is generated. A collection of distribution probabilities. The calculation formula 2 of the probability of the second distribution is as follows:

和

是变换矩阵。where h ^t is the output state of the LSTM at decoding time t, I _c is the output of the object detector,

and

is the transformation matrix.

S306, the pointing mechanism dynamically assigns weight values to the first distribution probability set and the second distribution probability set.

Here, after the first distribution probability set and the second distribution probability set are input into the pointing mechanism, the pointing mechanism dynamically assigns the weight value, and the specific calculation process is embodied by the following calculation formula 3:

Pt=σ(G _s w _t +G _h h ^t +b _p ) (3)

和

分别表示文本特征的变换矩阵和LSTM模型输出的变换矩阵，b_p为偏置向量。in,

and

respectively represent the transformation matrix of the text feature and the transformation matrix output by the LSTM model, and b _p is the bias vector.

S307: Determine a third distribution probability set according to the assigned weight value.

Here, based on the first weight value assigned to the first distribution probability of the target word and the second weight value assigned to the second distribution probability, a third distribution probability including the third distribution probability corresponding to the target word at each decoding time is calculated A set, wherein the sum of the first weight value and the second weight value is 1. Among them, the final distribution probability of the target vocabulary ω _t+1 is obtained by assigning the weight value p _t calculated by the formula 3 to the above formula 1 and formula 2:

和

分别表示第一权重值和第二权重值，φ表示归一化函数，如Softmax。in,

and

represent the first weight value and the second weight value, respectively, and φ represents a normalization function, such as Softmax.

S308, determine the language sequence loss function.

In this step, the language sequence loss function is determined based on the prediction sentence and the description sentence for the image sample output by the first prediction model. Since the first prediction model generates a corresponding target word at each decoding time, the chain rule is used to model the distribution probability of multiple words. The negative log probability of the entire output prediction sentence is equal to the sum of the negative log first distribution probabilities of all target words, which can be expressed by the following formula 5:

By minimizing the language sequence loss function shown in Equation 5, the context dependencies between each target vocabulary in the output predicted sentence can be made more accurate.

S309: Determine the sentence entity object coverage loss function.

Here, based on the second distribution probability of the target vocabulary represented by the aforementioned formula 2 and the weight value determined by the formula 3, the coverage of each target vocabulary included in the output prediction sentence is further calculated. The specific calculation formula is as follows:

Among them, ω _oi represents the copied target vocabulary, σ is a nonlinear activation function, such as the Sigmoid function, as shown on the right side of Figure 2, the product of the second distribution probability of each target vocabulary and the weight value is obtained at each decoding time , and get the sum of the products of each target vocabulary in the sentence, the weighted sum. Determine coverage for each target vocabulary. Further, the sentence entity object coverage loss function is determined according to the following formula:

Equation 7 is the final statement entity object coverage loss function. By minimizing the sentence entity object coverage loss function, the description sentence is closer to the real meaning of the image sample.

S310, determine the model loss function.

Here, the model loss function is determined according to the language sequence loss function determined in Equation 5 and the sentence entity object coverage loss function determined in Equation 7. The specific model loss function is expressed as follows:

where λ is a trade-off parameter. The purpose of the model loss function is to optimize the image description model, so that the last description sentence predicted by the image description model conforms to the language logic, and can accurately describe the input image to be predicted.

S311, the image description model is optimized, and the model is generated.

The embodiments of the present application implement a method for generating an image description model based on the above steps. The method of image description is realized by constructing an image description model based on LSTM-P structure. At the same time, the configuration method of vocabulary expansion and the training method of hybrid network of image description model are implemented. Among them, based on the available known image data set, the object detector is pre-trained, and then a pointing mechanism is constructed to balance the first distribution probability of the word recognition result generated based on the first prediction model composed of CNN and LSTM , and copy the extracted target vocabulary directly from the data recognized by the object detector. In addition, the coverage of the target vocabulary in the predicted sentence is further mined, thereby improving the accuracy of image description. Experiments based on COCO image description dataset and ImageNet image recognition dataset fully validate the model and analysis results.

In addition, as shown in FIG. 4 , a method for obtaining image description provided by Embodiment 40 of the present application is to input the image to be recognized into the image description model in each step of the above-mentioned method for generating an image description model, so as to generate A description of the image to be recognized.

S41, acquiring an image to be recognized.

S42, and input the image to be recognized into the image description model to generate a description sentence of the image to be recognized.

Based on the same inventive concept, Embodiment 50 of the present application further provides a device for generating an image description model, wherein, as shown in FIG. 5 , the generating device includes:

an acquisition module 51, configured to acquire an image sample and a description sentence corresponding to the image sample;

The first generation module 52 is used to sequentially input each target word in the image sample and the description sentence into the pre-trained first prediction model, and generate the first prediction model corresponding to each target word included in the description sentence at each decoding moment. a first set of distribution probabilities of distribution probabilities;

The second generation module 53 is used to obtain the second distribution probability set of the second distribution probability corresponding to the target vocabulary corresponding to each entity object in the image sample at each decoding moment;

The third generating module 54 is configured to respectively assign weight values to the first distribution probability corresponding to the target word in the first distribution probability set and the second distribution probability corresponding to the second distribution probability set, so as to obtain the target word in each The third distribution probability set of the third distribution probability corresponding to each decoding moment, and output the prediction sentence;

The first determination module 55 is configured to determine the sentence entity object coverage loss function and the language sequence loss function respectively based on the first distribution probability set, the second distribution probability set and the weight value;

The second determination module 56 is configured to cover the loss function and the language sequence loss function according to the sentence entity object, determine the model loss function, and optimize the image description model according to the model loss function until the image description model is generated.

In this embodiment, the specific functions and interaction methods of the acquisition module 51 , the first generation module 52 , the second generation module 53 , the third generation module 54 , the first determination module 55 and the second determination module 56 can be referred to in FIG. 1 . The description of the embodiment is not repeated here.

Optionally, optionally, the generating device further includes:

Extraction module 57, for inputting the image sample into the pre-trained convolutional neural network model, and extracting the feature vector of the image sample;

The input module 58 is configured to input the feature vector and the target vocabulary corresponding to the description sentence at the last decoding moment into the long short-term memory network model.

Optionally, the second generation module 53 includes:

The input unit is used to input the image sample into the object detector pre-trained based on the preset image data set, and generate a prediction score set in which each entity object included in the image sample corresponds to the prediction score of the target vocabulary;

The generating unit is used to determine the target vocabulary as the correct semantic word of the entity object based on the prediction score set and the hidden state of the long short-term memory network model at the current decoding moment, and generate the second distribution probability of the target vocabulary at each decoding moment. The second distribution probability set.

Optionally, the third generation module 54 is also used for:

Based on the first weight value assigned to the first distribution probability of the target word and the second weight value assigned to the second distribution probability, a third distribution probability set containing the third distribution probability corresponding to the target word at each decoding time is calculated, The sum of the first weight value and the second weight value is 1.

Optionally, the first determining module 55 includes:

The first determining unit is configured to, based on the sum of the second distribution probability corresponding to each target word at each decoding moment in the second distribution probability set and the assigned weight value, and the actual distribution probability of the target word in the description sentence, Determine the statement entity object coverage loss function;

The second determination unit is configured to determine the language sequence loss function based on the prediction sentence and the description sentence for the image sample output by the first prediction model.

Based on the same inventive concept, Embodiment 60 of the present application further provides an apparatus for acquiring image description, wherein, as shown in FIG. 6 , the apparatus includes:

an acquisition module 61, configured to acquire an image to be recognized;

The generating module 62 is configured to input the image to be recognized into the image description model, so as to generate a description sentence of the image to be recognized.

In this embodiment, for the specific functions and interaction methods of the acquisition module 61 and the generation module 62, reference may be made to the description of the embodiment corresponding to FIG. 4, and details are not described herein again.

As shown in FIG. 7 , another embodiment 70 of the present application further provides a terminal device, including a processor 70, wherein the processor 70 is configured to execute the steps of one of the above methods.

It can also be seen from FIG. 7 that the terminal device provided in the above embodiment further includes a non-transitory computer-readable storage medium 71 , and a computer program is stored on the non-transitory computer-readable storage medium 71 , and the computer program is executed by the processor 70 . When performing the steps of the above-mentioned method for generating an image description model.

Specifically, the storage medium can be a general storage medium, such as a removable disk, a hard disk, and FLASH, etc. When the computer program on the storage medium is run, the above-mentioned method for generating an image description model can be executed.

Finally, it should be noted that the above-mentioned embodiments are only specific implementations of the present application, and are used to illustrate the technical solutions of the present application, rather than limit them. The embodiments describe the application in detail, and those of ordinary skill in the art should understand that: any person skilled in the art can still modify the technical solutions described in the foregoing embodiments within the technical scope disclosed in the application. Or can easily think of changes, or equivalently replace some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions in the embodiments of the application, and should be covered in this application. within the scope of protection. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (14)

1. a generation method of image description model, is characterized in that, comprises: obtaining an image sample and a description sentence corresponding to the image sample; Inputting the image sample and each target word in the description sentence into a pre-trained first prediction model in turn, and generating a first distribution probability corresponding to each target word contained in the description sentence at each decoding moment The first distribution probability set of ; obtaining a second distribution probability set of the second distribution probability corresponding to the target vocabulary corresponding to each entity object in the image sample at each decoding moment; Weights are respectively assigned to the first distribution probability corresponding to the target vocabulary in the first distribution probability set and the second distribution probability corresponding to the second distribution probability set, so as to obtain the target vocabulary in each The third distribution probability set of the third distribution probability corresponding to each decoding moment, and output the prediction sentence; Based on the first distribution probability set, the second distribution probability set, and the weight value, determine a semantic consistency loss function and a syntactic consistency loss function, respectively; A model loss function is determined according to the semantic consistency loss function and the syntactic coherence loss function, and the image description model is optimized according to the model loss function until the image description model is generated. 2. The generation method according to claim 1, wherein the first prediction model is composed of a convolutional neural network model and a long short-term memory network model, and, in the image sample and the description sentence, each of the Between the step of sequentially inputting the target words into the pre-trained first prediction model and the step of generating the first distribution probability set of the first distribution probability corresponding to each target word included in the description sentence at each decoding moment, the The generating method further includes: Inputting the image sample into the pre-trained convolutional neural network model, and extracting the feature vector of the image sample; The feature vector and the target vocabulary corresponding to the description sentence at the last decoding time are input into the long short-term memory network model. 3 . The generating method according to claim 2 , wherein the target vocabulary corresponding to each entity object in the acquired image sample corresponds to the second distribution probability set of the second distribution probability corresponding to each decoding moment. 4 . steps, including: Inputting the image sample into an object detector pre-trained based on a preset image data set, and generating a prediction score set in which each entity object included in the image sample corresponds to the prediction score of the target vocabulary; Based on the prediction score set and the hidden state of the long short-term memory network model at the current decoding moment, the target vocabulary is determined as the correct semantic word of the entity object, and the target vocabulary at each decoding moment is generated. A second set of distribution probabilities for the second distribution probability. 4. The generation method according to claim 3, wherein the step of obtaining a third distribution probability set comprising the third distribution probability corresponding to the target vocabulary at each decoding moment comprises: Based on the first weight value assigned to the first distribution probability of the target word and the second weight value assigned to the second distribution probability, a third distribution corresponding to the target word containing the target word at each decoding time is calculated A third distribution probability set of probabilities, wherein the sum of the first weight value and the second weight value is 1. 5. The generating method according to claim 4, wherein the step of respectively determining the semantic consistency loss function and the syntactic consistency loss function comprises: Based on the sum of the second distribution probability corresponding to each of the target words at each of the decoding moments in the second distribution probability set and the assigned weight value, the target word in the description The actual distribution probability in the sentence to determine the semantic consistency loss function; A syntactic coherence loss function is determined based on the prediction sentence for the image sample and the description sentence output by the first prediction model. 6. A method for obtaining image description, characterized in that, based on the method according to any one of the steps of claims 1 to 5: Get the image to be recognized; The to-be-recognized image is input into an image description model to generate a description sentence of the to-be-recognized image. 7. A device for generating an image description model, comprising: an acquisition module for acquiring an image sample and a description sentence corresponding to the image sample; The first generation module is used to sequentially input the image sample and each target word in the description sentence into the pre-trained first prediction model, and generate each target word contained in the description sentence in each decoding. the first distribution probability set of the first distribution probability corresponding to the moment; A second generation module, configured to obtain a second distribution probability set of the second distribution probability corresponding to the target vocabulary corresponding to each entity object in the image sample at each decoding moment; The third generation module is configured to assign weight values to the first distribution probability corresponding to the target word in the first distribution probability set and the second distribution probability corresponding to the second distribution probability set, to obtain A third distribution probability set containing the third distribution probability corresponding to the target vocabulary at each decoding moment, and outputting a predicted sentence; a first determination module, configured to respectively determine a semantic consistency loss function and a syntactic consistency loss function based on the first distribution probability set, the second distribution probability set and the weight value; The second determination module is configured to determine a model loss function according to the semantic consistency loss function and the syntactic coherence loss function, and optimize the image description model according to the model loss function until the generation of the Image description model. 8. The generating device according to claim 7, wherein the generating device further comprises: an extraction module for inputting the image sample into the pre-trained convolutional neural network model, and extracting the feature vector of the image sample; The input module is configured to input the feature vector and the target vocabulary corresponding to the description sentence at the last decoding moment into the long short-term memory network model. 9. The generating apparatus according to claim 8, wherein the second generating module comprises: an input unit, configured to input the image sample into an object detector pre-trained based on a preset image data set, and generate a predicted score corresponding to the predicted score of the target vocabulary for each entity object contained in the image sample gather; The generating unit is used to determine the target vocabulary as the correct semantic word of the entity object based on the prediction score set and the hidden state of the long short-term memory network model at the current decoding moment, and generate the target vocabulary in The second distribution probability set of the second distribution probability for each decoding moment. 10. The generating apparatus according to claim 9, wherein the third generating module is further configured to: Based on the first weight value assigned to the first distribution probability of the target word and the second weight value assigned to the second distribution probability, a third distribution corresponding to the target word containing the target word at each decoding time is calculated A third distribution probability set of probabilities, wherein the sum of the first weight value and the second weight value is 1. 11. The generating apparatus according to claim 10, wherein the first determining module comprises: The first determination unit is configured to, based on the sum of the second distribution probability corresponding to each of the target words at each of the decoding moments in the second distribution probability set and the assigned weight value, and the Determine the actual distribution probability of the target vocabulary in the description sentence, and determine the semantic consistency loss function; A second determining unit, configured to determine a syntactic coherence loss function based on the prediction sentence for the image sample and the description sentence output by the first prediction model. 12. A device for acquiring image description, characterized in that it comprises: an acquisition module for acquiring the image to be recognized; A generating module is configured to input the image to be recognized into an image description model to generate a description sentence of the image to be recognized. 13. A non-transitory computer-readable storage medium, characterized in that the non-transitory computer-readable storage medium stores instructions that, when executed by a processor, cause the processor to perform any of claims 1 to 5. Each step in a method for generating an image description model described in one item. 14. A terminal device, characterized by comprising a processor, wherein the processor is configured to execute each step in the method for generating an image description model according to any one of claims 1 to 5.

Images