CN117671678A - Image labeling method and device - Google Patents

Abstract

The application discloses an image labeling method and device, and belongs to the technical field of computer vision. After the computer equipment acquires a training image set corresponding to the target image task and a language description text set corresponding to the training image set, a target image annotation model is called to determine a prediction label of each training sample image, the prediction label of the training sample image is obtained by the target image annotation model based on a characteristic matching result of image characteristics corresponding to the training sample image and language characteristics corresponding to each language description text in the language description text set, and then the target image annotation model is trained according to errors between real labels and prediction labels of a plurality of training sample images in the training image set until the target image annotation model converges. According to the method and the device, the image task is converted into the matching task of the image features and the language features, so that the initial labeling performance of the image labeling model is greatly improved, and the image labeling efficiency is improved.

Description

Image annotation method and device

Technical field

The present application relates to the field of computer vision technology, and in particular to an image annotation method and device.

Background technique

When developing image-oriented artificial intelligence (AI) applications, manpower is required to annotate training images one by one. Since successful AI models require thousands or even millions of accurately annotated training images, image annotation tasks are often time-consuming and costly.

Intelligent image annotation is one of the most practical technologies when developing image-oriented AI applications. It is based on a small number of labeled images and uses AI algorithms to quickly realize automatic labeling of images to be labeled. With the help of intelligent image annotation technology, users can save a lot of image annotation costs. When using intelligent image annotation technology, how to improve the efficiency of image annotation is an urgent problem that needs to be solved.

Contents of the invention

This application provides an image annotation method and device, which can improve image annotation efficiency.

The first aspect provides an image annotation method. This method is used on computer equipment. The method includes: obtaining a training image set corresponding to the target image task and a language description text set corresponding to the training image set. The training image set includes multiple training sample images. Each training sample image among the plurality of training sample images is annotated with a real label. The language description text set includes a plurality of language description texts. The multiple language description texts are in one-to-one correspondence with the multi-category labels corresponding to the training image set. Each language description text is used to describe the semantics of one category of tags among multiple categories of tags. Call the target image annotation model corresponding to the target image task to determine the predicted label of each training sample image in the training image set. The predicted label of the training sample image is determined by the target image annotation model based on the image features corresponding to the training sample image and the language description text set. The feature matching results of the language features corresponding to each language description text are obtained. Based on the error between the real labels and the predicted labels of multiple training sample images in the training image set, the target image annotation model is trained until the target image annotation model converges. The target image annotation model is used to determine the annotation label of the image to be annotated under the target image task.

Among them, the image features corresponding to the training sample images are image features obtained by performing feature extraction on the training sample images. The language features corresponding to the language description text are the language features obtained by feature extraction from the language description text. Model convergence can be when the loss value of the model is less than the preset threshold, or when the weight change of the model in two adjacent iterations is less than the preset threshold, or when the number of model iterations reaches the preset number.

In this application, the computer device obtains the language description text set corresponding to the training image set, introduces the language prior association knowledge of the annotation task into the training sample image, and transforms the image task into a match between image features and language features in the image annotation model. Task. This can greatly improve the initial annotation performance of the image annotation model. When the number of initial training sample images is small, it can improve the first round annotation accuracy of the image annotation model, effectively reduce the number of training rounds of the image annotation model, and thereby improve the efficiency of image annotation. .

Optionally, after the target image annotation model converges, the computer device can also call the target image annotation model to determine the predicted labels of the multiple verification sample images in the verification image set. The verification image set includes a plurality of verification sample images, and each of the plurality of verification sample images is annotated with a true label. The computer device determines the annotation accuracy of the target image annotation model based on the real labels and predicted labels of multiple verification sample images. When the annotation accuracy of the target image annotation model does not reach the preset threshold, the computer device performs one or more model training processes until the annotation accuracy of the target image annotation model reaches the preset threshold. Among them, the model training process includes: calling the target image annotation model, determining the predicted labels of multiple images to be annotated and the confidence of the predicted labels of the images to be annotated. Obtain difficult-to-label images whose predicted label confidence is lower than the confidence threshold from multiple images to be labeled. Output difficult-to-label images and predicted labels of difficult-to-label images for manual correction. In response to receiving the manual annotation results for the difficult-to-label images, the difficult-to-label images are added to the training image set as new training sample images to obtain an updated training image set. Call the target image annotation model to determine the predicted label of each training sample image in the updated training image set. The predicted label of the training sample image is based on the image features corresponding to the training sample image and the language description text set corresponding to the updated training image set. The feature matching results of the language features corresponding to each language description text are obtained. Based on the error between the real labels and the predicted labels of multiple training sample images in the updated training image set, the target image annotation model is trained until the target image annotation model converges again.

Optionally, when the annotation accuracy of the target image annotation model reaches a preset threshold, the computer device uses the predicted label of the image to be annotated determined by calling the target image annotation model as the annotation label of the image to be annotated.

Optionally, the target image annotation model includes a first feature extraction layer, a second feature extraction layer and a feature matching layer. The first feature extraction layer includes an image feature output end and a language feature output end. The feature matching layer includes an image feature input terminal and a language feature input terminal. The image feature output terminal is connected to the input terminal of the second feature extraction layer. The language feature output terminal is connected to the language feature input terminal. The output end of the second feature extraction layer is connected to the image feature input end. The computer device calls the target image annotation model corresponding to the target image task and determines the implementation process of the prediction label of each training sample image in the training image set, including: the computer device uses the first feature extraction layer to identify each language description text in the language description text set. Features are extracted separately to obtain a language feature set. The language feature set includes multiple sets of language features, and each set of language features is a language feature corresponding to a language description text in the language description text set. For each training sample image in the training image set, the computer device performs feature extraction on the training sample image through the first feature extraction layer to obtain the global image features corresponding to the training sample image, and performs feature extraction on the global image features through the second feature extraction layer , obtain the target image features, which are associated with the target image task. The target image features are matched with each group of language features in the language feature set through the feature matching layer to obtain the feature matching results. The feature matching results include the target image features. With the feature similarity between each group of language features in the language feature set, the label described by the target language description text is used as the predicted label of the training sample image. The target language description text is the language description text corresponding to a set of language features in the language feature set that has the highest feature similarity with the target image features.

Optionally, the target image annotation model also includes a supervision module, and a loss function matching the target image task is set in the supervision module. The input terminal of the supervision module is connected to the output terminal of the feature matching layer. The implementation of the computer device training a target image annotation model based on the error between the real labels and the predicted labels of multiple training sample images in the training image set includes: the computer device uses a supervision module based on the error between the multiple training sample images in the training image set. Real labels and predicted labels, calculate the loss value of the loss function; reversely transmit the gradient information of the loss function to the second feature extraction layer to adjust the network parameters from the second feature extraction layer to the feature matching layer.

Optionally, the image annotation model framework is pre-stored in the computer device. The image annotation model framework includes the first feature extraction layer, the downstream task model head and the feature matching layer. The image feature output end of the first feature extraction layer is connected to the input end of the downstream task model head. The output end of the downstream task model head is connected to the image feature input end of the feature matching layer. The downstream task model head includes multiple feature extraction layers that correspond to multiple image tasks one-to-one. The second feature extraction layer is the feature extraction layer corresponding to the target image task in the downstream task model header. Wherein, the image feature output end of the first feature extraction layer is configured to connect to one feature extraction layer in the downstream task model head at a time.

In this application, by designing the downstream task model header in the image annotation model framework, the image annotation model framework can adapt to diverse downstream needs. In this way, you only need to select different feature extraction layers in the downstream task model header to construct the corresponding image annotation model. Since multiple image tasks can share the image annotation model framework, there is no need to design corresponding image annotation models for each image task. Unified and standardized management of intelligent image annotation can be achieved, which can reduce the complexity of development and maintenance, thereby reducing technical costs.

Optionally, the first feature extraction layer is implemented by a pre-trained visual language pre-training model. This visual language pre-training model can extract features from the input image to obtain image features, and can also extract features from the input language description text to obtain language features.

Optionally, the implementation of the computer device to obtain the language description text set corresponding to the training image set includes: in response to receiving a startup instruction for the target image task, the computer device displays a template setting prompt, the template setting prompt is used to prompt the user to set The language description template corresponding to the target image task. For each type of label corresponding to the training image set, the computer device generates a language description text based on the set language description template and label.

This application provides a human-computer interaction interface so that users can manually set the language description template corresponding to the image task. This can improve the accuracy of the semantic expression of the label by the language description text, and further improve the accuracy of subsequent language features extracted based on the language description text. accuracy, thereby improving the accuracy of the image annotation model.

Optionally, after generating the language description text, the computer device may display the language description text.

In this application, the computer device displays the language description text for the user to check whether the generated language description text can accurately express the meaning of the label, so that the user can adjust the language description template.

Optionally, the computer device can also display multiple image tasks, and the target image task is one of the multiple image tasks. In response to detecting the selection operation on the target image task, the computer device determines that a start instruction for the target image task is received.

Optionally, the multiple image tasks include but are not limited to one or more of image classification, object detection, or action recognition.

In a second aspect, an image annotation device is provided. The device includes multiple functional modules, and the multiple functional modules interact to implement the method in the above-mentioned first aspect and its various implementations. The multiple functional modules can be implemented based on software, hardware, or a combination of software and hardware, and the multiple functional modules can be arbitrarily combined or divided based on specific implementation.

In a third aspect, a computer device is provided, including: a processor and a memory;

The memory is used to store a computer program, the computer program includes program instructions;

The processor is configured to call the computer program to implement the method in the above first aspect and its various implementations.

In a fourth aspect, a computer-readable storage medium is provided. Instructions are stored on the computer-readable storage medium. When the instructions are executed by a processor, the methods in the above-mentioned first aspect and its various embodiments are implemented.

In a fifth aspect, a computer program product is provided, which includes a computer program. When the computer program is executed by a processor, the method in the above first aspect and its various embodiments is implemented.

In a sixth aspect, a chip is provided. The chip includes programmable logic circuits and/or program instructions. When the chip is run, the method in the first aspect and its various embodiments is implemented.

Description of drawings

Figure 1 is a schematic diagram of an image annotation model framework provided by an embodiment of the present application;

Figure 2 is a schematic flowchart of an image annotation method provided by an embodiment of the present application;

Figure 3 is a schematic diagram of a display interface provided by an embodiment of the present application;

Figure 4 is a schematic diagram of another display interface provided by an embodiment of the present application;

Figure 5 is a schematic structural diagram of a target image annotation model provided by an embodiment of the present application;

Figure 6 is a schematic diagram of the architecture involved in an image annotation method provided by an embodiment of the present application;

Figure 7 is a schematic structural diagram of an image annotation device provided by an embodiment of the present application;

Figure 8 is a schematic structural diagram of another image annotation device provided by an embodiment of the present application;

Figure 9 is a schematic diagram of the hardware structure of an image annotation device provided by an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

Intelligent image annotation is a technology that uses AI algorithms to automatically annotate unlabeled images based on a small number of annotated images. With intelligent image annotation, users can reduce image data annotation costs by 50% to 90%. Currently, the basic process of intelligent image annotation includes the following five steps.

Step 1. Train to obtain an AI model corresponding to the image task based on a training image set. The training image set includes multiple training sample images labeled with real labels.

Step 2: Perform inference on the image to be annotated based on the AI model to obtain predicted labels (also called pseudo labels).

Step 3: Screen out valuable samples and their predicted labels from the inferred images according to the confidence strategy for manual correction and annotation.

Step 4. Add the manually corrected and annotated images as new training sample images to the training image set, and re-optimize the AI model based on the updated training image set.

Step 5. Repeat the iterative optimization of steps 1-4 until a high-accuracy AI model is obtained. Then use the AI model for reasoning on all images to be annotated, output the predicted labels of the images to be annotated, and output the final predicted labels. As the final annotation label corresponding to the image to be annotated, the intelligent image annotation result is obtained.

However, when the current intelligent image annotation technology has a small number of initial training sample images (images annotated with real labels), the accuracy of the first round of model training is low, and the confidence of the predicted labels obtained based on AI model inference is not reliable. The poor performance makes it difficult to find valuable samples in the image screening process based on the confidence strategy (step 3 above), which in turn leads to more rounds of model training and manual correction annotation, so the current image annotation efficiency is low.

Based on this, embodiments of the present application provide an image annotation method. First, the computer device obtains a training image set corresponding to the image task and a language description text set corresponding to the training image set. The training image set includes multiple training sample images annotated with real labels. The language description text set includes a plurality of language description texts. The multiple language description texts have a one-to-one correspondence with the multi-category labels corresponding to the training image set. That is, the number of language description texts in the language description text set is the same as the number of label categories of the training sample images in the training image set. Each language description text is used to describe the semantics of a class of labels corresponding to the training image set. Then, the computer device calls the image annotation model corresponding to the image task to determine the prediction label of each training sample image in the training image set. The prediction label is specifically determined by the image annotation model based on the image features corresponding to the training sample image and the language description text set. The feature matching results of the language features corresponding to each language description text are obtained. Finally, the computer device trains the image annotation model corresponding to the image task based on the error between the real labels and the predicted labels of the multiple training sample images in the training image set until the image annotation model converges. The final trained image annotation model whose annotation accuracy reaches the preset threshold is used to determine the annotation label of the image to be annotated under the image task. The embodiment of this application obtains the language description text set corresponding to the training image set, introduces the language prior association knowledge of the annotation task into the training sample image, and transforms the image task into a matching task of image features and language features in the image annotation model. This can greatly improve the initial annotation performance of the image annotation model. When the number of initial training sample images is small, it can improve the first round annotation accuracy of the image annotation model, effectively reduce the number of training rounds of the image annotation model, and thereby improve the efficiency of image annotation. .

In addition, when developing image-oriented AI applications, there are various types of image tasks. For example, types of image tasks include but are not limited to image classification, object detection, and action recognition. Currently, for each image task, it is necessary to design an AI model separately to train an image annotation model that can complete the automatic annotation of images under that image task. Designing AI models separately for each image task has high development and maintenance costs.

Optionally, embodiments of the present application provide an image annotation model framework suitable for multiple image tasks. The image annotation model framework includes a first feature extraction layer, a downstream task model head and a feature matching layer. For example, FIG. 1 is a schematic diagram of an image annotation model framework provided by an embodiment of the present application. As shown in Figure 1, the first feature extraction layer includes an image feature output terminal m1 and a language feature output terminal m2. The feature matching layer includes an image feature input terminal n1 and a language feature input terminal n2. The image feature output terminal m1 of the first feature extraction layer is connected to the input terminal of the downstream task model head. The language feature output terminal m2 of the first feature extraction layer is connected to the language feature input terminal n2 of the feature matching layer. The output end of the downstream task model head is connected to the image feature input end n1 of the feature matching layer. The image tasks corresponding to the downstream task model head include image classification, target detection and action recognition.

The first feature extraction layer is used to extract features from the input language description text and obtain the language features corresponding to the language description text. The first feature extraction layer is also used to extract features from the input image to obtain global image features corresponding to the image. Since the global image features obtained by extracting features from the image in the first feature extraction layer can reflect the global features of the entire image, different image tasks can share the first feature extraction layer. Optionally, the first feature extraction layer is implemented by a pre-trained visual language pre-training model. The visual language pre-training model is a model trained based on large-scale visual image data and corresponding language descriptions. The embodiment of the present application does not elaborate on the training method of the visual language pre-training model.

The downstream task model head includes multiple feature extraction layers that correspond to multiple image tasks one-to-one. Each feature extraction layer in the downstream task model header is used to extract image features under the corresponding image task, that is, the image features extracted by the feature extraction layer in the downstream task model header are associated with the corresponding image task. For example, the feature extraction layer corresponding to the image classification task in the downstream task model header is used to extract features of the image area containing the classification object. The feature extraction layer corresponding to the target detection task in the downstream task model header is used to extract features of the image area containing the detection target and position features of the image area containing the detection target. The feature extraction layer corresponding to the action recognition task in the downstream task model header is used to extract image area features related to the object to be recognized. The image feature output end of the first feature extraction layer is configured to be connected to one feature extraction layer in the downstream task model head at a time. When using the image annotation model framework, you can select the corresponding feature extraction layer in the downstream task model header according to the image task that needs to be performed to build an image annotation model corresponding to the image task, and then when using the built image annotation model, The selected feature extraction layer performs further feature extraction on the global image features output by the first feature extraction layer, and outputs the finally extracted image features to the feature matching layer.

The feature matching layer is used to match the input image features and language features to obtain the feature similarity between the image features and the language features. Since the function of the feature matching layer is the same under different image tasks, the feature matching layer can be constructed uniformly for multiple image tasks. In the subsequent model training process, the network parameters of the feature matching layer can be automatically adjusted according to the actual image task. .

In the embodiment of this application, the downstream task model header is designed in the image annotation model framework, so that the image annotation model framework can adapt to diverse downstream needs. It can be constructed by simply selecting different feature extraction layers in the downstream task model header. Corresponding image annotation model. Since multiple image tasks can share the image annotation model framework, there is no need to design corresponding image annotation models for each image task. Unified and standardized management of intelligent image annotation can be achieved, which can reduce the complexity of development and maintenance, thereby reducing technical costs.

The technical solution provided by this application is introduced in detail below from multiple perspectives such as application scenarios, method processes, software devices, and hardware devices.

The application scenarios of the embodiments of this application are illustrated below with examples.

The image annotation method provided by the embodiment of the present application is used in computer equipment. The computer device can be a server, a server cluster composed of several servers, or a cloud computing center. For example, the image annotation method can be applied to a cloud computing development platform as a web service. Alternatively, the image annotation method can also be applied to the server, and users interact with functions through a software user interface (UI).

The following is an example of the method flow of the embodiment of the present application.

For example, FIG. 2 is a schematic flowchart of an image annotation method provided by an embodiment of the present application. As shown in Figure 2, the method includes:

Step 201: The computer device acquires a training image set corresponding to the target image task and a language description text set corresponding to the training image set.

The training image set includes a plurality of training sample images, and each training sample image in the plurality of training sample images is labeled with a real label. The real labels of the training sample images can be manually annotated. The language description text set includes multiple language description texts, and the multiple language description texts correspond to the multi-category labels corresponding to the training image set. That is, the number of language description texts in the language description text set corresponds to the number of training samples in the training image set. Images have the same number of label categories. Each language description text is used to describe the semantics of one category of tags among the multiple categories of tags.

Alternatively, the target image task can be image classification, object detection, or action recognition.

For example, the target image task is image classification, specifically classifying animal pictures. The training image set includes two types of training sample images. The real label of one type of training sample image is dog, and the real label of the other type of training sample image is cat. That is, the training image set corresponds to two types of labels. Then the language description text set corresponding to the training image set includes two language description texts, one language description text is used to describe that the image contains a dog, and the other language description text is used to describe that the image contains a cat. For example, the format of the language description text can be "a photo of{}", "a xx (xx is an adjective){}" or "this is a{}", and "{}" is used to add the real value of the training sample image. Label.

For another example, the target image task is target detection, specifically the detection of leftover garbage. It is necessary to detect garbage and pedestrians carrying garbage. The training image set includes two types of training sample images. One type of training sample image only contains garbage. The real label of this type of training sample image is garbage. The other type of training sample image contains pedestrians carrying garbage. The real label of this type of training sample image is Including garbage and pedestrians, that is, the training image set corresponds to two types of labels. Then the language description text set corresponding to the training image set includes two language description texts, one language description text is used to describe that the image contains garbage, and the other language description text is used to describe that the image contains pedestrians. For example, the two language description texts can be "there are bags of {garbage}" and "a{person} on the road" respectively. The format of the language description text under the target detection task can be "detect: {}", "there is {} on the xx (xx is a noun)" or "{}, which is xx (xx is an adjective)", "{ }" is used to add the real label of the training sample image. It is worth noting that in addition to being marked with real labels, the training sample images under the target detection task can also be marked with ground truth (GT) positions. The ground truth positions reflect the area of the detection target in the image. Real boxes are usually rectangular boxes.

For another example, the target image task is action recognition, specifically identifying the decontamination actions of people. The training image set includes three types of training sample images. The real label of one type of training sample image is foot washing, the real label of another type of training sample image is hand washing, and the real label of another type of training sample image is disinfection, that is, the training image set Corresponds to three types of labels. Then the language description text set corresponding to the training image set includes three language description texts, one language description text is used to describe the person in the image washing his feet, another language description text is used to describe the person in the image washing his hands, and A verbal description text used to describe the person in the image being disinfected. For example, the format of the language description text can be "the man is{}", "humanaction of{}" or "this is{}, a frame of action", "{}" is used to add the real label of the training sample image .

Optionally, an implementation manner in which the computer device obtains the language description text set corresponding to the training image set includes the following steps 2011 to 2012.

In step 2011, in response to receiving a startup instruction for the target image task, the computer device displays a template setting prompt, where the template setting prompt is used to prompt the user to set a language description template corresponding to the target image task.

Optionally, the template setting prompt may include one or more alternative semantic description templates under the target image task. And/or, the template setting prompt may include a custom control for the user to enter a semantic description template themselves. For example, the target image task is image classification. Figure 3 is a schematic diagram of a display interface provided by an embodiment of the present application. This display interface is the template setting interface. As shown in Figure 3, display interface A includes language description template option A1 and custom control A2 corresponding to the target image task. Language description template option A1 includes two language description templates, one language description template is "a photo of {}", and the other language description template is "this is a{}". Custom control A2 includes input box, add option and confirmation option. When the computer device detects the selection operation of the add option, the computer device uses the input content in the input box as a language description template corresponding to the target image task, and activates the input box again for the user to add a new language description template. When the computer device detects the selection operation of the confirmation option, the computer device uses the input content in the input box as the language description template corresponding to the target image task, and ends the template customization setting process. When setting the language description template, the user can select the language description template provided by the computer device, or can define the language description template by himself, or can combine the selection of the language description template provided by the computer device and the self-defined language description template, that is, The language description template corresponding to an image task finally determined by the user may include the language description template corresponding to the image task provided by the computer device and/or the language description template defined by the user.

Optionally, after the computer device obtains the language description template set by the user, it can also fine-tune the language description template according to the specific image task currently performed, so that the language description template can better match the current image task, thereby improving the subsequent generated language. How accurately the description text describes the semantics of the tag. For example, the image task currently performed by the computer device is to identify the category of flowers in the picture, and the language description template set by the user is "a photo of {}". The computer device can fine-tune the language description template to obtain the language description template "a flower" photo of{}" to more accurately convey that this is the task of classifying flowers.

Optionally, before the computer device displays the template setting prompt, the computer device displays a plurality of image tasks, and the target image task is one of the plurality of image tasks. In response to detecting the selection operation on the target image task, the computer device determines that a start instruction for the target image task is received. For example, FIG. 4 is a schematic diagram of another display interface provided by an embodiment of the present application. This display interface is the image task startup interface. As shown in Figure 4, display interface B includes an image task option, which includes three image tasks, namely image classification, target detection and action recognition. For example, the target image task is image classification. When the computer device detects a selection operation for image classification through display interface B, the computer device can display display interface A as shown in Figure 3.

In step 2012, for each type of label corresponding to the training image set, the computer device generates a language description text based on the set language description template and the label.

For example, the target image task is image classification. The training image set includes two types of training sample images. The real label of one type of training sample image is dog, and the real label of the other type of training sample image is cat. The language description template set by the user is "a photo of {}", the computer device generates two language description texts, namely "a photo of {dog}" and "a photo of {cat}".

The embodiments of the present application provide a human-computer interaction interface so that users can manually set the language description template corresponding to the image task. This can improve the accuracy of the semantic expression of the label by the language description text, and further improve the subsequent language extraction based on the language description text. Feature accuracy, thereby improving the accuracy of the image annotation model.

Optionally, after the computer device generates the language description text, it may also display the language description text. The computer device displays the language description text for the user to check whether the generated language description text can accurately express the meaning of the label, so that the user can adjust the language description template.

Alternatively, when the user knows all the tags corresponding to the training image set, the user can also set a language description text according to each tag to obtain a set of language description texts corresponding to the training image set.

Step 202: The computer device calls the target image annotation model corresponding to the target image task and determines the prediction label of each training sample image in the training image set.

The predicted label of the training sample image is obtained by the target image annotation model based on the feature matching results of the image features corresponding to the training sample image and the language features corresponding to each language description text in the language description text set.

Optionally, FIG. 5 is a schematic structural diagram of a target image annotation model provided by an embodiment of the present application. As shown in Figure 5, the target image annotation model includes a first feature extraction layer, a second feature extraction layer and a feature matching layer. The first feature extraction layer includes an image feature output terminal m1 and a language feature output terminal m2. The feature matching layer includes an image feature input terminal n1 and a language feature input terminal n2. The image feature output terminal m1 of the first feature extraction layer is connected to the input terminal of the second feature extraction layer. The language feature output terminal m2 of the first feature extraction layer is connected to the language feature input terminal n2 of the feature matching layer. The output terminal of the second feature extraction layer is connected to the image feature input terminal n1 of the feature matching layer.

Optionally, an image annotation model framework is pre-stored in the computer device, and the image annotation model framework can be, for example, as shown in Figure 1 . The above-mentioned second feature extraction layer is the feature extraction layer corresponding to the target image task in the downstream task model header. Optionally, in response to receiving a startup instruction for the target image task, the computer device selects a feature extraction layer corresponding to the target image task in the downstream task model header, and constructs a target image annotation model.

Optionally, combined with the target image annotation model shown in Figure 5, the implementation process of the above step 202 may include the following steps 2021 to 2022.

In step 2021, the computer device performs feature extraction on each language description text in the language description text set through the first feature extraction layer to obtain a language feature set.

The language feature set includes multiple sets of language features, and each set of language features is a language feature corresponding to a language description text in the language description text set. Optionally, the language description text set includes M language description texts, and the language feature corresponding to the m-th language description text is expressed as L _m . Then the language feature set can be expressed as: L={L ₁ , L ₂ , ..., L _m ,…, L _M }. Among them, M is an integer greater than 1, 1≤m≤M. The first feature extraction layer performs feature extraction on each language description text in the input language description text set, and then outputs the obtained language feature set to the feature matching layer.

In step 2022, for each training sample image in the training image set, the computer device performs a label prediction process respectively to obtain the predicted label of each training sample image.

The label prediction process includes the following steps S1 to S4.

In step S1, the computer device performs feature extraction on the training sample image through the first feature extraction layer to obtain global image features corresponding to the training sample image.

Optionally, the first feature extraction layer is implemented by a pre-trained visual language pre-training model. Global image features are used to reflect the global features of the entire image, and global image features have nothing to do with the image task. After the first feature extraction layer extracts features from the input training sample image, it outputs the obtained global image features to the second feature extraction layer.

In step S2, the computer device performs feature extraction on the global image features through the second feature extraction layer to obtain target image features, and the target image features are associated with the target image task.

Optionally, the second feature extraction layer uses a feature extraction algorithm that matches the target image task to extract global image features. The second feature extraction layer is mainly used to extract feature sets with strong correlation with categories from global image features. For example, if the target image task is image classification or action recognition, the second feature extraction layer is used to extract features of the image area containing the classified object or the object to be recognized. For another example, if the target image task is target detection, the second feature extraction layer is used to extract features of the image area containing the detection target and location features of the image area containing the detection target. The positional features of the image area containing the detection target can be represented by the predicted frame position. The prediction box is usually a rectangular box. After the second feature extraction layer extracts features from the input global image features, it outputs the obtained target image features to the feature matching layer.

In step S3, the computer device performs feature matching on the target image feature and each group of language features in the language feature set through the feature matching layer to obtain a feature matching result. The feature matching result includes the target image feature and each set of language features in the language feature set. Feature similarity between groups of linguistic features.

Optionally, the feature similarity between image features and language features can be cosine similarity or Euclidean similarity, etc.

Optionally, the target image task is image classification or action recognition. For the training sample images in the training image set, the image features output by the second feature extraction layer are F, and i is a positive integer. Language feature set L={L ₁ , L ₂ ,..., L _m ,..., L _M }.

Then the feature matching layer obtains the feature matching result based on the input language feature set and target image features, which can be expressed as: Logits=func_similarity(F, L). Among them, func_similarity is a function that calculates the feature similarity between image features and each group of language features in the language feature set.

Optionally, the target image task is target detection. For the training sample images in the training image set, the second feature extraction layer can extract N regional features. The N regional features can be expressed as: O _F = {O ₁ , O ₂ ,..., O _n ,..., O _N }. Among them, N is an integer greater than 1, 1≤n≤N. Language feature set L={L ₁ , L ₂ ,..., L _m ,..., L _M }. Then the feature matching layer based on the input language feature set and target image features, the obtained feature matching result can be expressed as: Logits=O _F L ^T , L ^T represents the transpose matrix of L. The feature matching results include the feature similarity between each regional feature and each set of language features.

In step S4, the computer device uses the label described by the target language description text as the predicted label of the training sample image, and the target language description text corresponds to a set of language features in the language feature set that have the highest feature similarity with the target image features. language description text.

For target detection tasks, an image may contain multiple detection targets. Then, for each detection target in the image, the computer device selects a set of language features in the language feature set that have the highest feature similarity with the image features of the detection target. The label described by the corresponding language description text is used as the prediction label of the detection target, and finally the prediction labels of all detection targets in the image are used as the prediction label of the image. For example, it is necessary to detect garbage and pedestrians carrying garbage from the image. If the image only contains garbage, then the predicted label of the image is garbage. If the image contains pedestrians carrying garbage, then the predicted label of the image includes both garbage and pedestrians.

Optionally, the above step S4 can be completed by the feature matching layer, that is, after the feature matching layer performs feature matching on the image features and language features to obtain the feature matching results, it determines the prediction label of the training sample image based on the feature matching results, and finally the feature matching layer outputs Predicted labels for training sample images. Alternatively, the above step S4 can also be completed by other modules in the image annotation model (such as a supervision module) based on the output results of the feature matching layer. Alternatively, the above step S4 may also be determined separately by the computer device based on the output result of the feature matching layer. For the latter two cases, the feature matching layer performs feature matching on image features and language features to obtain the feature matching results, and then directly outputs the feature matching results.

Step 203: The computer device trains the target image annotation model based on the error between the real labels and the predicted labels of the multiple training sample images in the training image set until the target image annotation model converges.

Optionally, please continue to refer to Figure 5. The target image annotation model also includes a supervision module, and a loss function matching the target image task is set in the supervision module. The input terminal of the supervision module is connected to the output terminal of the feature matching layer. Referring to Figure 5, the feature matching layer can output the feature matching results to the supervision module, and the supervision module determines the prediction label of the training sample image based on the input feature matching results.

Optionally, the implementation process of the above step 203 may include the following steps 2031 to 2032.

In step 2031, the computer device calculates the loss value of the loss function based on the real labels and predicted labels of the multiple training sample images in the training image set through the supervision module.

Optionally, the target image task is image classification or action recognition. Then the loss function set in the supervision module can be expressed as: loss=func_classify(Logits, G). Among them, G is the real label of the training sample image. The meaning of Logits refers to the relevant definition in the above step S3 when the target image task is image classification or action recognition. func_classify is a function that calculates classification loss based on labels, such as cross-entropy loss function, focal loss (a difficult sample mining loss) function or its variants and other loss functions.

Optionally, the target image task is target detection. The loss function set in the supervision module consists of two parts, including classification loss and positioning loss. The classification loss function can be expressed as: loss_cls=func_classify(Logits, T _c ). Among them, T _c is the real label of the training sample image. Assuming that the training sample image includes K detection targets, then T _c = {C ₁ ,..., C _k ,..., C _K }. K is a positive integer, 1≤k≤K. The meaning of Logits refers to the relevant definition in the above step S3 when the target image task is target detection. The positioning loss function can be expressed as: loss_loc=func_iou(O _B , T _B ). Among them, _OB is the N prediction box positions corresponding to the N regional features extracted by the second feature extraction layer, and _OB can be output to the supervision module by the second feature extraction layer. T _B is the K real box positions corresponding to the K detection targets in the training sample image, which can be expressed as: T _B = {Box ₁ ,..., Box _k ,..., Box _K }. func_iou is a function that calculates the intersection over union (IOU) of the predicted box and the real box, such as generalized IOU (GIOU), complete intersection and union ratio (complete IOU, IOU) and other loss functions. The loss value of the loss function under the target detection task can be the sum of the loss value of the classification loss function and the loss value of the positioning loss function.

In step 2032, the computer device reversely transmits the gradient information of the loss function to the second feature extraction layer to adjust network parameters from the second feature extraction layer to the feature matching layer.

Alternatively, the above step 2032 can also be replaced by: the computer device reversely transmits the gradient information of the loss function to the first feature extraction layer to adjust the network parameters from the first feature extraction layer to the feature matching layer. That is, the first feature extraction layer may not be adjusted during the model training process, or the first feature extraction layer may be fine-tuned according to the actual image task during the model training process.

In a round of model training, the computer device repeatedly trains the image annotation model based on the same training image set (that is, continuously adjusts the network parameters) until the loss function converges, that is, the image annotation model that converges under this round of training is obtained. The convergence of the loss function may be when the loss value of the loss function reaches a preset value.

It is worth mentioning that during the model training process, a supervision module can be set up in the image annotation model. After the model training is completed, since the supervision module no longer plays a role in reverse tuning of network parameters, the image annotation model can be The supervision module is deleted or retained.

The above steps 201 to 203 describe a round of training process for the target image annotation model. After the computer device completes a round of training on the target image annotation model, it can further verify the accuracy of the target image annotation model. If the accuracy of the target image annotation model reaches the preset requirements, the computer device stops training the target image annotation model, and uses the finally trained target image annotation model to determine the annotation label of the image to be annotated under the target image task. If the accuracy of the target image annotation model does not meet the preset requirements, the computer device continues to conduct a new round of training on the target image annotation model. In this way, the target image annotation model is iteratively updated until the accuracy of the target image annotation model reaches the preset requirements. For the specific implementation process, please refer to the following steps 204 to 207.

Step 204: After the target image annotation model converges, the computer device calls the target image annotation model to determine the predicted labels of the multiple verification sample images in the verification image set.

Wherein, the verification image set includes a plurality of verification sample images, and each verification sample image in the plurality of verification sample images is marked with a real label. The true labels of the verification sample images can be manually annotated. The predicted label of the verification sample image is obtained by the target image annotation model based on the feature matching results of the image features corresponding to the verification sample image and the language features corresponding to each language description text in the language description text set. The way in which the target image annotation model predicts and determines the predicted label of the verification sample image can refer to the method of determining the predicted label of the training sample image in the above step 202, which will not be described again in the embodiment of the present application.

Optionally, there is no intersection between the verification image set and the training image set. The verification image set can be a fixed image set, that is, the verification sample images in the verification image set remain unchanged. For example, the intelligent annotation data set includes a total of 1,000 images, of which 100 images are annotated with real labels, and the remaining 900 images are images to be annotated. Then 50 images annotated with real labels can be used as verification sample images to obtain a verification image set. . And another 50 images marked with real labels are used as training sample images to obtain an initial training image set.

Step 205: The computer device determines the annotation accuracy of the target image annotation model based on the real labels and predicted labels of the multiple verification sample images.

For example, the verification image set includes 50 verification sample images. If the real labels of 30 of the verification sample images are the same as the predicted labels, the annotation accuracy of the target image annotation model is 60%.

Step 206: When the annotation accuracy of the target image annotation model does not reach the preset threshold, the computer device performs one or more model training processes until the annotation accuracy of the target image annotation model reaches the preset threshold.

In the embodiment of the present application, when there are few initial training sample images, the images to be annotated can be used for model training together. Combined with the active learning strategy, the difficult examples in the images to be annotated are selected for manual correction and annotation, and the manually corrected and annotated images are added to the training image set as new training sample images, thereby expanding the scale of the training image set, and The model undergoes a new round of training to improve model accuracy. The above model training process includes the following steps 2061 to 2066.

In step 2061, the computer device calls the target image annotation model to determine the predicted labels of multiple images to be annotated and the confidence of the predicted labels of the images to be annotated.

In step 2062, the computer device obtains a difficult-to-label image whose predicted label confidence is lower than a confidence threshold from a plurality of images to be labeled.

In step 2063, the computer device outputs the difficult-to-label image and the predicted label of the difficult-to-label image for manual correction.

In step 2064, in response to receiving the manual annotation result for the difficult-to-label image, the computer device adds the difficult-to-label image as a new training sample image to the training image set to obtain an updated training image set.

Among them, the manual annotation results include the real labels of manually annotated image annotations. The above-mentioned steps 2061 to 2064 are active learning in the model training process, that is, the part with manual participation, and iterative learning process in which suitable candidate sets are selected by computer equipment for manual annotation.

In step 2065, the computer device calls the target image annotation model to determine the predicted label of each training sample image in the updated training image set. The predicted label of the training sample image is based on the image features corresponding to the training sample image and the updated training image. The feature matching results of the language features corresponding to each language description text in the set of language description texts corresponding to the set are obtained.

Optionally, if the labels corresponding to the updated training image set change, the computer device re-obtains the language description text set corresponding to the updated training image set. For specific implementation methods, please refer to the above steps 2011 to 2012. This application implements The example will not be repeated here.

In step 2066, the computer device trains the target image annotation model based on the error between the real labels and the predicted labels of the multiple training sample images in the updated training image set until the target image annotation model converges again.

The implementation process of this step 2066 can be referred to the implementation process of the above-mentioned step 203, which will not be described again in the embodiment of the present application.

Step 207: When the annotation accuracy of the target image annotation model reaches a preset threshold, the computer device uses the predicted label of the image to be annotated determined by calling the target image annotation model as the annotation label of the image to be annotated.

Optionally, the computer device may further output annotation labels of all images to be annotated.

For example, FIG. 6 is a schematic diagram of the architecture involved in an image annotation method provided by an embodiment of the present application. As shown in Figure 6, this architecture includes data storage media, processors and interactive UIs. The data storage medium is used to store intelligent annotation data sets, and the intelligent annotation data sets include annotated images and images to be annotated. The processor may be a central processing unit (CPU) or a graphics processing unit (GPU). The processor is used to run and train the image annotation model. The image annotation model includes a sequentially cascaded visual language pre-training model, a downstream task model head and a language image feature matching layer. The visual language pre-training model is also connected to the language image feature matching layer. The interactive UI is used for users to add language descriptions, including setting language description templates and adding language description texts in batches. The interactive UI is also used to output annotation results. For example, it can display the predicted labels of the images to be annotated by the image annotation model, and present active learning/difficult example mining results for users to perform manual correction annotation, etc. Downstream task model heads include but are not limited to image tasks such as image classification, target detection, and action recognition.

To sum up, in the image annotation method provided by the embodiment of the present application, by obtaining the language description text set corresponding to the training image set, the language prior associated knowledge of the annotation task is introduced into the training sample image, and the language a priori associated knowledge is introduced into the image annotation model. The image task is transformed into a matching task of image features and language features. This can greatly improve the initial annotation performance of the image annotation model. When the number of initial training sample images is small, it can improve the first round annotation accuracy of the image annotation model, effectively reduce the number of training rounds of the image annotation model, and thereby improve the efficiency of image annotation. . In addition, embodiments of the present application can also provide an image annotation model framework suitable for multiple image tasks. By designing a downstream task model header in the image annotation model framework, the image annotation model framework can adapt to diverse downstream needs by simply Select different feature extraction layers in the downstream task model header to build the corresponding image annotation model. Since multiple image tasks can share the image annotation model framework, there is no need to design corresponding image annotation models for each image task. Unified and standardized management of intelligent image annotation can be achieved, which can reduce the complexity of development and maintenance, thereby reducing technical costs.

The sequence of the steps of the image annotation method provided by the embodiments of the present application can be adjusted appropriately, and the steps can also be increased or decreased accordingly according to the situation. Any person familiar with the technical field can easily think of modified methods within the technical scope disclosed in this application, and they should be covered by the protection scope of this application. For example, when the computer device displays information, the computer device may display the information on its own display interface, or the computer device may send the information to other display devices, and the other display devices display the information.

The following is an example of the virtual device in the embodiment of the present application.

For example, FIG. 7 is a schematic structural diagram of an image annotation device provided by an embodiment of the present application. As shown in FIG. 7 , the image annotation device 700 includes: an acquisition module 701 , a determination module 702 and a training module 703 .

The acquisition module 701 is used to acquire the training image set corresponding to the target image task and the language description text set corresponding to the training image set. The training image set includes multiple training sample images, and each training sample image in the multiple training sample images is marked with The real label and language description text set includes multiple language description texts. The multiple language description texts correspond to the multi-category labels corresponding to the training image set. Each language description text is used to describe the semantics of one type of label in the multi-category label. .

The determination module 702 is used to call the target image annotation model corresponding to the target image task, and determine the prediction label of each training sample image in the training image set. The prediction label of the training sample image is determined by the target image annotation model based on the image features corresponding to the training sample image. The feature matching results of the language features corresponding to each language description text in the language description text collection are obtained.

The training module 703 is used to train the target image annotation model according to the error between the real labels and the predicted labels of the multiple training sample images in the training image set until the target image annotation model converges. The target image annotation model is used to determine the image to be annotated. Annotation labels under the target image task.

Optionally, the determination module 702 is also configured to call the target image annotation model after the target image annotation model converges to determine the predicted labels of multiple verification sample images in the verification image set. The verification image set includes multiple verification sample images. Each verification sample image in the verification sample images is annotated with a true label. The determination module 702 is also used to determine the annotation accuracy of the target image annotation model based on the real labels and predicted labels of the multiple verification sample images. The training module 703 is also configured to perform one or more model training processes when the annotation accuracy of the target image annotation model does not reach the preset threshold, until the annotation accuracy of the target image annotation model reaches the preset threshold.

Among them, the model training process includes: calling the target image annotation model, determining the predicted labels of multiple images to be annotated and the confidence of the predicted labels of the images to be annotated. Obtain difficult-to-label images whose predicted label confidence is lower than the confidence threshold from multiple images to be labeled. Output difficult-to-label images and predicted labels of difficult-to-label images for manual correction. In response to receiving the manual annotation results for the difficult-to-label images, the difficult-to-label images are added to the training image set as new training sample images to obtain an updated training image set. Call the target image annotation model to determine the predicted label of each training sample image in the updated training image set. The predicted label of the training sample image is based on the image features corresponding to the training sample image and the language description text set corresponding to the updated training image set. The feature matching results of the language features corresponding to each language description text are obtained. Based on the error between the real labels and the predicted labels of multiple training sample images in the updated training image set, the target image annotation model is trained until the target image annotation model converges again.

Optionally, the determination module 702 is also configured to use the predicted label of the image to be annotated determined by calling the target image annotation model as the annotation label of the image to be annotated when the annotation accuracy of the target image annotation model reaches a preset threshold.

Optionally, the target image annotation model includes a first feature extraction layer, a second feature extraction layer and a feature matching layer. The first feature extraction layer includes an image feature output end and a language feature output end. The feature matching layer includes an image feature input end and a language feature output end. The language feature input end and the image feature output end are connected to the input end of the second feature extraction layer, the language feature output end is connected to the language feature input end, and the output end of the second feature extraction layer is connected to the image feature input end. The determination module 702 is configured to: perform feature extraction on each language description text in the language description text set through the first feature extraction layer to obtain a language feature set. The language feature set includes multiple sets of language features, and each set of language features is a language description. A language in the text collection describes the language features corresponding to the text. For each training sample image in the training image set, feature extraction is performed on the training sample image through the first feature extraction layer to obtain the global image features corresponding to the training sample image, and feature extraction is performed on the global image features through the second feature extraction layer to obtain The target image features are associated with the target image task. The target image features are matched with each group of language features in the language feature set through the feature matching layer to obtain the feature matching results. The feature matching results include the target image features and language. The feature similarity between each group of language features in the feature set, the label described by the target language description text is used as the predicted label of the training sample image, the target language description text is the feature similarity between the language feature set and the target image feature The language description text corresponding to the highest set of language features.

Optionally, the target image annotation model also includes a supervision module. The supervision module is provided with a loss function that matches the target image task. The input end of the supervision module is connected to the output end of the feature matching layer. The training module 703 is used to: calculate the loss value of the loss function based on the real labels and predicted labels of multiple training sample images in the training image set through the supervision module. Reversely transmit the gradient information of the loss function to the second feature extraction layer to adjust network parameters from the second feature extraction layer to the feature matching layer.

Optionally, an image annotation model framework is pre-stored in the computer device. The image annotation model framework includes a first feature extraction layer, a downstream task model head and a feature matching layer. The image feature output end is connected to the input end of the downstream task model head. The downstream The output end of the task model head is connected to the image feature input end of the feature matching layer. The downstream task model head includes multiple feature extraction layers that correspond to multiple image tasks one-to-one. The second feature extraction layer is the target in the downstream task model head. The feature extraction layer corresponding to the image task, wherein the image feature output end is configured to connect to one feature extraction layer in the downstream task model head at a time.

Optionally, the first feature extraction layer is implemented by a pre-trained visual language pre-training model.

Optionally, as shown in FIG. 8 , the image annotation device 700 further includes: a display module 704 .

Optionally, the display module 704 is configured to display a template setting prompt in response to receiving a startup instruction for the target image task. The template setting prompt is used to prompt the user to set a language description template corresponding to the target image task. The acquisition module 701 is used to generate a language description text for each type of label corresponding to the training image set according to the set language description template and label.

Optionally, the display module 704 is also configured to display the language description text after generating the language description text.

Optionally, the display module 704 is also used to display multiple image tasks, and the target image task is one of the multiple image tasks. The acquisition module 701 is configured to determine that a start instruction for the target image task is received in response to detecting a selection operation on the target image task.

Optionally, the plurality of image tasks include one or more of image classification, object detection, or action recognition.

Regarding the devices in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

The following is an example of the basic hardware structure involved in the embodiment of the present application.

For example, FIG. 9 is a schematic diagram of the hardware structure of an image annotation device provided by an embodiment of the present application. As shown in FIG. 9 , the image annotation device 900 includes a processor 901 and a memory 902 . The memory 901 and the memory 902 are connected through a bus 903 . Figure 9 illustrates the processor 901 and the memory 902 as independent of each other. Optionally, processor 901 and memory 902 are integrated together. Optionally, the image annotation device 900 in Figure 9 is any computer device with computing capabilities.

Among them, the memory 902 is used to store computer programs, and the computer programs include operating systems and program codes. Memory 902 is various types of storage media, such as read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (electrically erasable programmable read-only memory) , EEPROM), compact disc read-only memory (CD-ROM), flash memory, optical memory, register, optical disk storage, optical disk storage, magnetic disk or other magnetic storage device.

Among them, the processor 901 is a general-purpose processor or a special-purpose processor. Processor 901 may be a single-core processor or a multi-core processor. The processor 901 includes at least one circuit to execute the above image annotation method provided by the embodiment of the present application.

Optionally, the image annotation device 900 also includes a network interface 904, which is connected to the processor 901 and the memory 902 through the bus 903. The network interface 904 enables the image annotation device 900 to communicate with other devices.

Optionally, the image annotation device 900 also includes an input/output (I/O) interface 905 , which is connected to the processor 901 and the memory 902 through a bus 903 . The processor 901 can receive input commands or data through the I/O interface 905. The I/O interface 905 is used for the image annotation device 900 to connect to input devices, such as a keyboard, a mouse, etc. Optionally, in some possible scenarios, the above-mentioned network interface 904 and I/O interface 905 are collectively referred to as communication interfaces.

Optionally, the image annotation device 900 also includes a display 906 , which is connected to the processor 901 and the memory 902 through a bus 903 . The display 906 can be used to display intermediate results and/or final results generated by the processor 901 when executing the above method, such as displaying image tasks, template setting prompts, language description text, etc. In a possible implementation, the display 906 is a touch display screen to provide a human-computer interaction interface.

The bus 903 is any type of communication bus used to interconnect internal devices of the image annotation device 900 . For example, system bus. The embodiment of the present application takes the example of the interconnection of the above-mentioned devices inside the image annotation device 900 through the bus 903. Optionally, the above-mentioned devices inside the image annotation device 900 communicate with each other using other connection methods besides the bus 903, such as image The above-mentioned devices inside the annotation device 900 are interconnected through logical interfaces inside the image annotation device 900 .

The above-mentioned devices may be arranged on separate chips, or at least part or all of them may be arranged on the same chip. Whether each device is independently installed on different chips or integrated on one or more chips often depends on the needs of product design. The embodiments of this application do not limit the specific implementation forms of the above devices.

The image annotation device 900 shown in FIG. 9 is only exemplary. During the implementation process, the image annotation device 900 includes other components, which will not be listed one by one in this article. The image annotation device 900 shown in FIG. 9 can realize intelligent annotation of images by executing all or part of the steps of the method provided by the above embodiments.

Embodiments of the present application also provide a computer-readable storage medium. Instructions are stored on the computer-readable storage medium. When the instructions are executed by a processor, the image annotation method shown in Figure 2 is implemented.

An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the image annotation method shown in Figure 2 is implemented.

Those of ordinary skill in the art can understand that all or part of the steps to implement the above embodiments can be completed by hardware, or can be completed by instructing relevant hardware through a program. The program can be stored in a computer-readable storage medium. The above-mentioned The storage media mentioned can be read-only memory, magnetic disks or optical disks, etc.

In the embodiments of the present application, the terms "first", "second" and "third" are only used for description purposes and cannot be understood as indicating or implying relative importance.

The term "and/or" in this application is just an association relationship describing related objects, indicating that there can be three relationships, for example, A and/or B, which can mean: A exists alone, A and B exist simultaneously, alone There are three situations B. In addition, the character "/" in this article generally indicates that the related objects are an "or" relationship.

It should be noted that the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application, All are authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the image data involved in this application were obtained with full authorization.

The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the concepts and principles of the present application shall be included in the protection of the present application. within the range.

Claims (25)

1. An image annotation method for a computer device, the method comprising:

acquiring a training image set corresponding to a target image task and a language description text set corresponding to the training image set, wherein the training image set comprises a plurality of training sample images, each training sample image in the plurality of training sample images is marked with a real label, the language description text set comprises a plurality of language description texts, the plurality of language description texts are in one-to-one correspondence with a plurality of types of labels corresponding to the training image set, and each language description text is used for describing the semantics of one type of labels in the plurality of types of labels;

Invoking a target image annotation model corresponding to the target image task, and determining a prediction label of each training sample image in the training image set, wherein the prediction label of the training sample image is obtained by the target image annotation model based on a feature matching result of image features corresponding to the training sample image and language features corresponding to each language description text in the language description text set;

and training the target image annotation model according to errors between real labels and predicted labels of a plurality of training sample images in the training image set until the target image annotation model converges, wherein the target image annotation model is used for determining the annotation labels of the images to be annotated under the target image task.

2. The method of claim 1, wherein after the target image annotation model converges, the method further comprises:

invoking the target image annotation model, and determining prediction labels of a plurality of verification sample images in a verification image set, wherein the verification image set comprises the plurality of verification sample images, and each verification sample image in the plurality of verification sample images is annotated with a real label;

Determining the labeling accuracy of the target image labeling model according to the real labels and the prediction labels of the verification sample images;

when the labeling accuracy of the target image labeling model does not reach a preset threshold, executing one or more model training processes until the labeling accuracy of the target image labeling model reaches the preset threshold;

wherein the model training process comprises:

invoking the target image annotation model, and determining the prediction labels of a plurality of images to be annotated and the confidence of the prediction labels of the images to be annotated;

acquiring difficultly-marked images with confidence coefficient of the predictive label lower than a confidence coefficient threshold value from the plurality of images to be marked;

outputting the difficultly-marked image and a prediction label of the difficultly-marked image for manual correction;

in response to receiving a manual annotation result for the difficultly-annotated image, adding the difficultly-annotated image as a new training sample image to the training image set to obtain an updated training image set;

invoking the target image annotation model, and determining a prediction label of each training sample image in the updated training image set, wherein the prediction label of the training sample image is obtained based on feature matching results of image features corresponding to the training sample image and language features corresponding to each language description text in a language description text set corresponding to the updated training image set;

And training the target image annotation model according to errors between the real labels and the predicted labels of the plurality of training sample images in the updated training image set until the target image annotation model is converged again.

3. The method according to claim 2, wherein the method further comprises:

when the labeling accuracy of the target image labeling model reaches the preset threshold, the prediction label of the image to be labeled, which is determined by calling the target image labeling model, is used as the labeling label of the image to be labeled.

4. A method according to any one of claims 1 to 3, wherein the target image annotation model comprises a first feature extraction layer, a second feature extraction layer and a feature matching layer, the first feature extraction layer comprising an image feature output and a language feature output, the feature matching layer comprising an image feature input and a language feature input, the image feature output being connected to the input of the second feature extraction layer, the language feature output being connected to the language feature input, the output of the second feature extraction layer being connected to the image feature input; the step of calling the target image annotation model corresponding to the target image task and determining the prediction label of each training sample image in the training image set comprises the following steps:

Extracting the characteristics of each language description text in the language description text set through the first characteristic extraction layer to obtain a language characteristic set, wherein the language characteristic set comprises a plurality of groups of language characteristics, and each group of language characteristics is a language characteristic corresponding to one language description text in the language description text set;

for each training sample image in the training image set,

extracting the characteristics of the training sample image through the first characteristic extraction layer to obtain the global image characteristics corresponding to the training sample image,

performing feature extraction on the global image features through the second feature extraction layer to obtain target image features, wherein the target image features are associated with the target image task,

the feature matching layer is used for respectively carrying out feature matching on the target image features and the language features of each group in the language feature set to obtain feature matching results, the feature matching results comprise feature similarity between the target image features and each group of language features in the language feature set,

and taking the label described by the target language description text as a prediction label of the training sample image, wherein the target language description text is the language description text corresponding to a group of language features with highest feature similarity between the language feature set and the target image feature.

5. The method of claim 4, wherein the target image annotation model further comprises a supervision module, wherein a loss function matched with the target image task is set in the supervision module, an input end of the supervision module is connected with an output end of the feature matching layer, and training the target image annotation model according to errors between real labels and predicted labels of a plurality of training sample images in the training image set comprises:

calculating, by the supervision module, a loss value of the loss function based on real labels and predictive labels of a plurality of training sample images in the training image set;

and reversely transmitting gradient information of the loss function to the second feature extraction layer so as to adjust network parameters from the second feature extraction layer to the feature matching layer.

6. The method according to claim 4 or 5, wherein an image annotation model framework is pre-stored in the computer device, the image annotation model framework comprises the first feature extraction layer, a downstream task model head and the feature matching layer, the image feature output end is connected with the input end of the downstream task model head, the output end of the downstream task model head is connected with the image feature input end of the feature matching layer, the downstream task model head comprises a plurality of feature extraction layers corresponding to a plurality of image tasks one by one, the second feature extraction layer is a feature extraction layer corresponding to the target image task in the downstream task model head, and the image feature output end is configured to connect one feature extraction layer in the downstream task model head at a time.

7. The method according to any of claims 4 to 6, wherein the first feature extraction layer is implemented by a pre-trained visual language pre-training model.

8. The method according to any one of claims 1 to 7, wherein the obtaining the language description text set corresponding to the training image set includes:

in response to receiving a start instruction for the target image task, displaying a template setting prompt, wherein the template setting prompt is used for prompting a user to set a language description template corresponding to the target image task;

and generating a language description text according to the set language description template and the labels aiming at each type of labels corresponding to the training image set.

9. The method of claim 8, wherein the method further comprises:

after the language description text is generated, the language description text is displayed.

10. The method according to claim 8 or 9, characterized in that the method further comprises:

displaying a plurality of image tasks, wherein the target image task is one of the plurality of image tasks;

in response to detecting a selection operation of the target image task, it is determined that a start instruction for the target image task is received.

11. The method of claim 6 or 10, wherein the plurality of image tasks includes one or more of image classification, object detection, or motion recognition.

12. An image annotation device, the device comprising:

the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a training image set corresponding to a target image task and a language description text set corresponding to the training image set, the training image set comprises a plurality of training sample images, each training sample image in the plurality of training sample images is marked with a real label, the language description text set comprises a plurality of language description texts, the plurality of language description texts are in one-to-one correspondence with a plurality of types of labels corresponding to the training image set, and each language description text is used for describing the semantics of one type of labels in the plurality of types of labels;

the determining module is used for calling a target image annotation model corresponding to the target image task and determining a prediction label of each training sample image in the training image set, wherein the prediction label of the training sample image is obtained by the target image annotation model based on the feature matching result of the image feature corresponding to the training sample image and the language feature corresponding to each language description text in the language description text set;

The training module is used for training the target image annotation model according to errors between real labels and predicted labels of a plurality of training sample images in the training image set until the target image annotation model converges, and the target image annotation model is used for determining the annotation labels of the images to be annotated under the target image task.

13. The apparatus of claim 12, wherein the device comprises a plurality of sensors,

the determining module is further configured to invoke the target image annotation model after the target image annotation model converges, determine prediction labels of a plurality of verification sample images in a verification image set, where the verification image set includes the plurality of verification sample images, and each verification sample image in the plurality of verification sample images is annotated with a real label;

the determining module is further used for determining the labeling accuracy of the target image labeling model according to the real labels and the prediction labels of the verification sample images;

the training module is further used for executing one or more model training processes when the labeling accuracy of the target image labeling model does not reach a preset threshold value until the labeling accuracy of the target image labeling model reaches the preset threshold value;

Wherein the model training process comprises:

invoking the target image annotation model, and determining the prediction labels of a plurality of images to be annotated and the confidence of the prediction labels of the images to be annotated;

acquiring difficultly-marked images with confidence coefficient of the predictive label lower than a confidence coefficient threshold value from the plurality of images to be marked;

outputting the difficultly-marked image and a prediction label of the difficultly-marked image for manual correction;

in response to receiving a manual annotation result for the difficultly-annotated image, adding the difficultly-annotated image as a new training sample image to the training image set to obtain an updated training image set;

invoking the target image annotation model, and determining a prediction label of each training sample image in the updated training image set, wherein the prediction label of the training sample image is obtained based on feature matching results of image features corresponding to the training sample image and language features corresponding to each language description text in a language description text set corresponding to the updated training image set;

and training the target image annotation model according to errors between the real labels and the predicted labels of the plurality of training sample images in the updated training image set until the target image annotation model is converged again.

14. The apparatus of claim 13, wherein the device comprises a plurality of sensors,

and the determining module is further used for taking the prediction label of the image to be marked, which is determined by calling the target image marking model, as the marking label of the image to be marked when the marking accuracy of the target image marking model reaches the preset threshold.

15. The apparatus according to any one of claims 12 to 14, wherein the target image annotation model comprises a first feature extraction layer, a second feature extraction layer and a feature matching layer, the first feature extraction layer comprising an image feature output and a language feature output, the feature matching layer comprising an image feature input and a language feature input, the image feature output being connected to the input of the second feature extraction layer, the language feature output being connected to the language feature input, the output of the second feature extraction layer being connected to the image feature input; the determining module is used for:

extracting the characteristics of each language description text in the language description text set through the first characteristic extraction layer to obtain a language characteristic set, wherein the language characteristic set comprises a plurality of groups of language characteristics, and each group of language characteristics is a language characteristic corresponding to one language description text in the language description text set;

For each training sample image in the training image set,

extracting the characteristics of the training sample image through the first characteristic extraction layer to obtain the global image characteristics corresponding to the training sample image,

performing feature extraction on the global image features through the second feature extraction layer to obtain target image features, wherein the target image features are associated with the target image task,

the feature matching layer is used for respectively carrying out feature matching on the target image features and the language features of each group in the language feature set to obtain feature matching results, the feature matching results comprise feature similarity between the target image features and each group of language features in the language feature set,

and taking the label described by the target language description text as a prediction label of the training sample image, wherein the target language description text is the language description text corresponding to a group of language features with highest feature similarity between the language feature set and the target image feature.

16. The apparatus of claim 15, wherein the target image annotation model further comprises a supervision module, the supervision module having a loss function matched to the target image task disposed therein, an input of the supervision module connected to an output of the feature matching layer, and the training module configured to:

Calculating, by the supervision module, a loss value of the loss function based on real labels and predictive labels of a plurality of training sample images in the training image set;

and reversely transmitting gradient information of the loss function to the second feature extraction layer so as to adjust network parameters from the second feature extraction layer to the feature matching layer.

17. The apparatus according to claim 15 or 16, wherein an image annotation model framework is pre-stored in the computer device, the image annotation model framework comprising the first feature extraction layer, a downstream task model head and the feature matching layer, the image feature output being connected to an input of the downstream task model head, an output of the downstream task model head being connected to an image feature input of the feature matching layer, the downstream task model head comprising a plurality of feature extraction layers in one-to-one correspondence with a plurality of image tasks, the second feature extraction layer being a feature extraction layer in the downstream task model head corresponding to the target image task, wherein the image feature output is configured to connect one feature extraction layer in the downstream task model head at a time.

18. The apparatus according to any of the claims 15 to 17, wherein the first feature extraction layer is implemented by a pre-trained visual language pre-training model.

19. The apparatus according to any one of claims 12 to 18, further comprising: a display module;

the display module is used for responding to the receiving of the starting instruction aiming at the target image task, displaying a template setting prompt, wherein the template setting prompt is used for prompting a user to set a language description template corresponding to the target image task;

the acquisition module is used for generating a language description text according to the set language description template and the labels aiming at each type of labels corresponding to the training image set.

20. The apparatus of claim 19, wherein the device comprises a plurality of sensors,

the display module is further used for displaying the language description text after the language description text is generated.

21. The device according to claim 19 or 20, wherein,

the display module is further used for displaying a plurality of image tasks, and the target image task is one of the plurality of image tasks;

the acquisition module is used for responding to detection of the selection operation of the target image task and determining that a starting instruction aiming at the target image task is received.

22. The apparatus of claim 17 or 21, wherein the plurality of image tasks comprises one or more of image classification, object detection, or motion recognition.

23. A computer device, comprising: a processor and a memory;

the memory is used for storing a computer program, and the computer program comprises program instructions;

the processor is configured to invoke the computer program to implement the image labeling method according to any of claims 1 to 11.

24. A computer readable storage medium having instructions stored thereon which, when executed by a processor, implement the image annotation method according to any of claims 1 to 11.

25. A computer program product comprising a computer program which, when executed by a processor, implements the image annotation method according to any one of claims 1 to 11.