CN114842261A

CN114842261A - Image processing method, image processing device, electronic equipment and storage medium

Info

Publication number: CN114842261A
Application number: CN202210507448.7A
Authority: CN
Inventors: 贺俊东; 赵婷婷; 徐小钦; 胥晓
Original assignee: China West Normal University
Current assignee: China West Normal University
Priority date: 2022-05-10
Filing date: 2022-05-10
Publication date: 2022-08-02

Abstract

The embodiment of the invention discloses an image processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring an image to be classified, and determining a feature vector of the image to be classified; respectively inputting the images to be classified into each image classification model to obtain first attribute values corresponding to each actual classification category; determining the similarity between the feature vector and each reference category vector to obtain a second attribute value corresponding to the image to be classified and each reference category identifier; and determining the target class corresponding to the image to be classified based on the first attribute values and the second attribute values. The problem of among the prior art treat categorised image based on single image recognition model discernment, obtain the identification result, lead to the recognition efficiency low is solved, when realizing reducing cost consumption, reach the effect that improves image recognition precision and efficiency.

Description

Image processing method, image processing device, electronic equipment and storage medium

Technical Field

Embodiments of the present invention relate to computer processing technologies, and in particular, to an image processing method and apparatus, an electronic device, and a storage medium.

Background

In the field of image recognition technology, in order to improve the efficiency of image recognition, people tend to recognize an image to be recognized by using a trained image recognition model to obtain a recognition result.

However, if a high-precision recognition model is to be obtained, the model is often trained by using a large number of images with labeled categories, and in the process of model training, the model is only learned in the direction which is considered to be correct by the model. For example, if a picture of a cat is recognized as a dog with a high probability, the model will always learn such wrong knowledge, which results in a problem of low recognition accuracy when performing recognition based on a single trained image recognition model. Meanwhile, if a new category is added for identification, the model can automatically identify the new category, and the model needs to be retrained by using a large amount of new category sample images, so that the problem of low working efficiency is caused while a large amount of time and cost are consumed.

Disclosure of Invention

The embodiment of the invention provides an image processing method, an image processing device, electronic equipment and a storage medium, which are used for improving the accuracy of identifying different types of images and reducing the cost consumption.

In a first aspect, an embodiment of the present invention provides an image processing method, where the method includes:

acquiring an image to be classified, and determining a feature vector of the image to be classified;

respectively inputting the images to be classified into each image classification model to obtain first attribute values corresponding to each actual classification category;

determining the similarity between the feature vector and each reference category vector to obtain a second attribute value corresponding to the image to be classified and each reference category identifier; wherein the reference category vector corresponds to each reference category identification;

and determining the target class corresponding to the image to be classified based on the first attribute values and the second attribute values.

In a second aspect, an embodiment of the present invention further provides an image processing apparatus, including:

the characteristic vector determining module is used for acquiring an image to be classified and determining a characteristic vector of the image to be classified;

the first attribute value acquisition module is used for respectively inputting the images to be classified into each image classification model to obtain first attribute values corresponding to each actual classification category;

the second attribute value acquisition module is used for determining the similarity between the feature vector and each reference category vector to obtain a second attribute value corresponding to the image to be classified and each reference category identifier; wherein the reference category vector corresponds to each reference category identification;

and the target class determining module is used for determining a target class corresponding to the image to be classified based on each first attribute value and each second attribute value.

In a third aspect, an embodiment of the present invention further provides an electronic device, where the electronic device includes:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement an image processing method according to any one of the embodiments of the present invention.

In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the image processing method according to any one of the embodiments of the present invention.

The technical scheme of the embodiment of the invention comprises the steps of obtaining the image to be classified, determining the characteristic vector of the image to be classified, respectively inputting the image to be classified into each image classification model to obtain the first attribute value corresponding to each actual classification category, determining the similarity between the characteristic vector and each reference category vector to obtain the second attribute value corresponding to the identifier of each reference category of the image to be classified, and determining the target category corresponding to the image to be classified based on each first attribute value and each second attribute value, thereby solving the problem of low identification efficiency caused by identifying the image to be classified based on a single image identification model in the prior art and obtaining the identification result, realizing respectively identifying the image to be classified based on each image classification model to obtain the first attribute value corresponding to each actual classification category, and simultaneously based on the similarity between each reference category vector and the characteristic vector of the image to be classified, the second attribute values corresponding to the reference category identifications are obtained, then the target categories corresponding to the images to be classified are comprehensively determined based on the first attribute values and the second attribute values, so that the images are comprehensively identified based on the image classification models and the reference category vectors, the accuracy of identifying the images of different categories is improved, the type images which cannot be identified by the image classification models can be determined based on the reference category vectors without reconstructing the models, the cost consumption is reduced, and the technical effects of improving the image identification precision and efficiency are achieved.

Drawings

In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, a brief description is given below of the drawings used in describing the embodiments. It should be clear that the described figures are only views of some of the embodiments of the invention to be described, not all, and that for a person skilled in the art, other figures can be derived from these figures without inventive effort.

Fig. 1 is a flowchart of an image processing method according to an embodiment of the present invention;

fig. 2 is a flowchart of an image processing method according to a second embodiment of the present invention;

fig. 3 is a block diagram of an image processing apparatus according to a fourth embodiment of the present invention;

fig. 4 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present invention.

Detailed Description

The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.

Example one

Fig. 1 is a flowchart of an image processing method according to a first embodiment of the present invention, where the present embodiment is applicable to image classification, and the method may be executed by an image processing apparatus according to the first embodiment of the present invention, where the apparatus may be implemented in a software and/or hardware manner, and optionally, the method is implemented by an electronic device, where the electronic device may be a mobile terminal, a PC end, a server end, or the like. The apparatus can be configured in a computing device, and the image processing method provided in this embodiment specifically includes the following steps:

s110, obtaining an image to be classified, and determining a feature vector of the image to be classified.

The image to be classified may be an image to be recognized, which may be an image collected by a camera device, or an image pre-stored in a storage space, and accordingly, the image to be classified may be an animal image, a focus image, a person image, or the like. The image to be classified may also be an image corresponding to the image classification model, for example, if the image classification model is an animal recognition model, the image to be classified may be an animal image, and if the image classification model is a human recognition model, the image to be classified may be a human image. It should be noted that, the image to be classified may be one image or multiple images, which is not limited herein, for example, in an actual scene, if a large number of labeled sample images are obtained, at this time, as many images to be classified as possible may be obtained, so that the image to be classified is identified based on the technical solution, and a sample image labeled with a good category is obtained. The feature vector may be used to represent uniqueness of the image, and optionally, the feature vector may be obtained by extracting pixel point information of the image, or the feature vector may be obtained by extracting key feature point information of the image.

In practical application, any image needing to be identified can be used as the image to be classified. The image corresponding to the image classification model can also be used as the image to be classified, so that the image to be classified is identified based on the image classification model. The feature extraction algorithm can be utilized to extract features of the image to be classified to obtain feature vectors corresponding to the image to be classified, so that the category of the image to be classified is determined comprehensively based on the feature vectors, and the accuracy of image identification is improved.

And S120, respectively inputting the images to be classified into each image classification model to obtain first attribute values corresponding to each actual classification category.

The image classification model may be a model trained in advance and used for performing class identification on the image. The actual classification category refers to a category that can be recognized by the image classification model, for example, if the image classification model is an animal recognition model, the actual classification category may be cauchy, chai-dog, hastelli, golden hair, and the like.

In practical applications, in order to improve the accuracy of image recognition and prevent the occurrence of the problem of model validation Bias, a plurality of independent image classification models can be trained in advance based on images of a large number of actual classification classes, and each image classification model has the recognition capability for each image of the actual classification class. And when the images to be classified are respectively input into the image classification models, finally obtaining confidence degrees corresponding to the actual classification categories as first attribute values. For example, when the image a to be classified is input into the model1, the model2, and the model3, the first attribute values corresponding to the actual classification categories, such as cauda, chai, hastelli, and golden hair, can be obtained. So that it is subsequently determined to which actual classification category the image a to be classified belongs by comparing the magnitudes of the respective first attribute values.

It should be noted that, when the images to be classified are respectively input into each image classification model, each image classification model can identify the images, each model can output the confidence corresponding to each actual classification category, correspondingly, there may be a plurality of confidence of each actual classification category, and a plurality of confidences included in a certain actual classification category can be fused to obtain a comprehensive confidence corresponding to the actual classification category as a first attribute value, and correspondingly, a first attribute value corresponding to each actual classification category can be obtained.

Optionally, the step of inputting the image to be classified into each image classification model respectively to obtain a first attribute value corresponding to each actual classification category includes: inputting the image to be classified into the current image classification model aiming at each image classification model to obtain the attribute value to be used of at least one actual classification category corresponding to the current image classification model; and determining at least one attribute value to be used corresponding to the current actual classification category according to each actual classification category, and carrying out mean processing on the at least one attribute value to be used to obtain a first attribute value corresponding to the current actual classification category.

It should be noted that the processing of each image classification model is the same, and one of the image classification models is taken as the current image classification model for description.

In practical applications, the image to be classified may be input into a current image classification model, and the current image classification model outputs a confidence corresponding to at least one actual classification category as an attribute value to be used, for example, when the image a to be classified is input into the model1, the model1 may output the attribute value to be used corresponding to each actual classification category, such as caudad, chai, hastelli, and golden hair. Correspondingly, each image classification model can output attribute values to be used corresponding to each actual classification category. It should be noted that the manner of determining the first attribute value corresponding to each actual classification category is the same, and one of the actual classification categories is taken as the current actual classification category for explanation. At least one attribute value to be used corresponding to the current actual classification category can be obtained, further, the average value of the attribute values to be used can be obtained, and the average value can be used as a first attribute value corresponding to the current actual classification category. For example, for the actual classification category a, the value of the attribute to be used output by model1 is 0.3, the value of the attribute to be used output by model2 is 0.4, and the value of the attribute to be used output by model3 is 0.5, then the first attribute value of the actual classification category a may be (0.3+0.4+0.5) ÷ 3 — 0.4. Correspondingly, a first attribute value corresponding to each actual classification category can be obtained. So that it is subsequently determined to which actual classification category the image a to be classified belongs by comparing the magnitudes of the respective first attribute values.

S130, determining the similarity between the feature vector and each reference category vector to obtain a second attribute value corresponding to the image to be classified and each reference category identifier.

Wherein the reference category vector corresponds to each reference category identification. The reference class identification may be used to characterize the uniqueness of the class, e.g., the Husky class may be denoted by A and the DeMux class may be denoted by B. It should be noted that the category corresponding to the reference category identifier may be consistent with the actual classification category or may not be consistent with the actual classification category, for example, the actual classification category is hastelly or golden hair, and the category corresponding to the reference category identifier may include hastelly or grazing. The image classification model can be recognized based on the technical scheme, such as grazing, without retraining the model for recognizing the grazing image, and the recognition efficiency is improved. The recognition accuracy of the HashChi type images in the image classification model can be enhanced based on the technical scheme. The vector-form representation of the reference class identifier is a reference class vector.

In practical applications, the feature vectors of the image to be classified may be compared with the reference class vectors corresponding to each reference class identifier, for example, the similarity between the vectors is calculated as the second attribute value corresponding to each reference class identifier.

It should be noted that before determining the similarity between the feature vector and each reference category vector and obtaining the second attribute values corresponding to the image to be classified and each reference category identifier, the reference category vector corresponding to a reference category identifier may be calculated in advance by using the image corresponding to a certain reference category identifier. For example, the feature vector of each image may be extracted, and then the feature vectors are subjected to fusion processing to obtain a vector that can represent the reference category identifier, and the vector is used as the reference category vector. Accordingly, a reference category vector corresponding to each reference category identification may be obtained.

Optionally, the method further includes: aiming at each reference category identification, acquiring at least one first original image corresponding to the current reference category identification, and respectively determining a feature vector corresponding to the first original image; and determining a reference category vector corresponding to each reference category identifier by performing mean processing on at least one feature vector corresponding to the same reference category identifier.

It should be noted that the way of determining the reference category vector corresponding to each reference category identifier is the same, and one of the reference category identifiers is taken as the current reference category identifier for explanation. The first original image may be an image of a marked category.

Specifically, in order to improve the accuracy of determining the reference category vector and further improve the accuracy of performing image recognition based on the reference category vector, as many first original images as possible corresponding to the current reference category identifier may be obtained, where the first original images are images marked with the reference category identifier. For example, the reference category identification is a category identification of a cat, and the first original image is an image of a cat category. Further, feature vectors of all first original images corresponding to the current reference category identifier may be extracted by using a feature extraction algorithm, and then, an average value may be obtained by performing an average value process on each feature vector, and the average value may be used as a reference category vector corresponding to the current reference category identifier. Accordingly, the reference category vector corresponding to each reference category identifier may be determined based on the above-mentioned manner of determining the reference category vector, so that the category of the image to be classified is subsequently determined based on the reference category vector.

It should be noted that S120 to S130 may be executed sequentially or in parallel, and a specific execution order is not limited, and the order is only an order explaining a technical solution in each step, and is not an execution order of each step.

S140, determining the target class corresponding to the image to be classified based on the first attribute values and the second attribute values.

Specifically, each first attribute value and each second attribute value may be compared, and the classification category with the largest attribute value is used as the target category corresponding to the image to be classified. Or normalizing each first attribute value and each second attribute value to obtain an attribute value after normalization processing of each attribute value, and taking the classification category with the maximum attribute value as a target category corresponding to the image to be classified.

It should be noted that there may be attribute values corresponding to the same classification category in each first attribute value and each second attribute value, for example, a first attribute value corresponding to a husky is obtained based on an image classification model, and a second attribute value corresponding to a husky is also obtained based on a reference category vector of the husky. In order to improve the accuracy of image identification, the two attribute values can be subjected to fusion processing, and the attribute values of the images to be classified belonging to the Husky category are comprehensively determined. Correspondingly, the attribute values after the fusion of the first attribute values and the second attribute values corresponding to all classification categories can be obtained, and further the final category of the image to be classified is determined based on the fused attribute values.

Optionally, determining a target category corresponding to the image to be classified based on each first attribute value and each second attribute value, including: determining a set comprising actual classification categories and reference category identifications, wherein the set comprises at least one category element; and aiming at each class element, determining an element type corresponding to the current class element, and processing a first attribute value and a second attribute value of the current class element according to the element type to determine a target class.

In practical application, each actual classification category and each reference category identifier may be merged to obtain a union set, and each category in the set may be used as a category element. For example, the actual classification category is husky, pasture, and golden hair, the category corresponding to the reference category identifier is husky, pasture, and faggish, and then the set includes four category elements such as husky, pasture, golden hair, and faggish. It should be noted that, the manner of determining the element type corresponding to each category element is the same, and one of the category elements may be used as the current category element for description. An element type corresponding to the current category element may be determined. The first attribute value and the second attribute value of the current category element can be processed according to the element type, and the target category is determined.

Optionally, the implementation process of determining the element type corresponding to the current category element may be: if the attribute value of the current category element comprises a first attribute value or a second attribute value, determining that the element type corresponding to the current category element is a first type; or if the attribute value of the current category element includes the first attribute value and the second attribute value, determining that the element type corresponding to the current category element is the second type.

For example, assuming that the current class element is a hesky class, if the first attribute value corresponding to the grazing class is obtained only based on the image classification model, or the second attribute value corresponding to the grazing class is obtained only based on the reference class vector, the element type corresponding to the grazing class may be the first type; if the first attribute value corresponding to the Hashquay class is obtained based on the image classification model, and the second attribute value corresponding to the Hashquay class is also obtained based on the reference class vector, the element type corresponding to the Hashquay class can be the second type.

It should be noted that, when each element type is a first type, it may be stated that each actual classification type is different from each reference type identifier, and no repeated type occurs, at this time, each first attribute value and each second attribute value may be normalized to obtain an attribute value after each attribute value normalization processing, and the classification type with the largest attribute value may be used as a target type corresponding to an image to be classified. When each element type includes the second type, it can be stated that there is a repeated type in each actual classification type and each reference type identifier, the repeated type, that is, the first attribute value and the second attribute value corresponding to the second type of category element, can be subjected to fusion processing to obtain the attribute value capable of comprehensively evaluating the category element, and then the target type corresponding to the image to be classified can be determined based on the attribute value of the category element and the attribute value of the first type of category element.

Optionally, the processing the first attribute value and the second attribute value of the current category element according to the element type to determine the target category includes: if all the element types are the first type, normalizing the first attribute values and the second attribute values to obtain third attribute values corresponding to all the category elements, and determining target categories corresponding to the images to be classified based on the third attribute values; or, if each element type comprises a second type, performing mean processing on the first attribute value and the second attribute value of the current type element to obtain a fourth attribute value corresponding to the current type element, and determining a target type corresponding to the image to be classified based on the fourth attribute value corresponding to each type element.

Specifically, if each element type is the first type, normalization processing may be performed on all the first attribute values and all the second attribute values, so that the sum of all the first attribute values and all the second attribute values is 1, and then the attribute value corresponding to each category element, that is, the third attribute value, may be obtained. The classification category corresponding to the maximum value in each third attribute value may be used as the target category of the image to be classified. Or, if each element type includes the second type, the first attribute value and the second attribute value corresponding to the current type element may be processed as an average value, and if the current type element is the first type, the corresponding first attribute value or the second attribute value is 0. For example, if the current category element a is of the first type, the first attribute value is 0.2, and the second attribute value is 0.3, the average value is (0.2+0.3) ÷ 2 ═ 0.25. The mean value may be used as a fourth attribute value of the current category element a, correspondingly, a fourth attribute value corresponding to each category element may be obtained, and a classification category corresponding to a maximum value among the fourth attribute values may be used as a target category of the image to be classified.

The technical scheme of this embodiment includes obtaining an image to be classified, determining a feature vector of the image to be classified, inputting the image to be classified into each image classification model to obtain a first attribute value corresponding to each actual classification category, determining a similarity between the feature vector and each reference category vector to obtain a second attribute value corresponding to the image to be classified and each reference category identifier, determining a target category corresponding to the image to be classified based on each first attribute value and each second attribute value, solving the problem of low recognition efficiency caused by recognition result obtained by recognizing the image to be classified based on a single image recognition model in the prior art, realizing respective recognition of the image to be classified based on each image classification model to obtain a first attribute value corresponding to each actual classification category, and simultaneously based on the similarity between each reference category vector and the feature vector of the image to be classified, the second attribute values corresponding to the reference category identifications are obtained, then the target categories corresponding to the images to be classified are comprehensively determined based on the first attribute values and the second attribute values, so that the images are comprehensively identified based on the image classification models and the reference category vectors, the accuracy of identifying the images of different categories is improved, the type images which cannot be identified by the image classification models can be determined based on the reference category vectors without reconstructing the models, the cost consumption is reduced, and the technical effects of improving the image identification precision and efficiency are achieved.

Example two

Fig. 2 is a flowchart of an image processing method according to a second embodiment of the present invention, and based on the foregoing embodiment, the method further includes training to obtain the image classification models. The specific implementation manner can be referred to the technical scheme of the embodiment. The technical terms that are the same as or corresponding to the above embodiments are not repeated herein.

As shown in fig. 2, the method specifically includes the following steps:

s210, obtaining a training sample set corresponding to each classification model to be trained.

It should be noted that, in order to avoid the problem of a validation bias in identifying an image by a single model and solve the problem of difficulty in obtaining a labeled sample in the conventional method, a small number of labeled samples are used as a training set, the training set is divided into a plurality of parts, corresponding classification models are trained respectively and independently based on the training set of each part to obtain a classification model after corresponding training, unlabeled samples are processed based on the classification model after training to obtain labeled samples, labeled samples with high confidence degrees are selected from the labeled samples and added into the training set, and the trained classification models are repeatedly trained, so that the model identification accuracy is improved, and the sample labeling efficiency and the cost are improved.

Wherein, the classification model to be trained can be understood as the classification model to be trained. The training sample set comprises at least one training sample, and the training sample comprises second original images corresponding to each actual classification category. The second original image may be an image of a marked category. For example, the actual classification category is a Husky category, and the second original image is an image of the Husky category.

Specifically, a small number of second original images corresponding to each actual classification category may be obtained, each second original image is divided into a plurality of parts, each second original image may be used as a training sample set corresponding to a classification model to be trained, and the training sample set trains the corresponding classification model to be trained.

S220, aiming at each classification model to be trained, taking a second original image in a current training sample corresponding to the current classification model to be trained as the input of the current classification model to be trained, taking an actual classification category as the output of the current classification model to be trained, training the current classification model to be trained, and obtaining the trained current classification model to be trained.

It should be noted that the processing manner of each to-be-trained classification model is the same, and one of the to-be-trained classification models may be taken as the current to-be-trained classification model for example.

Specifically, the current classification model to be trained may be trained according to each training sample in a training sample set corresponding to the current classification model to be trained, so as to obtain a trained current classification model to be trained. It should be noted that the processing manner of each training sample is the same, and the processing of one of the training samples is taken as an example for description. The second original image in the current training sample can be input to the current classification model to be trained, the model can perform learning processing on the second original image, can output the category corresponding to the second original image, further, the algorithm can be used for carrying out loss processing on the output category and the actual classification category of the expected output to obtain a loss value, so as to modify the model parameters in the current classification model to be trained based on the loss value, train the current classification model to be trained, the penalty function convergence of the current classification model to be trained can be used as a training target, e.g., the training error of the penalty function, that is, the loss parameter is used as a condition for detecting whether the loss function reaches convergence currently, for example, whether the training error is smaller than a preset error or whether the error variation trend tends to be stable, or whether the current iteration number is equal to the preset number. If the detection reaches the convergence condition, for example, the training error of the loss function is smaller than the preset error or the error change tends to be stable, indicating that the training of the current classification model to be trained is completed, at this time, the iterative training may be stopped. If the current condition is not met, the training sample can be further obtained to train the current classification model to be trained until the training error of the loss function is within the preset range. When the training error of the loss function reaches convergence, the current classification model to be trained can be considered to be well trained, and the trained current classification model to be trained can be obtained. Therefore, when a certain image to be classified is input, the trained current classification model to be trained can output the classification of the image to be classified.

And S230, taking the trained current classification model to be trained as the current classification model to be trained again and training until the loss function of the current classification model to be trained is converged to obtain a corresponding image classification model.

In practical application, in order to improve the accuracy of model training, a labeled sample, that is, a second original image, may be continuously obtained, so as to continuously train the trained current classification model to be trained based on the second original image until the loss function of the current classification model to be trained converges, thereby obtaining a corresponding image classification model. In order to improve the recognition accuracy of the image classification model, the image classification model can be used as the current classification model to be trained again, the step of training the current classification model to be trained is repeatedly executed to obtain the image classification model, and the obtained image classification model can be used as the final image classification model until the recognition state of the image classification model meets the preset requirement.

It should be noted that, in order to reduce the cost of acquiring labeled samples, after a trained current classification model to be trained is obtained, unlabeled sample images may be processed based on the trained current classification model to be trained to obtain categories corresponding to the unlabeled sample images, and then labeled images may be obtained based on the categories and the corresponding unlabeled sample images, and the labeled images may be used as a training set, or images with confidence levels higher than a preset threshold value in the labeled images may be used as a training set, and the current classification model to be trained is continuously trained.

Optionally, the trained classification model to be trained at present is used as the classification model to be trained again and is trained until the loss function of the classification model to be trained at present converges, so as to obtain a corresponding image classification model, including: acquiring at least one image to be marked; inputting each image to be labeled to a trained current classification model to be trained, outputting an actual classification class corresponding to each image to be labeled and a corresponding attribute value to be processed, and taking the trained current classification model to be trained as the current classification model to be trained again; determining a target annotation image from each image to be annotated based on a preset attribute value threshold, each attribute value to be processed and a corresponding actual classification category; and updating the training sample set based on each target labeling image, and re-executing the step of training the current classification model to be trained based on the updated training sample set until the loss function of the current classification model to be trained is converged to obtain the corresponding image classification model.

Wherein, the image to be marked can be understood as the image to be marked.

In this embodiment, the images to be labeled can be obtained as many and as abundant as possible, and then each image to be labeled can be used as an input of a trained current classification model to be trained, and the model outputs an actual classification category corresponding to each image to be labeled and a corresponding confidence coefficient, that is, an attribute value to be processed. The image to be labeled, of which the attribute value to be processed is greater than the preset attribute value threshold, can be used as the image to be processed, and each image to be processed and the corresponding actual classification category can be labeled to obtain an image labeled with the image to be processed, and the image labeled with the image to be processed is used as a target labeling image. Each target labeling image can be added into a historical training sample set corresponding to the current classification model to be trained to obtain a new training sample set, and the step of training the current classification model to be trained is executed again based on the new training sample set until the loss function of the current classification model to be trained is converged to obtain a corresponding image classification model, so that when a certain image to be classified is input, the trained image classification model can accurately output the target class of the image to be classified.

S240, obtaining an image to be classified, and determining a feature vector of the image to be classified.

And S250, respectively inputting the images to be classified into each image classification model to obtain first attribute values corresponding to each actual classification category.

And S260, determining the similarity between the feature vector and each reference category vector to obtain a second attribute value corresponding to the image to be classified and each reference category identifier.

S270, determining a target class corresponding to the image to be classified based on the first attribute values and the second attribute values.

According to the technical scheme, a small number of marked samples are used as a training set to train a plurality of classification models to be trained, the problem that a single model is prone to identifying images is solved, corresponding classification models are trained respectively and independently based on the training sets of all parts, classification models after corresponding training are obtained, unmarked samples are processed based on the trained classification models, marked samples are obtained, marked samples are selected from the marked samples, marked samples with high confidence degrees are selected to be added into the training set, the trained classification models are repeatedly trained, and the sample marking efficiency and the sample marking cost are improved while the model identification precision is improved.

EXAMPLE III

As an alternative embodiment of the above embodiment, in order to make the technical solutions of the embodiments of the present invention further clear to those skilled in the art, a specific application scenario example is given. Specifically, the following details can be referred to.

For example, assuming that the number of image classification models is 3, when training the three image classification models, the three image classification models may include at least one second original image (LabelDataSset) as a total training sample set (TrainDataSset), and the TrainDataSset is divided into three parts to obtain TrainDataSset1, TrainDataSset2, and TrainDataSset 3. Each training sample set is used as a training sample set of a classification model to be trained, for example, the classification model to be trained may be a model1, a model2, or a model 3. Training samples (train _ sample) in each training sample set may be respectively input into a corresponding classification model to be trained, for example, train _ sample1 is input into model1, train _ sample2 is input into model2, train _ sample3 is input into model3, and the model is trained to obtain trained models 1, model2, and model 3. Further, each image to be labeled (UnLabelDataSset) can be input into the trained classification model to be trained to obtain a labeled image, and a label sample with the confidence coefficient larger than a preset attribute threshold value can be selected from the labeled image and added into the total training sample set (TrainDataSset). Repeatedly executing the steps of obtaining TrainDataSset1, TrainDataSset2 and TrainDataSset3 based on the division of the TrainDataSset, and training the trained model1, model2 and model3 based on the TrainDataSset1, the TrainDataSset2 and the TrainDataSset3 until obtaining each trained image classification model, and realizing pseudo codes of the flow, wherein the steps are as follows:

#TrainDataSset＝LabelDataSset

For epoch in Epoch:

# splitting the Total training sample set into triplicates

TrainDataSset1,TrainDataSset2,TrainDataSset3＝split(TrainData Sset)

For train_sample1,train_sample2,train_sample3 in TrainDataSset1，TrainDataSset2，TrainDataSset3:

model1＝train_model(model1,train_sample1)

model2＝train_model(model2,train_sample2)

model3＝train_model(model3,train_sample3)

Prediction of unlabeled data set Using the current model, the New Total training sample set, TrainedDataSet ═ LabelDataSet + FindNewData (model1, model2, model3, UnLabelDataSet)

On the basis of the above scheme, at least one first original image corresponding to the current reference category identifier may also be obtained, feature vectors corresponding to the first original images are respectively determined, and reference category vectors corresponding to the reference category identifiers are determined by performing mean processing on at least one feature vector corresponding to the same reference category identifier, so as to implement a pseudo code of the flow, as shown below:

Animal2Embedding＝{}

# traverse the first raw image collected

For image,label in InternetImages:

# adding the feature vector corresponding to the first original image to the corresponding reference class identification

Animal2Embedding[label].append(model(image))

For animal in Animal2Embedding:

The mean value of each feature vector corresponding to the same reference category identifier is used as the reference category vector corresponding to each reference category identifier

Animal2Embedding[label]＝mean(Animal2Embedding[label])

On the basis of the scheme, a trained model can be used for calculating a reference class vector of a reference class identifier, and the reference class vector is used for enhancing the effect of the model during reasoning. The overall process of comprehensively determining the target class of the image to be classified by combining the reference class vector and the image classification model is as follows:

prob＝model(x)+Top1Sim(x,model,Animal2Embedding)

wherein x represents an image to be classified, model (x) represents a first attribute value output by an image classification model, Animal2Embedding represents a reference class vector, Top1Sim (x, model, Animal2Embedding) represents the similarity between a feature vector of the image to be classified and the reference class vector, namely a second attribute value, and prob represents a comprehensive evaluation result of the image to be classified. In an actual scene, when reference category vectors of a plurality of reference category identifiers exist, a second attribute value corresponding to the reference category identifier can be obtained by calculating the similarity between the feature vector of the image to be classified and each reference category vector. When the actual classification category corresponding to the image classification model is different from the reference category identification, the first attribute value and each second attribute value can be normalized through SoftMax to obtain a third attribute value corresponding to each category, the maximum value in the third attribute values can be used as a probability value representing a target category, and accordingly, the target category of the image to be classified is obtained.

According to the technical scheme of the embodiment, the images to be classified are respectively identified based on the image classification models to obtain the first attribute values corresponding to the actual classification categories, the second attribute values corresponding to the reference category identifications are obtained based on the similarity between the reference category vectors and the feature vectors of the images to be classified, the target categories corresponding to the images to be classified are comprehensively determined based on the first attribute values and the second attribute values, the images are comprehensively identified based on the image classification models and the reference category vectors, the accuracy of identifying the images of different categories is improved, the models do not need to be reconstructed, the type images which cannot be identified by the image classification models can be determined based on the reference category vectors, the cost consumption is reduced, and the technical effects of improving the image identification precision and efficiency are achieved.

Example four

Fig. 3 is a block diagram of an image processing apparatus according to a fourth embodiment of the present invention. The device includes: a feature vector determination module 310, a first attribute value acquisition module 320, a second attribute value acquisition module 330, and a target class determination module 340.

The feature vector determining module 310 is configured to obtain an image to be classified and determine a feature vector of the image to be classified;

a first attribute value obtaining module 320, configured to input the image to be classified into each image classification model respectively, so as to obtain a first attribute value corresponding to each actual classification category;

a second attribute value obtaining module 330, configured to determine a similarity between the feature vector and each reference category vector, to obtain a second attribute value corresponding to the image to be classified and each reference category identifier; wherein the reference category vector corresponds to each reference category identification;

and the target class determining module 340 is configured to determine a target class corresponding to the image to be classified based on each first attribute value and each second attribute value.

On the basis of the foregoing apparatus, optionally, the first attribute value obtaining module 320 includes an attribute value determining unit to be used and a first attribute value determining unit.

The to-be-used attribute value determining unit is used for inputting the to-be-classified images into the current image classification model aiming at each image classification model to obtain the to-be-used attribute value of at least one actual classification category corresponding to the current image classification model;

and the first attribute value determining unit is used for determining at least one attribute value to be used corresponding to the current actual classification category aiming at each actual classification category, and carrying out mean processing on the at least one attribute value to be used to obtain a first attribute value corresponding to the current actual classification category.

On the basis of the above apparatus, optionally, the apparatus includes a reference category vector determination module, where the reference category vector determination module includes a feature vector determination unit and a reference category vector determination unit.

The characteristic vector determining unit is used for acquiring at least one first original image corresponding to the current reference category identification aiming at each reference category identification and respectively determining characteristic vectors corresponding to the first original images;

and the reference category vector determining unit is used for determining the reference category vectors corresponding to the reference category identifications by performing mean processing on at least one feature vector corresponding to the same reference category identification.

On the basis of the foregoing apparatus, optionally, the object class determination module 340 includes a set determination unit and an object class determination unit.

The device comprises a set determining unit, a classification determining unit and a classification determining unit, wherein the set determining unit is used for determining a set comprising each actual classification category and each reference category identification, and the set comprises at least one category element;

and the target class determining unit is used for determining the element type corresponding to the current class element according to each class element, processing the first attribute value and the second attribute value of the current class element according to the element type and determining the target class.

On the basis of the above apparatus, optionally, the object class determination module 340 further includes an element type determination unit.

An element type determining unit, configured to determine, if the attribute value of the current category element includes a first attribute value or a second attribute value, that an element type corresponding to the current category element is a first type; or the like, or, alternatively,

and if the attribute value of the current category element comprises a first attribute value and a second attribute value, determining that the element type corresponding to the current category element is a second type.

On the basis of the above device, optionally, the object class determination unit includes an object class determination subunit.

A target category determining subunit, configured to, if the element types are all of the first type, perform normalization processing on each first attribute value and each second attribute value to obtain a third attribute value corresponding to each category element, and determine, based on each third attribute value, a target category corresponding to the image to be classified; or the like, or, alternatively,

and if each element type comprises a second type, performing mean processing on the first attribute value and the second attribute value of the current type element to obtain a fourth attribute value corresponding to the current type element, and determining a target type corresponding to the image to be classified based on the fourth attribute value corresponding to each type element.

On the basis of the device, optionally, the device further comprises an image classification model obtaining module, wherein the image classification model obtaining module comprises a training sample set obtaining unit, a to-be-trained classification model training unit and an image classification model determining unit.

The training sample set acquisition unit is used for acquiring a training sample set corresponding to each classification model to be trained; the training sample set comprises at least one training sample, and the training sample comprises second original images corresponding to each actual classification category;

the training unit of the classification model to be trained is used for taking a second original image in a current training sample corresponding to the current classification model to be trained as the input of the current classification model to be trained and taking an actual classification category as the output of the current classification model to be trained aiming at each classification model to be trained, and training the current classification model to be trained to obtain the trained current classification model to be trained;

and the image classification model determining unit is used for taking the trained current to-be-trained classification model as the current to-be-trained classification model again and training the model until the loss function of the current to-be-trained classification model is converged to obtain a corresponding image classification model.

On the basis of the above device, optionally, the image classification model determining unit includes an image to be labeled obtaining subunit, an attribute value output subunit to be processed, a target labeled image determining subunit, and an image classification model determining subunit.

The image to be annotated acquiring subunit is used for acquiring at least one image to be annotated;

the to-be-processed attribute value output subunit is used for inputting each to-be-labeled image into the trained current to-be-trained classification model, outputting the actual classification category corresponding to each to-be-labeled image and the corresponding to-be-processed attribute value, and taking the trained current to-be-trained classification model as the current to-be-trained classification model again;

the target annotation image determining subunit is used for determining a target annotation image from each image to be annotated based on a preset attribute value threshold, each attribute value to be processed and a corresponding actual classification category;

and the image classification model determining subunit is used for updating the training sample set based on each target labeling image, and re-executing the step of training the current classification model to be trained based on the updated training sample set until the loss function of the current classification model to be trained is converged to obtain the corresponding image classification model.

The image processing device provided by the embodiment of the invention can execute the image processing method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.

It should be noted that, the units and modules included in the apparatus are merely divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be implemented; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the embodiment of the invention.

EXAMPLE five

Fig. 4 is a schematic structural diagram of an electronic device according to a fifth embodiment of the present invention. FIG. 4 illustrates a block diagram of an exemplary electronic device 40 suitable for use in implementing embodiments of the present invention. The electronic device 40 shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiment of the present invention.

As shown in fig. 4, electronic device 40 is in the form of a general purpose computing device. The components of electronic device 40 may include, but are not limited to: one or more processors or processing units 401, a system memory 402, and a bus 403 that couples the various system components (including the system memory 402 and the processing unit 401).

Bus 403 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, micro-channel architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

Electronic device 40 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by electronic device 40 and includes both volatile and nonvolatile media, removable and non-removable media.

The system memory 402 may include computer system readable media in the form of volatile memory, such as Random Access Memory (RAM)404 and/or cache memory 405. The electronic device 40 may further include other removable/non-removable, volatile/nonvolatile computer system storage media. By way of example only, storage system 406 may be used to read from and write to non-removable, nonvolatile magnetic media (not shown in FIG. 4, and commonly referred to as a "hard drive"). Although not shown in FIG. 4, a magnetic disk drive for reading from and writing to a removable, nonvolatile magnetic disk (e.g., a "floppy disk") and an optical disk drive for reading from or writing to a removable, nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 403 by one or more data media interfaces. Memory 402 may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the invention.

A program/utility 408 having a set (at least one) of program modules 407 may be stored, for example, in the memory 402, such program modules 407 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination of which may comprise an implementation of a network environment. Program modules 407 generally perform the functions and/or methods of the described embodiments of the invention.

The electronic device 40 may also communicate with one or more external devices 409 (e.g., keyboard, pointing device, display 410, etc.), with one or more devices that enable a user to interact with the electronic device 40, and/or with any devices (e.g., network card, modem, etc.) that enable the electronic device 40 to communicate with one or more other computing devices. Such communication may be through input/output (I/O) interface 411. Also, the electronic device 40 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet) via the network adapter 412. As shown, the network adapter 412 communicates with the other modules of the electronic device 40 over the bus 403. It should be appreciated that although not shown in FIG. 4, other hardware and/or software modules may be used in conjunction with electronic device 40, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, among others.

The processing unit 401 executes various functional applications and data processing, for example, implementing an image processing method provided by an embodiment of the present invention, by running a program stored in the system memory 402.

EXAMPLE six

An embodiment of the present invention also provides a storage medium containing computer-executable instructions for performing an image processing method when executed by a computer processor. The method comprises the following steps:

Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations for embodiments of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. An image processing method, comprising:

2. The method according to claim 1, wherein the step of inputting the image to be classified into each image classification model to obtain a first attribute value corresponding to each actual classification category comprises:

inputting the image to be classified into a current image classification model aiming at each image classification model to obtain an attribute value to be used of at least one actual classification category corresponding to the current image classification model;

and determining at least one attribute value to be used corresponding to the current actual classification category aiming at each actual classification category, and carrying out mean processing on the at least one attribute value to be used to obtain a first attribute value corresponding to the current actual classification category.

3. The method of claim 1, further comprising:

aiming at each reference category identification, acquiring at least one first original image corresponding to the current reference category identification, and respectively determining a feature vector corresponding to the first original image;

and determining a reference category vector corresponding to each reference category identifier by performing mean processing on at least one feature vector corresponding to the same reference category identifier.

4. The method according to claim 1, wherein the determining the target class corresponding to the image to be classified based on each first attribute value and each second attribute value comprises:

determining a set comprising actual classification categories and reference category identifications, wherein the set comprises at least one category element;

and aiming at each class element, determining an element type corresponding to the current class element, and processing a first attribute value and a second attribute value of the current class element according to the element type to determine a target class.

5. The method of claim 4, further comprising:

if the attribute value of the current category element comprises a first attribute value or a second attribute value, determining that the element type corresponding to the current category element is a first type; or the like, or, alternatively,

6. The method of claim 5, wherein the processing the first attribute value and the second attribute value of the current category element according to the element type to determine the target category comprises:

if all the element types are the first type, normalizing the first attribute values and the second attribute values to obtain third attribute values corresponding to all the class elements, and determining a target class corresponding to the image to be classified based on the third attribute values; or the like, or, alternatively,

7. The method of claim 1, further comprising:

training to obtain the image classification models;

the training to obtain the image classification models comprises:

acquiring a training sample set corresponding to each classification model to be trained; the training sample set comprises at least one training sample, and the training sample comprises second original images corresponding to each actual classification category;

aiming at each classification model to be trained, taking a second original image in a current training sample corresponding to the current classification model to be trained as the input of the current classification model to be trained, taking an actual classification category as the output of the current classification model to be trained, and training the current classification model to be trained to obtain a trained current classification model to be trained;

and taking the trained current classification model to be trained as the current classification model to be trained again and training until the loss function of the current classification model to be trained is converged to obtain a corresponding image classification model.

8. The method according to claim 7, wherein the step of using the trained current classification model to be trained as the current classification model to be trained again and training until a loss function of the current classification model to be trained converges to obtain a corresponding image classification model comprises:

acquiring at least one image to be marked;

inputting each image to be labeled to the trained current classification model to be trained, outputting the actual classification category corresponding to each image to be labeled and the corresponding attribute value to be processed, and taking the trained current classification model to be trained as the current classification model to be trained again;

determining a target annotation image from each image to be annotated based on a preset attribute value threshold, each attribute value to be processed and a corresponding actual classification category;

and updating the training sample set based on each target labeling image, and re-executing the step of training the current classification model to be trained based on the updated training sample set until the loss function of the current classification model to be trained is converged to obtain the corresponding image classification model.

9. An image processing apparatus characterized by comprising:

10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the image processing method according to any one of claims 1 to 8.