CN113902944A - Model training and scene recognition method, device, equipment and medium - Google Patents

Abstract

The application discloses a model training and scene recognition method, device, equipment and medium. The original scene recognition model can be trained based on the scene probability vector, the scene label, the sample feature, the class center feature corresponding to the first scene category, the sample feature and the class center feature corresponding to the second scene category, so that the trained scene recognition model can be drawn close to the class center feature of the scene category according to the image features of the images in the same scene category, and is far away from the features of the class center features of other scene categories, and the feature level of the images is further combined, thereby realizing accurate processing of the scene category images which are not contained in the closed image set, and improving the precision, performance and naturalness of the scene recognition model.

Description

Model training and scene recognition method, device, equipment and medium

technical field

The present application relates to the technical field of image processing, and in particular, to a method, apparatus, device and medium for model training and scene recognition.

Background technique

With the development of multimedia technology, people watch more and more kinds of video images every day, and the products involved in the video content are also more and more abundant. Automatically identifying and classifying image scene information helps machines better understand images and helps downstream algorithms develop functions for different scenes.

With the development of neural networks in the field of vision, their performance in image classification tasks has also surpassed most traditional algorithms. However, most neural network-based scene recognition systems are trained and tested in a closed image set, that is, the scene recognition system can only identify the scene categories contained in the closed image set. However, in practical applications, since the scene categories to which all images may belong are inexhaustible, the scene category to which the images currently required for scene recognition actually belong may not be the scene categories contained in the closed image set, but if the scene category is If the scene recognition system tries to identify the scene category to which the image belongs, an erroneous result will be obtained, thereby affecting the processing of the downstream algorithm.

Therefore, there is an urgent need for a scene recognition system that can not only accurately identify images belonging to scene categories contained in a closed image set, but also process images that do not belong to scene categories contained in a closed image set.

SUMMARY OF THE INVENTION

The present application provides a model training and scene recognition method, device, device and medium to solve the problem that the existing scene recognition system cannot accurately process images of scene categories that are not included in a closed image set.

The application provides a scene recognition model training method, the method includes:

Obtain any sample image in the sample set; wherein, the sample image corresponds to a scene label, and the scene label is used to identify the first scene category to which the sample image belongs;

Determine the scene probability vector corresponding to the sample image and the sample feature of the sample image through the original scene recognition model; wherein, the scene probability vector includes the probability value of the sample image belonging to each scene category respectively;

Identify the original scene based on the scene probability vector and the scene label, the sample feature and the class center feature corresponding to the first scene category, the sample feature and the class center feature corresponding to the second scene category The model is trained to obtain a trained scene recognition model; wherein the second scene category is a scene category other than the first scene category in each of the scene categories.

The application provides a scene recognition method, the method includes:

Determine the image features of the image to be recognized through the pre-trained scene recognition model;

Determine the similarity of the image feature with the target class center feature of each scene category;

According to each of the similarity and the similarity threshold, determine whether each scene category includes the scene category to which the to-be-recognized image belongs;

If it is determined that each scene category includes the scene category to which the to-be-recognized image belongs, determining the scene category to which the to-be-identified image belongs by using the scene recognition model;

If it is determined that each scene category does not include the scene category to which the to-be-identified image belongs, the scene category to which the to-be-identified image belongs is not continued to be identified.

The application provides a scene recognition model training device, the device includes:

an obtaining unit, configured to obtain any sample image in the sample set; wherein, the sample image corresponds to a scene label, and the scene label is used to identify the first scene category to which the sample image belongs;

The processing unit is used to determine the scene probability vector corresponding to the sample image and the sample feature of the sample image through the original scene recognition model; wherein, the scene probability vector includes the sample images belonging to each scene category respectively. probability value;

The training unit is configured to, based on the scene probability vector and the scene label, the sample feature and the class center feature corresponding to the first scene category, the sample feature and the class center feature corresponding to the second scene category, to The original scene recognition model is trained to obtain a trained scene recognition model; wherein the second scene category is a scene category other than the first scene category in each scene category.

The application provides a scene recognition device, the device includes:

The first processing module is used to determine the image feature of the image to be recognized by using a pre-trained scene recognition model;

The second processing module is used to determine the similarity between the image features and the target class center features of each scene category;

a third processing module, configured to determine whether each scene category includes the scene category to which the to-be-recognized image belongs according to each of the similarity and the similarity threshold; if it is determined that each scene category includes the The scene category to which the image to be recognized belongs, the scene recognition model is used to determine the scene category to which the image to be recognized belongs; if it is determined that each scene category does not include the scene category to which the image to be recognized belongs, Then, the scene category to which the to-be-identified image belongs is not continued to be identified.

The application provides an electronic device, the electronic device includes a processor, and the processor is configured to implement the steps of the above-mentioned scene recognition model training method when executing the computer program stored in the memory, or, to implement the above-mentioned method The steps of the scene recognition method.

The present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, realizes the steps of the above-mentioned scene recognition model training method, or, realizes the above-mentioned scene recognition method A step of.

Because in the process of training the original scene recognition model based on the sample images in the sample set, through the original scene recognition model, the scene probability vector corresponding to the input sample image and the sample features of the sample image can be obtained, so that the follow-up can be based on the The scene probability vector and the scene label, the sample feature and the class center feature corresponding to the first scene category, the sample feature and the class center feature corresponding to the second scene category, to train the original scene recognition model , to obtain the trained scene recognition model, so that the trained scene recognition model can move closer to the class center feature of the scene category according to the image features of the images in the same scene category, while staying away from the characteristics of the class center features of other scene categories , further combined with the feature level of the image, to determine whether the scene category of the image can be identified and if the scene category of the image can be identified, the scene category to which the image belongs, not only realizes accurate identification belonging to a closed image set The included scene category images can also process scene category images that are not included in the closed image set, which improves the accuracy, performance and naturalness of the scene recognition model.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without any creative effort.

1 is a schematic diagram of a scene recognition model training process provided by some embodiments of the present application;

2 is a schematic diagram of a specific scene recognition model training process provided by some embodiments of the present application;

3 is a schematic structural diagram of an original scene recognition model provided by some embodiments of the present application;

FIG. 4 is a schematic diagram of a scene recognition process provided by some embodiments of the present application;

5 is a schematic diagram of a specific scene recognition process provided by some embodiments of the present application;

6 is a schematic structural diagram of a scene recognition model training apparatus provided by some embodiments of the present application;

FIG. 7 is a schematic structural diagram of a scene recognition apparatus according to some embodiments of the present application;

FIG. 8 is a schematic structural diagram of an electronic device provided by some embodiments of the present application;

FIG. 9 is a schematic structural diagram of an electronic device according to some embodiments of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

How to enable the scene recognition system to accurately process images that do not belong to the scene categories contained in the closed image set is essentially an open set recognition problem. The scene recognition system needs to be able to discover and learn the scene categories to which images of unknown scene categories belong. In conclusion, the open set recognition problem is an important and challenging problem in the pattern recognition and multimedia community.

Therefore, in order to realize that the scene recognition system can accurately process images of scene categories that are not included in a closed image set, the present application provides a model training and scene recognition method, device, device and medium.

Example 1:

1 is a schematic diagram of a scene recognition model training process provided by some embodiments of the present application, and the process includes:

S101: Acquire any sample image in the sample set; wherein, the sample image corresponds to a scene label, and the scene label is used to identify the first scene category to which the sample image belongs.

The scene recognition model training method provided in the present application is applied to an electronic device, and the electronic device may be a smart device such as a mobile terminal, or a server such as a home brain. Of course, the electronic device may also be a display device such as a television.

In order to obtain an accurate scene recognition model, the original scene recognition model needs to be trained according to each sample image in the pre-acquired sample set. Wherein, any sample image in the sample set is obtained by: determining the collected original image as the sample image; and/or, after adjusting the pixel values of the pixels in the collected original image, the adjusted The image is determined to be a sample image.

It should be noted that, in order to facilitate the training of the scene recognition model, any sample image in the sample set corresponds to a scene label, and any scene label is used to identify the scene category to which the sample image belongs (for convenience of description, it is recorded as the first scene category. ). For example, the scene categories are live broadcast scenes, game scenes, eating and broadcasting scenes, and other categories.

As a possible implementation, if the sample set contains a sufficient number of sample images, that is, contains a large number of original images collected in different environments, the original scene recognition model can be trained according to the sample images in the sample set.

As another possible implementation, in order to ensure the diversity of sample images and improve the accuracy of the scene recognition model, the pixel values of the pixels in the original image can be adjusted, for example, the original image Perform blur processing, sharpening processing, contrast processing, etc. to obtain a large number of adjusted images, and determine the adjusted images as sample images to train the original scene recognition model.

According to statistics, taking an electronic device as a display device, such as a TV, in the working scene of the display device, the more common image quality problems in the acquired images include: blur, exposure, too dark, too low contrast, There is noise in the picture, for example, in the live broadcast scene, there may be exposure problems in the acquired image. In order to ensure the diversity of sample images and improve the accuracy of the scene recognition model, it is possible to adjust the image quality of the collected original images in advance for possible image quality problems in the acquired images in the working scene of the display device. . The pixel value of the pixel point in the collected original image can be adjusted in at least one of the following ways, including:

Method 1: Adjust the pixel value of the pixel in the original image through a preset convolution kernel;

Mode 2: Contrast adjustment is performed on the pixel values of the pixel points in the original image;

Mode 3: Adjust the brightness of the pixel values of the pixel points in the original image;

Manner 4: Perform noise processing on the pixel values of the pixel points in the original image.

For example, if you want to perform noise processing on the original image to obtain an adjusted image with different noises, you can perform noise processing on the pixel values of the pixel points in the original image, that is, randomly add noise to the original image. Among them, in the process of adding noise to the original image, the types of noise used should be as many as possible, such as white noise, salt and pepper noise, Gaussian noise, etc., so as to make the sample images in the sample set more diverse. Thereby improving the accuracy and robustness of the scene recognition model.

It should be noted that the process of processing the pixel values of the pixel points in the original image belongs to the prior art, and details are not described here.

Obtaining sample images in the above manner can double the number of sample images in the sample set, so that a large number of sample images can be quickly obtained, and the difficulty, cost and resource consumption of obtaining sample images can be reduced. Subsequently, the original scene recognition model can be trained according to more sample images, which improves the accuracy and robustness of the scene recognition model.

As another possible implementation manner, the collected original image and the adjusted image obtained by adjusting the pixel values of the pixel points in the collected original image may also be determined as sample images. Based on the original images in the sample set and the adjusted images, the original scene recognition model is trained together.

S102: Determine the scene probability vector corresponding to the sample image and the sample feature of the sample image by using the original scene recognition model; wherein, the scene probability vector includes the probability value that the sample image belongs to each scene category respectively.

After the sample set for training the original scene recognition model is obtained based on the above-mentioned embodiment, the original scene recognition image can be trained based on each sample image in the sample set.

In the specific implementation process, any sample image is input into the original scene recognition model. Through the original scene recognition model, the scene probability vector corresponding to the sample image and the image feature of the sample image (for convenience of description, denoted as sample feature) can be obtained. Wherein, the scene probability vector includes a probability value that the sample image belongs to each scene category, and each scene category is determined by the scene category to which each sample image in the sample set belongs. Any sample feature represents a higher-dimensional, more abstract image feature extracted from the sample image.

The original scene recognition model may be a decision tree, logistic regression (LR), naive bayes (NB) classification algorithm, random forest (RF) algorithm, support vector machine (support vector machine) machines, SVM) classification algorithms, histogram of oriented gradients (HOG), deep learning algorithms, etc. The deep learning algorithm may include a neural network, a deep neural network, a convolutional neural network (Convolutional Neuron Network, CNN), and the like.

In a possible implementation manner, in order to perform scene recognition through the scene recognition model, the original scene recognition model includes a feature extraction layer, a feature output layer and a classification output layer. The feature extraction layer outputs the feature output layer, and the feature output layer is connected with the classification output layer. When the sample image is input into the original scene model, the sample features of the input sample image can be obtained through the feature extraction layer in the original scene recognition model. . Then, the sample features can be output through the feature output layer in the original scene recognition model. Through the classification output layer in the original scene recognition model, based on the sample feature, the scene probability vector corresponding to the sample image can be obtained and output.

S103: Based on the scene probability vector and the scene label, the sample feature and the class center feature corresponding to the first scene category, the sample feature and the class center feature corresponding to the second scene category, perform a The scene recognition model is trained to obtain the trained scene recognition model; wherein, the second scene category is a scene category other than the first scene category in each of the scene categories.

Since any sample image in the sample set corresponds to a scene label, that is, it identifies the scene category to which the sample image actually belongs. Therefore, in this application, the scene probability vector corresponding to the sample image and the sample of the sample image can be determined. After the feature, the original scene recognition model can be trained by using the scene recognition model training method provided by the present application based on the scene probability vector, the corresponding scene label and the sample feature.

If the scene classification to which an image belongs is a scene classification that can be recognized by a pre-trained scene recognition model, the image features of the image generally have a greater metric distance from the image features of the sample images belonging to the scene type in the sample set. The smaller the metric distance from the image features of the sample images in the sample set that do not belong to the scene type. Based on this, when identifying the scene classification of an image, the metric distance between the image feature of the image and the image feature of each sample image in the sample set can be determined, and the scene to which the image belongs is determined according to the obtained metric distance. Classification is the classification of scenes that a pre-trained scene recognition model can recognize.

The metric distance can be obtained by means of Euclidean distance, cosine similarity, KL divergence function, or the like.

Further, since the sample set will contain a large number of sample images, if the metric distance between the image features of an image and the image features of each sample image in the sample set is determined, it will consume a lot of computing resources and reduce scene recognition. The system determines the efficiency of the scene category to which the image belongs. Based on this, the class center feature of the scene category to which each sample image in the sample set belongs can be obtained, so that the features generally possessed by the images of the scene category can be represented by the class center feature. Subsequently, when the scene classification of the image is identified, the metric distance between the image feature of the image and the class center feature of the scene category to which each sample image in the sample set belongs can be determined. According to the obtained metric distance, it is determined whether the scene classification to which the image belongs is a scene classification that can be recognized by the pre-trained scene recognition model.

It should be noted that the dimension of the sample feature is the same as the dimension of the class center feature.

In a possible implementation, in order to accurately acquire the class-centric feature of each scene category, the class-centric feature of each scene category can be acquired in the following manner:

Method 1. In order to combine the process of obtaining the class center features of each scene category into the model training process, each iteration of the original scene recognition model can be trained through the current iteration of the scene recognition model to obtain each item in the sample set. Sample features of one or more sample images of the scene category. Then, according to each sample feature, the candidate class center feature corresponding to each scene category is determined. Based on each candidate class center feature, the class center feature of each scene category in the next iterative training is determined.

In a possible implementation, for each scene category, among the sample images of the scene category, the sample images correctly recognized by the currently iterative scene recognition model (for convenience of description, denoted as target sample images) can be determined. The correct recognition by the currently iterative scene recognition model can be understood as determining that the scene category of the sample image is the same as the first scene category of the sample image determined by the currently iterative scene recognition model. Then, according to the sample feature of the target sample image and the weight value of the target sample image, a weighted average vector is determined, and based on the weighted average vector, the candidate class center feature corresponding to the scene category is determined.

Among them, the weight value of the target sample image can be pre-configured by manual configuration, for example, the weight value of each target sample image is set to be 1, or the current iterative scene recognition model can be used to obtain the target sample image belonging to The probability value of the scene category, and the probability value is determined as the weight value of the target sample image.

For example, if the weight value of the target sample image is pre-configured by manual configuration, then according to the sample characteristics of the target sample image and the weight value of the target sample image, the weighted average vector can be determined by the following formula:

为被正确识别为场景分类i的第j个目标样本图像的样本特征，

为被正确识别为场景分类i的目标样本图像的数量，目标样本图像的权重值为1。Among them, C _i is the class center feature of scene category i,

is the sample feature of the j-th target sample image correctly identified as scene classification i,

To be the number of target sample images correctly identified as scene classification i, the weight of target sample images is 1.

For another example, if the probability value of the target sample image belonging to the scene category is obtained through the current iterative scene recognition model, and the probability value is determined as the weight value of the target sample image, then according to the sample characteristics of the target sample image and the target sample image The weight value of the image to determine the weighted average vector can be expressed by the following formula:

为被正确识别为场景分类i的第j个目标样本图像的样本特征，

为被正确识别为场景分类i的目标样本图像的数量，

为通过当前迭代的场景识别模型，获取第j个目标样本图像归属于该场景类别i的概率值，该权重值度越高说明，当前迭代的场景识别模型的识别结果越准确，被越准确的识别的样本图像的样本特征对类中心的贡献更大。Among them, C _i is the class center feature of scene category i,

is the sample feature of the j-th target sample image correctly identified as scene classification i,

is the number of target sample images correctly identified as scene class i,

In order to obtain the probability value of the j-th target sample image belonging to the scene category i through the scene recognition model of the current iteration, the higher the weight value is, the more accurate the recognition result of the scene recognition model of the current iteration is, and the more accurate it is. The sample features of the identified sample images contribute more to the class centers.

In another possible implementation, it is also possible to determine, for each scene category, the target sample image correctly identified by the currently iterative scene recognition model in each sample image of the scene category; based on the preset target algorithm, obtain The target feature in the sample features of the target sample image, and based on the target feature, the candidate class center features corresponding to the scene categories respectively are determined. Wherein, the target feature is a principal component feature, or a normalized feature.

The process of obtaining the principal component feature in the pattern feature of the image, or the process of normalizing the feature, belongs to the prior art, and will not be repeated here.

After the candidate class center feature of each scene category is obtained based on the above-mentioned embodiment, the class center feature of each scene category may be determined according to the candidate class center feature of each scene category. Specifically, according to the candidate class center feature of each scene category, the process of determining the class center feature of each scene category mainly includes the following two cases:

Case 1. Before training the original scene recognition model, the class center features of each scene category are randomly initialized and generated, resulting in the first iteration training of the original scene model, each scene category of the current iteration The class center feature of is inaccurate. Therefore, in the present application, if it is determined that the current iteration is the first iteration, each candidate class center feature can be directly determined as the class center feature of each scene category in the next iterative training, that is, according to each candidate class Center feature, update the class center feature of each scene category in the current iteration, so as to improve the accuracy of the obtained class center feature of each scene category.

The dimension of the class center feature generated by the random initialization is the same as the dimension of the sample feature.

Case 2. In order to make the class center feature of each scene category determined in each iteration more accurate and the change more stable, in this application, a weight vector is pre-configured, and the weight vector is used to adjust the class center feature for each update. Amplitude. After the candidate class center feature of each scene category is determined based on the above-mentioned embodiment, if it is determined that the current iteration is not the first iteration, then for each scene category, the candidate class center feature corresponding to the scene category is determined, which is consistent with the current iteration. The iteratively determined difference vector of the class center feature corresponding to the scene category. Then, the difference vector is adjusted according to the difference vector and the preconfigured weight vector. According to the adjusted difference vector and the class center feature corresponding to the scene category determined in the current iteration, the class center feature corresponding to the scene category in the next iterative training is determined.

In a possible implementation manner, a product vector of the difference vector and a pre-configured weight vector may be obtained, and the product vector may be determined as the adjusted difference vector.

In a possible implementation manner, a sum vector may be determined according to the adjusted difference vector and the currently determined class center feature corresponding to the scene category, and the sum vector may be determined as the class corresponding to the scene category in the next iterative training central feature.

For example, based on the above cases 1 and 2, the process of determining the class center feature of each scene category according to the candidate class center feature of each scene category can be expressed by the following formula:

为下一迭代训练中该场景类别i对应的类中心特征，

为场景类别i对应的候选类中心向量，

为当前迭代确定的该场景类别i对应的类中心特征，W为预先配置的权重向量。in,

is the class center feature corresponding to the scene category i in the next iterative training,

is the candidate class center vector corresponding to scene category i,

is the class center feature corresponding to the scene category i determined for the current iteration, and W is a pre-configured weight vector.

Manner 2: The sample features of each sample image in the sample set can also be obtained through a pre-trained feature extraction model. It can be understood that the feature extraction model is also a feature extraction algorithm. Then, clustering algorithms, such as fuzzy clustering algorithm, K-means clustering, maximum and minimum distance clustering algorithm, etc., are used to cluster each sample feature, so as to obtain the cluster corresponding to each scene category. The cluster corresponding to any scene category includes the sample features of the scene category. Then, according to the sample features contained in the cluster corresponding to each scene category, the class center feature in the cluster is determined, that is, the class center feature of each scene category is determined.

Wherein, any sample feature included in the cluster may be determined as the class center feature, or the average vector may be determined as the class center feature according to the average vector of each sample feature included in the cluster. In the specific implementation process, it can be used flexibly according to actual needs, which will not be described in detail here.

It should be noted that the process of training the feature extraction model and the process of clustering the sample features according to the clustering algorithm belong to the prior art, and will not be described in detail here.

In order to facilitate the training of the obtained scene recognition model, the scene to which the image belongs can be determined from the feature level of the image. In this application, in the process of training the original scene recognition model, the sample characteristics of the sample image and each The metric distance between the class center features of the scene class to which the sample image belongs. Then based on the metric distance, the scene probability vector and the scene label, the original scene recognition model is performed. It can be understood that the original scene recognition model is trained based on the scene probability vector, the scene label, the sample feature, the class center feature corresponding to the first scene category, the sample feature and the class center feature corresponding to the second scene category. The second scene category is a scene category other than the first scene category among scene categories to which each sample image in the sample set belongs.

In a possible implementation manner, when determining the metric distance between the sample feature and the class center feature of the scene category to which each sample image belongs, it can be determined by the following Euclidean distance formula:

Among them, d(x, y _i ) represents the metric distance between the sample feature x and the class center feature of the ith scene category.

In another possible implementation, since the Euclidean distance represents the closeness of the two vectors in absolute distance, the cosine similarity represents the closeness of the two vectors in the direction. Therefore, when determining the metric distance between the sample feature and the class center feature of the scene category to which each sample image belongs, it can be determined by the following formula:

Among them, d(x, y _i ) represents the metric distance between the sample feature x and the class center feature of the ith scene category, and cos_sim(x, y _i ) represents the sample feature x and the class center feature of the ith scene category The cosine similarity between the two, α ₁ represents the weight value corresponding to the Euclidean distance, and α ₂ represents the weight value corresponding to the cosine similarity.

In a possible implementation, a loss value (denoted as the first loss value for convenience of description) may be determined based on the scene probability vector and the scene label; the loss may be determined based on the sample feature and the class center feature corresponding to the first scene category value (for convenience of description, denoted as the second loss value); based on the sample feature and the class center feature corresponding to the second scene category, determine the loss value (for convenience of description, denoted as the third loss value). Then, the comprehensive loss value is determined according to the first loss value and its corresponding first weight value, the second loss value and its corresponding second weight value, and the third loss value and its corresponding third weight value. Based on the comprehensive loss value, the original scene recognition model is trained to update the parameter values of the parameters in the original scene recognition model, so as to obtain the trained scene recognition model.

In specific implementation, when training the original scene recognition model according to the comprehensive loss value, a gradient descent algorithm can be used to back-propagate the gradients of the parameters in the original scene recognition model, so as to realize the training of the original scene recognition model. .

It can be understood that the second loss value can be determined by the metric distance between the sample feature and the class center feature corresponding to the first scene category, and the third loss value can also be determined by the difference between the sample feature and the class center feature corresponding to the second scene category. The metric distance between them is determined.

For example, according to the first loss value and its corresponding first weight value, the second loss value and its corresponding second weight value, and the third loss value and its corresponding third weight value, the comprehensive loss value can be determined by Determined by the following formula:

为根据场景概率向量y以及场景标签

所确定的第一损失值，d(x_i，C_i)为样本特征x_i与第一场景类别对应的类中心特征C_i之间的度量距离，d(x_i，C_cls！＝i)为样本特征x_i与第二场景类别对应的类中心特征C_cls！＝i之间的度量距离，ω₁为第一权重值，ω₂为第二权重值，ω₃为第三权重值。in,

is based on the scene probability vector y and the scene label

The determined first loss value, d( _xi , C _i ) is the metric distance between the sample feature _{xi and the class center feature C i corresponding} to the first scene category, d( _xi , C _{cls !=i} ) is the class center feature C _cls! corresponding to the sample feature _xi and the second scene category! _{= metric distance between i} , ω ₁ is the first weight value, ω ₂ is the second weight value, and ω ₃ is the third weight value.

In practical application, the smaller the metric distance between image features of images of the same scene category, the greater the metric distance between image features of images of different scene categories. Therefore, when setting the first weight value, the second weight value and the third weight value, the second loss value may be a negative number, and the first loss value and the third loss value may be positive numbers, so that when optimizing the comprehensive loss value , the optimization direction of the scene recognition model is to minimize the first loss value, minimize the second loss value, and maximize the third loss value, so as to increase the metric distance between different scene categories and reduce the same scene category. The metric distance between sample features. From the perspective of feature space, the distribution of sample features of different scene categories is relatively scattered, but the distribution of sample features of the same scene category is relatively concentrated.

By training the scene recognition model through the above comprehensive loss value, the similarity of the sample features of different scene categories in the feature space can be reduced, and the similarity of the sample features of the same scene category can be increased in the feature space. It is beneficial to improve the accuracy of determining whether the scene category of a certain image can be recognized by the scene recognition model.

Since the sample set contains a large number of sample images, the above operations are performed on each sample image. When the preset convergence conditions are met, the scene recognition model training is completed.

Wherein, satisfying the preset convergence condition may be that the sum of each comprehensive loss value obtained according to the current iterative training is less than the set loss value threshold, the number of iterations for training the model reaches the set maximum number of iterations, and the like. In the specific implementation, it can be set flexibly, which is not specifically limited here.

In a possible implementation, in order to determine the accuracy of the trained scene recognition model, before releasing the scene recognition model online, the scene recognition model may be tested to determine whether the scene recognition model can accurately The images belonging to the scene categories contained in the sample set, and the recognition accuracy of the scene recognition model for the images that can be recognized.

In the specific implementation process, a test set for testing the trained scene recognition model is obtained, and the test set includes test sample images, so as to verify the reliability of the above-mentioned trained scene recognition model based on the test sample images. When acquiring the test sample images contained in the test set, the test sample images contained in the test set can be obtained by re-collecting the test sample images, and/or, the sample images in the sample set can also be divided into training sample images and test sample images. Obtain. It should be noted that the specific process of collecting the test sample images included in the test set is similar to the above-mentioned process of collecting the sample images included in the sample set, and the repeated parts will not be repeated.

In order to test whether the scene recognition model can accurately process images of scene categories not included in the sample set, in each acquired test sample image, there must be at least one scene category to which the test sample image belongs, and The scene categories included in the sample set are all different.

Among them, each test sample image corresponds to a scene label and a processing label. The scene label is used to identify the scene category to which the test sample image belongs (for the convenience of description, it is recorded as the third scene category), and the processing label is used to identify the sample set. Whether the included scene category contains this third scene category.

For each test sample image in the test set, the test sample image is input into the scene recognition model. Through the scene recognition model, the image features of the test sample image (for convenience of description, denoted as test sample features) are obtained. Then determine the test sample features and the similarity with the target class center features of each scene category respectively. Wherein, the target class center feature may be the class center feature of each scene category included in the sample set during the last iterative training of the original scene recognition model. Then, according to the similarity threshold and the obtained similarity, it is determined whether each scene category includes the scene category to which the test sample image belongs.

In a possible implementation, the similarity threshold may be configured manually, or the reference similarity between the target class center feature and other target class center features may be determined for each target class center feature. . Then, the similarity threshold is determined according to the reference similarity corresponding to each target class center feature respectively.

In a possible implementation manner, if the reference similarity is determined according to metric distances such as Euclidean distance, the similarity threshold may be determined according to the minimum value of each reference similarity.

In a possible implementation manner, if the reference similarity is determined according to metric distances such as cosine similarity, the similarity threshold may be determined according to the maximum value of each reference similarity.

In a possible implementation, if the similarity is determined according to the Euclidean distance and other metric distances, the smaller the similarity between the two image features, the higher the similarity between the two image features, the higher the similarity between the two image features. It is possible to belong to the same scene category; the greater the similarity of the two image features, the lower the similarity between the two image features, and the more likely that the two image features do not belong to the same scene category. Therefore, according to each of the similarity and the similarity threshold, it is determined whether each scene category includes the scene category to which the test sample image belongs, and if any similarity is smaller than the similarity threshold, it means that the test sample image has The image feature and the target class center feature corresponding to the similarity are very likely to belong to the same scene category, then it is determined that each scene category included in the sample set includes the scene category to which the test sample image belongs; if each similarity is not less than If the similarity threshold is set, it means that the image features of the test sample image and the central feature of each target class are from different scene categories, then it is determined that each scene category included in the sample set does not include the scene category to which the test sample image belongs.

In a possible implementation, if the similarity is determined according to a metric distance such as cosine similarity, the smaller the similarity between the two image features is, the lower the similarity between the two image features is, and the lower the similarity between the two image features is. The less likely they belong to the same scene category; the smaller the similarity between the two image features, the higher the similarity between the two image features, the more likely the two image features belong to the same scene category. Therefore, according to each of the similarity and the similarity threshold, it is determined whether each scene category includes the scene category to which the test sample image belongs. If there is any similarity greater than the similarity threshold, it means that the test sample image has The image feature and the target class center feature corresponding to the similarity are very likely to belong to the same scene category, then it is determined that each scene category included in the sample set includes the scene category to which the test sample image belongs; if each similarity is not greater than The similarity threshold indicates that the image features of the test sample images and the central feature of each target class are from different scene categories, and it is determined that each scene category included in the sample set does not include the scene category to which the test sample images belong.

Specifically, if it is determined that each scene category includes the scene category to which the test sample image belongs, it means that the scene category to which the test sample image belongs can be accurately determined by the scene recognition model, that is, the scene category to which the test sample image belongs. is known, the scene recognition model is used to determine the scene category to which the test sample image belongs; if it is determined that each scene category does not contain the scene category to which the test sample image belongs, the scene category to which the test sample image belongs is indicated. If the scene recognition model can not be accurately determined, that is, the scene category to which the test sample image belongs is unknown, then the scene category to which the image belongs will not be recognized.

Since the test set contains a large number of test sample images, the above operations are performed for each test sample image. Based on the processing result of each test sample image (including whether the scene recognition model recognizes the result of the scene category to which the test sample image belongs, and when the scene recognition model recognizes the scene category to which the test sample image belongs, the obtained test sample The scene probability vector of the image), the processing label of each test sample image, and the scene label of each test sample image, perform corresponding calculations to determine various evaluation indicators of the scene recognition model, such as accuracy rate, error rate, precision rate Wait. If it is determined that each evaluation index of the scene recognition model meets the preset release requirements, the scene recognition model can be released online. If it is determined that the evaluation indicators of the scene recognition model do not meet the preset release requirements, the scene recognition model can be further trained based on the sample images in the sample set.

Since the present application learns the sample features of the sample images of each scene category in the sample set at the same time when training the scene recognition model, so as to obtain the class center feature of each scene category, so that the trained scene recognition model can be used or tested in the future. In the model, the image feature of the input image can be obtained through the scene recognition model, and then the metric distance between the image feature and the class center feature of each scene category can be determined. The metric distances are not close, indicating that each known scene category does not contain the scene category to which the image belongs, that is, the scene category to which the image belongs is also unknown, then the subsequent scene type identification for the image is not performed, so that It realizes that the image features extracted by the scene recognition model are discriminative to help the scene recognition model determine whether the scene category to which the image belongs is recognizable by the scene recognition image, and avoids the misidentification of the scene category to which the image belongs. Below is the precision of the algorithm.

Because in the process of training the original scene recognition model based on the sample images in the sample set, through the original scene recognition model, the scene probability vector corresponding to the input sample image and the sample features of the sample image can be obtained, so that the follow-up can be based on the The scene probability vector and the scene label, the sample feature and the class center feature corresponding to the first scene category, the sample feature and the class center feature corresponding to the second scene category, to train the original scene recognition model , to obtain the trained scene recognition model, so that the trained scene recognition model can move closer to the class center feature of the scene category according to the image features of the images in the same scene category, while staying away from the characteristics of the class center features of other scene categories , further combined with the feature level of the image, to determine whether the scene category of the image can be identified and if the scene category of the image can be identified, the scene category to which the image belongs, not only realizes accurate identification belonging to a closed image set The included scene category images can also process scene category images that are not included in the closed image set, which improves the accuracy, performance and naturalness of the scene recognition model.

Example 2:

Taking the execution subject as the display device as an example, the following describes the scene recognition model training method provided by the present application in detail through specific embodiments. FIG. 2 is a schematic diagram of a specific scene recognition model training process provided by some embodiments of the present application. The process includes: :

S201: Build an original scene recognition model.

S202: Randomly construct the class center feature of each scene category.

S203: Obtain any sample image in the sample set.

The sample image corresponds to a scene label, and the scene label is used to identify the first scene category to which the sample image belongs.

S204: Determine the scene probability vector corresponding to the sample image and the sample feature of the sample image through the original scene recognition model.

Wherein, the scene probability vector includes the probability values that the sample images belong to each scene category respectively.

The process of determining the scene probability vector corresponding to the sample image and the sample feature of the sample image by using the original scene recognition model will be described in detail below with reference to FIG. 3 . FIG. 3 is a structure of an original scene recognition model provided by some embodiments of the application. Schematic. After inputting any sample image into the original scene recognition model, the sample features of the input sample image can be obtained through the feature extraction layer in the original scene recognition model. Then, the sample features can be output through the feature output layer in the original scene recognition model. Through the classification output layer in the original scene recognition model, based on the sample features, the scene probability vector corresponding to the sample image can be obtained and output.

Since the sample set includes a large number of sample images, the above steps of operations S203 to S204 are performed for each sample image.

S205: Update the class center feature of each scene category of the current iteration.

Among them, if the current iteration is the first iteration, the candidate class center feature corresponding to each scene category is determined according to each sample feature obtained in the current iteration; each candidate class center feature is determined as the next iteration of training. Class-centric features for each scene category.

If the current iteration is the first iteration, according to each sample feature obtained in the current iteration, determine the candidate class center feature corresponding to each scene category; for each scene category, determine the candidate class center feature corresponding to the scene category, The difference vector of the class center feature corresponding to the scene category determined in the current iteration; according to the difference vector, the preconfigured weight vector, and the class center feature corresponding to the scene category determined in the current iteration, determine the scene category corresponding to the next iteration training. class-centric features.

S206: For each sample image, according to the scene probability vector and the scene label of the sample image, the sample feature of the sample image and the class center feature corresponding to the first scene category to which the sample image belongs, the sample feature of the sample image and The class center feature corresponding to the second scene category of the sample image determines the comprehensive loss value.

S207: Determine whether the sum of each comprehensive loss value is less than the preset loss value threshold, if it is less than, execute S208, otherwise, execute S209.

S208: Acquire and save the scene recognition model that has been trained.

S209: Adjust the parameter values of the parameters of the original scene recognition model, and execute S203.

Example 3:

The present application also provides a scene recognition method. FIG. 4 is a schematic diagram of a scene recognition process provided by some embodiments of the present application, and the process includes:

S401: Determine image features of the image to be recognized by using a pre-trained scene recognition model.

S402: Determine the similarity between the image features and the target class center feature of each scene category respectively.

S403: Determine whether each scene category includes the scene category to which the to-be-identified image belongs, according to each of the similarity and the similarity threshold.

S404: If it is determined that each scene category includes the scene category to which the to-be-identified image belongs, determine the scene category to which the to-be-identified image belongs by using the scene recognition model.

S405: If it is determined that each scene category does not include the scene category to which the to-be-identified image belongs, do not continue to identify the scene category to which the to-be-identified image belongs.

The scene recognition method provided in this application is applied to an electronic device, and the electronic device may be a smart device such as a mobile terminal, or a server. Of course, the electronic device may also be a display device such as a television.

In a possible application scenario, taking the electronic device as a TV set as an example, the real-time scene classification of the video screen played by the TV is an example. In order to better analyze the video screen, the TV festival can first analyze the video Perform scene recognition on the images contained in the video, so as to process the video picture according to the scene category to which the video belongs, combined with the downstream algorithm, for example, optimize the picture quality of the video picture.

In a possible implementation, after it is determined that the electronic device receives a processing request for scene recognition of an image in a video, the image is determined as an image to be recognized, and based on the image to be recognized, this Apply for the provided scene recognition method and process accordingly.

Wherein, the electronic device that performs scene recognition receives a processing request to perform scene recognition on an image in a video, which mainly includes at least one of the following situations:

Situation 1. When scene recognition is required, the user can input a service processing request for scene recognition to the smart device. After the smart device receives the service processing request, it can send the scene recognition electronic device to perform a scene on the image in the video. Identified processing request.

Scenario 2: After the smart device determines that the video is recorded, it generates a processing request for scene recognition of the image in the recorded video and sends it to the electronic device for scene recognition.

Situation 3. When the user needs to perform scene recognition on a specific video, he can input a business processing request for scene recognition of the video to the smart device. After receiving the business processing request, the smart device can send the scene recognition electronic The device sends a processing request for scene recognition of the image in the video.

It should be noted that the electronic device for scene recognition may be the same as the smart device, or may be different.

As a possible implementation, scene recognition conditions may also be preset, for example, when a video sent by a display device is received, scene recognition is performed on an image in the video, and when a video sent by a display device is received When the preset number of frame images is preset, scene recognition is performed on the preset number of frame images, and scene recognition is performed on the images in the currently acquired video according to a preset period, and the like. When the electronic device determines that the current time satisfies the preset scene recognition condition, it recognizes the scene of the image in a certain video.

In this application, when acquiring an image in a video, some video frames can be extracted from the video according to a preset frame extraction strategy, and the extracted part of the video frames can be converted into corresponding images, or the whole amount of frames can be extracted. , convert all video frames in the video into corresponding images.

In order to accurately determine the scene to which an image belongs, a scene recognition model is pre-trained. When the electronic device for scene recognition needs to perform scene recognition on a certain image to be recognized, the to-be-recognized image can be input into a pre-trained scene recognition model, so that the input to-be-recognized image can be determined through the pre-trained scene recognition model. The scene category to which it belongs.

Wherein, the process of training the scene recognition model has been described in the above-mentioned embodiment, and repeated description will not be repeated. For the above-mentioned training method of the scene recognition model, in the process of training the original scene recognition model based on the sample images in the sample set, through the original scene recognition model, the scene probability vector corresponding to the input sample image and the probability vector of the sample image can be obtained. The sample feature, so that the subsequent can be based on the scene probability vector and the scene label, the sample feature and the class center feature corresponding to the first scene category, the sample feature and the class center feature corresponding to the second scene category, The original scene recognition model is trained to obtain the trained scene recognition model, so that the trained scene recognition model can move closer to the class center feature of the scene category according to the image features of the images in the same scene category, while staying away from the class center feature of the scene category. The characteristics of the class-centric features of other scene categories, further combined with the feature level of the image, determine whether the scene category of the image can be identified and if the scene category of the image can be identified, the scene category to which the image belongs is not only realized. The scene category images included in the closed image set are accurately identified, and the scene category images that are not included in the closed image set can also be processed, thereby improving the accuracy, performance and naturalness of the scene recognition model.

It should be noted that the electronic device for training the scene recognition model and the electronic device for scene recognition may be the same or different.

Since the scene category to which the image to be recognized belongs is unpredictable and has a certain diversity, the scene category to which the image to be recognized actually belongs may not be the scene category included in the sample set used to train the scene recognition model, but If the scene category to which the image belongs is directly identified by the scene recognition model, erroneous results will be obtained, which will affect the processing of downstream algorithms. Moreover, if the scene classification to which an image belongs is a scene classification that can be recognized by the pre-trained scene recognition model, the image features of the image generally have a greater metric distance from the image features of the sample images belonging to the scene type in the sample set. The larger the metric distance, the smaller the metric distance from the image features of the sample images that do not belong to the scene type in the sample set. Therefore, in order to ensure that the scene recognition model can accurately identify the images belonging to the scene categories included in the closed image set, in this application, the image features of the images to be recognized can be obtained through the pre-trained scene recognition model, and for the images used for training The scene category included in the sample set of the scene recognition model is pre-obtained with the target class center feature of the scene category. After the to-be-recognized image is input into the pre-trained scene recognition model based on the above-mentioned embodiment, the image features of the to-be-recognized image can be acquired through the pre-trained scene recognition model. Then determine the similarity of the image feature with the target class center feature of each scene category, respectively. According to each similarity, it is determined whether the scene category to which the image to be recognized belongs is any scene category included in the sample set.

Wherein, the target class center feature may be the class center feature of each scene category included in the sample set during the last iterative training of the original scene recognition model.

In the specific implementation process, through the feature extraction layer in the scene recognition model, the image features of the input image to be recognized can be acquired. Then, through the feature output layer in the scene recognition model, the image features can be output. Then determine the similarity of the image feature with the target class center feature of each scene category, respectively. According to each similarity, it is determined whether the scene category to which the image to be recognized belongs is any scene category included in the sample set.

It should be noted that, in the last iterative training of the original scene recognition model, the acquisition method of the class center feature of each scene category contained in the sample set can refer to the acquisition methods in Case 1 and Case 2, and the repetition is insufficient. Repeat.

In a possible implementation manner, the image feature may be determined according to the metric distance between the image feature and the target class center feature of each scene category, respectively, which is similar to the target class center feature of each scene category. Spend. The metric distance can be obtained by means of Euclidean distance, cosine similarity, KL divergence function, or the like.

In a possible implementation manner, when determining the metric distance between the image feature and the target class center feature of the scene category to which each sample image belongs, it can be determined by the following Euclidean distance formula:

Among them, d(x, y _i ) represents the metric distance between the image feature x and the target class center feature of the ith scene category.

In another possible implementation, since the Euclidean distance represents the closeness of the two vectors in absolute distance, the cosine similarity represents the closeness of the two vectors in the direction. Therefore, when determining the metric distance between the image feature and the target class center feature of the scene category to which each sample image belongs, it can be determined by the following formula:

Among them, d(x, y _i ) represents the metric distance between the image feature x and the target class center feature of the ith scene category, and cos_sim(x, _yi ) represents the image feature x and the target class of the ith scene category Cosine similarity between central features, α ₁ represents the weight value corresponding to the Euclidean distance, α ₂ represents the weight value corresponding to the cosine similarity.

In a possible implementation, the similarity threshold may be configured manually, or the reference similarity between the target class center feature and other target class center features may be determined for each target class center feature. . Then, the similarity threshold is determined according to the reference similarity corresponding to each target class center feature respectively.

In a possible implementation manner, if the reference similarity is determined according to metric distances such as Euclidean distance, the similarity threshold may be determined according to the minimum value of each reference similarity.

In a possible implementation manner, if the reference similarity is determined according to metric distances such as cosine similarity, the similarity threshold may be determined according to the maximum value of each reference similarity.

In a possible implementation, if the similarity is determined according to the Euclidean distance and other metric distances, the smaller the similarity between the two image features, the higher the similarity between the two image features, the higher the similarity between the two image features. It is possible to belong to the same scene category; the greater the similarity of the two image features, the lower the similarity between the two image features, and the more likely that the two image features do not belong to the same scene category. Therefore, according to each of the similarity and the similarity threshold, it is determined whether each scene category includes the scene category to which the to-be-recognized image belongs. The image feature and the target class center feature corresponding to the similarity are very likely to belong to the same scene category, then it is determined that each scene category included in the sample set contains the scene category to which the image to be recognized belongs; if each similarity is not less than The similarity threshold indicates that the image feature of the image to be recognized and the central feature of each target class are from different scene categories, and it is determined that each scene category included in the sample set does not include the scene category to which the image to be identified belongs.

In a possible implementation, if the similarity is determined according to a metric distance such as cosine similarity, the smaller the similarity between the two image features is, the lower the similarity between the two image features is, and the lower the similarity between the two image features is. The less likely they belong to the same scene category; the smaller the similarity between the two image features, the higher the similarity between the two image features, the more likely the two image features belong to the same scene category. Therefore, according to each of the similarity and the similarity threshold, it is determined whether each scene category includes the scene category to which the to-be-identified image belongs, and if any similarity is greater than the similarity threshold, it means that the to-be-identified image features The image and the target class center feature corresponding to the similarity are very likely to belong to the same scene category, then it is determined that each scene category included in the sample set includes the scene category to which the image to be recognized belongs; if each similarity is not greater than The similarity threshold indicates that the image feature of the image to be recognized and the central feature of each target class are from different scene categories, and it is determined that each scene category included in the sample set does not include the scene category to which the image to be identified belongs.

In the specific implementation process, if it is determined that each scene category includes the scene category to which the to-be-recognized image belongs, it means that the scene category to which the to-be-recognized image belongs can be accurately determined by the scene recognition model, that is, the to-be-recognized image belongs to the scene category. If the scene category is known, the pre-trained scene recognition model is used to determine the scene category to which the to-be-recognized image belongs; if it is determined that each scene category does not contain the scene category to which the to-be-identified image belongs, it indicates that the to-be-recognized image belongs to the scene category. The to-be-recognized scene category cannot be accurately determined by the scene recognition model, that is, the scene category to which the to-be-recognized image belongs is unknown, and the scene category to which the image belongs is not identified.

Further, if it is determined that each scene category includes the scene category to which the to-be-recognized image belongs, then through the classification output layer in the scene-recognition model, based on the image features of the to-be-recognized image, it is possible to obtain and output the to-be-recognized image to which the image belongs. scene category.

Since the scene recognition model is pre-trained, and the scene recognition model is based on the scene probability vector of the sample image, the scene label of the sample image, the sample feature of the sample image, the class center feature corresponding to the first scene category of the sample image, the The sample feature and the class center feature corresponding to the second scene category of the sample image are obtained by training the original scene recognition model, so that based on the scene recognition model, in the process of recognizing the scene category to which the image to be recognized belongs, according to The image features of the images in the same scene category are closer to the class center features of the scene category, while being far away from the characteristics of the class center features of other scene categories. When the scene category of an image can be identified, the scene category to which the image belongs can not only accurately identify the scene category images included in the closed image set, but also process the scene categories that do not belong to the closed image set. images, improving the accuracy, performance, and naturalness of scene recognition models.

Example 4:

In the following, taking the electronic device for scene recognition as a TV set as an example, the scene recognition method provided by the present application will be described in detail through specific embodiments. FIG. 5 is a schematic diagram of a specific scene recognition process provided by some embodiments of the present application. The process includes: :

S501: Obtain a pre-trained scene recognition model.

S502: Determine an image feature of an image to be recognized by using a pre-trained scene recognition model.

S503: Determine the image features and the similarity with the target class center feature of each scene category respectively.

S504: If the similarity is the Euclidean distance, determine whether there is any similarity smaller than the similarity threshold, if so, execute S505, otherwise, execute S506.

S505: Determine the scene category to which the image to be recognized belongs by the scene recognition model.

S506: Do not continue to identify the scene category to which the image to be identified belongs.

Example 5:

The application provides a scene recognition model training device, and FIG. 6 is a schematic structural diagram of a scene recognition model training device provided by some embodiments of the application, and the device includes:

The obtaining unit 61 is configured to obtain any sample image in the sample set; wherein, the sample image corresponds to a scene label, and the scene label is used to identify the first scene category to which the sample image belongs;

The processing unit 62 is configured to determine the scene probability vector corresponding to the sample image and the sample feature of the sample image through the original scene recognition model; wherein, the scene probability vector includes that the sample image belongs to each scene category respectively the probability value;

A training unit 63 is configured to, based on the scene probability vector and the scene label, the sample feature, and the class center feature corresponding to the first scene category, the sample feature, and the class center feature corresponding to the second scene category, The original scene recognition model is trained to obtain the trained scene recognition model; wherein the second scene category is a scene category other than the first scene category in each scene category.

In some possible implementations, the training unit 63 is further configured to, for each iteration training of the original scene recognition model, obtain the each scene in the sample set through the scene recognition model of the current iteration The sample features of the sample images of the categories; according to each of the sample features, determine the candidate class center features corresponding to each scene category respectively; based on each of the candidate class center features, determine the Class-center features of each scene category; or, obtaining sample features of each of the sample images in the sample set through a pre-trained feature extraction model; clustering the sample features of each of the sample images to determine the Describe the class-centric features of each scene category.

In some possible implementations, the training unit 63 is specifically configured to, for each scene category, determine the target sample correctly identified by the currently iterative scene recognition model in each sample image of the scene category image; according to the sample feature of the target sample image and the weight value of the target sample image, determine a weighted average vector, and based on the weighted average vector, determine the candidate class center feature corresponding to the scene category; wherein, the target The weight value of the sample image is preset, or determined according to the probability value of the target sample image belonging to the scene category obtained through the scene recognition model of the current iteration; or, for each scene category , determine the target sample image correctly identified by the current iterative scene recognition model in each sample image of the scene category; based on the preset target algorithm, obtain the target feature in the sample features of the target sample image, and based on For the target feature, the candidate class center feature corresponding to the scene category is determined; wherein, the target feature is a principal component feature, or a normalized feature.

In some possible implementations, the training unit 63 is specifically configured to, if the current iteration is the first iteration, determine each candidate class center feature as each scene in the next iteration of training The class center feature of the category; if the current iteration is not the first iteration, for each scene category, determine the candidate class center feature corresponding to the scene category, and the class center corresponding to the scene category determined by the current iteration The difference vector of the features; according to the difference vector, the preconfigured weight vector, and the class center feature corresponding to the scene category determined by the current iteration, determine the class center feature corresponding to the scene category in the next iterative training.

In some possible implementations, the training unit 63 is specifically configured to determine a first loss value, a second loss value and a third loss value; wherein the first loss value is based on the scene probability vector and The scene label is determined; the second loss value is determined based on the sample feature and the class center feature corresponding to the first scene category; the third loss value is based on the sample feature and the first The class center feature corresponding to the second scene category is determined; according to the first loss value and its corresponding first weight value, the second loss value and its corresponding second weight value, and the third loss value and its corresponding The third weight value is determined, and the comprehensive loss value is determined; based on the comprehensive loss value, the original scene recognition model is trained.

Because in the process of training the original scene recognition model based on the sample images in the sample set, through the original scene recognition model, the scene probability vector corresponding to the input sample image and the sample features of the sample image can be obtained, so that the follow-up can be based on the The scene probability vector and the scene label, the sample feature and the class center feature corresponding to the first scene category, the sample feature and the class center feature corresponding to the second scene category, to train the original scene recognition model , to obtain the trained scene recognition model, so that the trained scene recognition model can move closer to the class center feature of the scene category according to the image features of the images in the same scene category, while staying away from the characteristics of the class center features of other scene categories , further combined with the feature level of the image, to determine whether the scene category of the image can be identified and if the scene category of the image can be identified, the scene category to which the image belongs, not only realizes accurate identification belonging to a closed image set The included scene category images can also process scene category images that are not included in the closed image set, which improves the accuracy, performance and naturalness of the scene recognition model.

Example 6:

FIG. 7 is a schematic structural diagram of a scene recognition apparatus provided by some embodiments of the present application. The present application provides a scene recognition apparatus, including:

The first processing module 71 is used to determine the image feature of the image to be recognized through a pre-trained scene recognition model;

The second processing module 72 is used to determine the similarity between the image features and the target class center feature of each scene category;

The third processing module 73 is configured to determine whether each scene category includes the scene category to which the to-be-recognized image belongs according to each similarity and the similarity threshold; if it is determined that each scene category includes the The scene category to which the image to be recognized belongs, then the scene recognition model is used to determine the scene category to which the image to be recognized belongs; if it is determined that each scene category does not include the scene category to which the image to be recognized belongs , the scene category to which the to-be-identified image belongs is not continued to be identified.

Since the scene recognition model is pre-trained, and the scene recognition model is based on the scene probability vector of the sample image, the scene label of the sample image, the sample feature of the sample image, the class center feature corresponding to the first scene category of the sample image, the The sample feature and the class center feature corresponding to the second scene category of the sample image are obtained by training the original scene recognition model, so that based on the scene recognition model, in the process of recognizing the scene category to which the image to be recognized belongs, according to The image features of the images in the same scene category are closer to the class center features of the scene category, while being far away from the characteristics of the class center features of other scene categories. When the scene category of an image can be identified, the scene category to which the image belongs can not only accurately identify the scene category images included in the closed image set, but also process the scene categories that do not belong to the closed image set. images, improving the accuracy, performance, and naturalness of scene recognition models.

Example 7:

FIG. 8 is a schematic structural diagram of an electronic device provided by some embodiments of the present application. On the basis of the above-mentioned embodiments, the present application also provides an electronic device, as shown in FIG. 8 , including: a processor 81 , a communication The interface 82, the memory 83 and the communication bus 84, wherein the processor 81, the communication interface 82, and the memory 83 complete the communication with each other through the communication bus 84;

A computer program is stored in the memory 83, and when the program is executed by the processor 81, the processor 81 is caused to perform the following steps:

Obtain any sample image in the sample set; wherein, the sample image corresponds to a scene label, and the scene label is used to identify the first scene category to which the sample image belongs;

Determine the scene probability vector corresponding to the sample image and the sample feature of the sample image through the original scene recognition model; wherein, the scene probability vector includes the probability value of the sample image belonging to each scene category respectively;

Identify the original scene based on the scene probability vector and the scene label, the sample feature and the class center feature corresponding to the first scene category, the sample feature and the class center feature corresponding to the second scene category The model is trained to obtain a trained scene recognition model; wherein the second scene category is a scene category other than the first scene category in each of the scene categories.

Since the principle of the above electronic device for solving the problem is similar to that of the scene recognition model training method, the implementation of the above electronic device can refer to the implementation of the method, and the repetition will not be repeated.

The communication bus mentioned in the above electronic device may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an Extended Industry Standard Architecture (Extended Industry Standard Architecture, EISA) bus or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

The communication interface 82 is used for communication between the above-mentioned electronic device and other devices.

The memory may include random access memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

The above-mentioned processor may be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it may also be a digital instruction processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array, or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

Because in the process of training the original scene recognition model based on the sample images in the sample set, through the original scene recognition model, the scene probability vector corresponding to the input sample image and the sample features of the sample image can be obtained, so that the follow-up can be based on the The scene probability vector and the scene label, the sample feature and the class center feature corresponding to the first scene category, the sample feature and the class center feature corresponding to the second scene category, to train the original scene recognition model , to obtain the trained scene recognition model, so that the trained scene recognition model can move closer to the class center feature of the scene category according to the image features of the images in the same scene category, while staying away from the characteristics of the class center features of other scene categories , further combined with the feature level of the image, to determine whether the scene category of the image can be identified and if the scene category of the image can be identified, the scene category to which the image belongs, not only realizes accurate identification belonging to a closed image set The included scene category images can also process scene category images that are not included in the closed image set, which improves the accuracy, performance and naturalness of the scene recognition model.

Example 8:

FIG. 9 is a schematic structural diagram of an electronic device provided by some embodiments of the present application. On the basis of the above embodiments, the present application also provides an electronic device, as shown in FIG. 9 , including: a processor 91, a communication The interface 92, the memory 93 and the communication bus 94, wherein the processor 91, the communication interface 92, and the memory 93 complete the communication with each other through the communication bus 94;

A computer program is stored in the memory 93, and when the program is executed by the processor 91, the processor 91 is caused to perform the following steps:

Determine the image features of the image to be recognized through the pre-trained scene recognition model;

Determine the similarity of the image feature with the target class center feature of each scene category;

According to each of the similarity and the similarity threshold, determine whether each scene category includes the scene category to which the to-be-recognized image belongs;

If it is determined that each scene category includes the scene category to which the to-be-recognized image belongs, determining the scene category to which the to-be-identified image belongs by using the scene recognition model;

If it is determined that each scene category does not include the scene category to which the to-be-identified image belongs, the scene category to which the to-be-identified image belongs is not continued to be identified.

Since the principle of the above electronic device for solving the problem is similar to the scene recognition method, the implementation of the above electronic device can refer to the implementation of the method, and the repetition will not be repeated.

The communication bus mentioned in the above electronic device may be a peripheral component interconnect standard (Peripheral Component Interconnect, PCI) bus or an Extended Industry Standard Architecture (Extended Industry Standard Architecture, EISA) bus or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

The communication interface 92 is used for communication between the above-mentioned electronic device and other devices.

The memory may include random access memory (Random Access Memory, RAM), and may also include non-volatile memory (Non-Volatile Memory, NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

The above-mentioned processor may be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it may also be a digital instruction processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array, or other Programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

Since the scene recognition model is pre-trained, and the scene recognition model is based on the scene probability vector of the sample image, the scene label of the sample image, the sample feature of the sample image, the class center feature corresponding to the first scene category of the sample image, the The sample feature and the class center feature corresponding to the second scene category of the sample image are obtained by training the original scene recognition model, so that based on the scene recognition model, in the process of recognizing the scene category to which the image to be recognized belongs, according to The image features of the images in the same scene category are closer to the class center features of the scene category, while being far away from the characteristics of the class center features of other scene categories. When the scene category of an image can be identified, the scene category to which the image belongs can not only accurately identify the scene category images included in the closed image set, but also process the scene categories that do not belong to the closed image set. images, improving the accuracy, performance, and naturalness of scene recognition models.

Example 9:

On the basis of the foregoing embodiments, the present application further provides a computer-readable storage medium, where a computer program executable by a processor is stored in the computer-readable storage medium, when the program is executed on the processor When running, the processor implements the following steps when executing:

Obtain any sample image in the sample set; wherein, the sample image corresponds to a scene label, and the scene label is used to identify the first scene category to which the sample image belongs;

Determine the scene probability vector corresponding to the sample image and the sample feature of the sample image through the original scene recognition model; wherein, the scene probability vector includes the probability value of the sample image belonging to each scene category respectively;

Identify the original scene based on the scene probability vector and the scene label, the sample feature and the class center feature corresponding to the first scene category, the sample feature and the class center feature corresponding to the second scene category The model is trained to obtain a trained scene recognition model; wherein the second scene category is a scene category other than the first scene category in each of the scene categories.

Since the principle of solving the problem by the computer-readable medium provided above is similar to the scene recognition model training method, after the processor executes the computer program in the computer-readable medium, the steps to be implemented can be referred to in the implementation of the method, and the repetition is not repeated. Repeat.

Because in the process of training the original scene recognition model based on the sample images in the sample set, through the original scene recognition model, the scene probability vector corresponding to the input sample image and the sample features of the sample image can be obtained, so that the follow-up can be based on the The scene probability vector and the scene label, the sample feature and the class center feature corresponding to the first scene category, the sample feature and the class center feature corresponding to the second scene category, to train the original scene recognition model , to obtain the trained scene recognition model, so that the trained scene recognition model can move closer to the class center feature of the scene category according to the image features of the images in the same scene category, while staying away from the characteristics of the class center features of other scene categories , further combined with the feature level of the image, to determine whether the scene category of the image can be identified and if the scene category of the image can be identified, the scene category to which the image belongs, not only realizes accurate identification belonging to a closed image set The included scene category images can also process scene category images that are not included in the closed image set, which improves the accuracy, performance and naturalness of the scene recognition model.

Example 10:

On the basis of the foregoing embodiments, the present application further provides a computer-readable storage medium, where a computer program executable by a processor is stored in the computer-readable storage medium, when the program is executed on the processor When running, the processor implements the following steps when executing:

Determine the image features of the image to be recognized through the pre-trained scene recognition model;

Determine the similarity of the image feature with the target class center feature of each scene category;

According to each of the similarity and the similarity threshold, determine whether each scene category includes the scene category to which the to-be-recognized image belongs;

If it is determined that each scene category includes the scene category to which the to-be-recognized image belongs, determining the scene category to which the to-be-identified image belongs by using the scene recognition model;

If it is determined that each scene category does not include the scene category to which the to-be-identified image belongs, the scene category to which the to-be-identified image belongs is not continued to be identified.

Since the principle of solving the problem by the computer-readable medium provided above is similar to that of the scene recognition method, after the processor executes the computer program in the computer-readable medium, the steps to be implemented can be referred to the implementation of the method, and the repetition will not be repeated. .

Since the scene recognition model is pre-trained, and the scene recognition model is based on the scene probability vector of the sample image, the scene label of the sample image, the sample feature of the sample image, the class center feature corresponding to the first scene category of the sample image, the The sample feature and the class center feature corresponding to the second scene category of the sample image are obtained by training the original scene recognition model, so that based on the scene recognition model, in the process of recognizing the scene category to which the image to be recognized belongs, according to The image features of the images in the same scene category are closer to the class center features of the scene category, while being far away from the characteristics of the class center features of other scene categories. When the scene category of an image can be identified, the scene category to which the image belongs can not only accurately identify the scene category images included in the closed image set, but also process the scene categories that do not belong to the closed image set. images, improving the accuracy, performance, and naturalness of scene recognition models.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims (10)

1. A method for training a scene recognition model, the method comprising:

acquiring any sample image in a sample set; the sample image corresponds to a scene label, and the scene label is used for identifying a first scene category to which the sample image belongs;

determining scene probability vectors corresponding to the sample images and sample characteristics of the sample images through an original scene recognition model; wherein the scene probability vector comprises probability values that the sample image respectively belongs to each scene category;

training the original scene recognition model based on the scene probability vector, the scene label, the sample feature, the class center feature corresponding to the first scene category, the sample feature and the class center feature corresponding to the second scene category to obtain a trained scene recognition model; wherein the second scene category is a scene category other than the first scene category in each of the scene categories.

2. The method of claim 1, wherein obtaining the class-centered feature for each scene category comprises:

for each iteration training of the original scene recognition model, acquiring sample characteristics of the sample image of each scene category in the sample set through the current iteration scene recognition model; determining candidate class center features respectively corresponding to each scene category according to each sample feature; determining the class center feature of each scene category in the next iterative training based on each candidate class center feature;

obtaining the sample characteristics of each sample image in the sample set through a pre-trained characteristic extraction model; and clustering the sample features of each sample image, and determining the class center feature of each scene category.

3. The method according to claim 1, wherein the determining, according to each of the sample features, a candidate class-center feature corresponding to each of the scene categories respectively comprises:

for each scene category, determining a target sample image correctly identified by the scene identification model of the current iteration in each sample image of the scene category; determining a weighted average vector according to the sample characteristics of the target sample image and the weight value of the target sample image, and determining candidate class center characteristics corresponding to the scene type based on the weighted average vector; the weight value of the target sample image is preset, or is determined according to a probability value of the target sample image belonging to the scene category, which is acquired through the current iterative scene identification model; or

For each scene category, determining a target sample image correctly identified by the scene identification model of the current iteration in each sample image of the scene category; acquiring a target feature in the sample features of the target sample image based on a preset target algorithm, and determining a candidate class center feature corresponding to the scene category based on the target feature; wherein the target feature is a principal component feature, or a normalized feature.

4. The method of claim 3, wherein said determining a class center feature for said each scene class in a next iteration of training based on each of said candidate class center features comprises:

if the current iteration is the first iteration, determining each candidate class center feature as the class center feature of each scene class in the next iteration training;

if the current iteration is not the first iteration, determining a candidate class center feature corresponding to the scene category and a difference vector of the class center feature corresponding to the scene category determined by the current iteration for each scene category; and determining the class center feature corresponding to the scene category in the next iteration training according to the difference vector, the pre-configured weight vector and the class center feature corresponding to the scene category determined by the current iteration.

5. The method of claim 1, wherein the training the original scene recognition model based on the scene probability vector and the scene label, the sample feature and the class-centered feature corresponding to the first scene class, the sample feature and the class-centered feature corresponding to the second scene class comprises:

determining a first loss value, a second loss value and a third loss value; wherein the first loss value is determined based on the scene probability vector and the scene tag; the second loss value is determined based on the sample feature and a class center feature corresponding to the first scene category; the third loss value is determined based on the sample feature and a class center feature corresponding to the second scene category;

determining a comprehensive loss value according to the first loss value and a first weight value corresponding to the first loss value, the second loss value and a second weight value corresponding to the second loss value, and a third loss value and a third weight value corresponding to the third loss value;

and training the original scene recognition model based on the comprehensive loss value.

6. A method for scene recognition, the method comprising:

determining the image characteristics of an image to be recognized through a pre-trained scene recognition model;

determining the similarity of the image features and the target class center features of each scene class;

determining whether each scene category comprises the scene category to which the image to be identified belongs according to each similarity and a similarity threshold;

if it is determined that each scene category comprises the scene category to which the image to be identified belongs, determining the scene category to which the image to be identified belongs through the scene identification model;

and if the scene category to which the image to be identified belongs is determined not to be contained in each scene category, the scene category to which the image to be identified belongs is not continuously identified.

7. An apparatus for training a scene recognition model, the apparatus comprising:

the acquisition unit is used for acquiring any sample image in the sample set; the sample image corresponds to a scene label, and the scene label is used for identifying a first scene category to which the sample image belongs;

the processing unit is used for determining a scene probability vector corresponding to the sample image and the sample characteristics of the sample image through an original scene recognition model; wherein the scene probability vector comprises probability values that the sample image respectively belongs to each scene category;

a training unit, configured to train the original scene identification model based on the scene probability vector and the scene label, the sample feature, the class center feature corresponding to the first scene category, the sample feature, and the class center feature corresponding to the second scene category, so as to obtain a trained scene identification model; wherein the second scene category is a scene category other than the first scene category in each of the scene categories.

8. A scene recognition apparatus, characterized in that the apparatus comprises:

the first processing module is used for determining the image characteristics of the image to be recognized through a pre-trained scene recognition model;

the second processing module is used for determining the similarity between the image characteristics and the target class center characteristics of each scene class;

a third processing module, configured to determine, according to each of the similarities and a similarity threshold, whether each of the scene categories includes a scene category to which the image to be identified belongs; if it is determined that each scene category comprises the scene category to which the image to be identified belongs, determining the scene category to which the image to be identified belongs through the scene identification model; and if the scene category to which the image to be identified belongs is determined not to be contained in each scene category, the scene category to which the image to be identified belongs is not continuously identified.

9. An electronic device, characterized in that the electronic device comprises a processor for implementing the steps of the scene recognition model training method as claimed in any one of claims 1 to 5, or the steps of the scene recognition method as claimed in claim 6, when executing a computer program stored in a memory.

10. A computer-readable storage medium, characterized in that it stores a computer program which, when being executed by a processor, carries out the steps of the scene recognition model training method as defined in any one of claims 1 to 5, or carries out the steps of the scene recognition method as defined in claim 6.

Images