CN116091867A - A model training, image recognition method, device, equipment and storage medium - Google Patents

Abstract

The application provides a model training method, an image recognition device, equipment and a storage medium, wherein the method comprises the following steps: randomly acquiring a plurality of image episodes in a source domain data set; constructing a task-aware self-adaptive learning network model; inputting the image episode into the self-adaptive learning network model to obtain a feature map of a support sample and a query sample in the image episode; determining classification loss according to the feature graphs of the support samples and the query samples, determining adaptive loss according to the domain offset of the image episode and the target domain data set, and determining overall loss according to the classification loss and the adaptive loss; and adjusting the self-adaptive learning network model according to the overall loss until the overall loss converges. In the method, the domain offset is introduced into the loss function, so that the trained model can give consideration to the target data set with different domain offsets, and a more accurate image recognition effect is achieved.

Description

A model training, image recognition method, device, equipment and storage medium

Technical Field

The present application relates to the field of image processing technology, and more specifically, to a model training, image recognition method, device, equipment and storage medium.

Background Art

Traditional deep learning models have demonstrated excellent generalization performance by training on a large number of labeled samples. However, rich samples and reliable annotations are difficult to obtain in practical applications, such as rare disease diagnosis and fine-grained recognition. Inspired by the ability of humans to quickly learn new knowledge, small sample learning aims to achieve that when there are only a small number of labeled samples in each category, the model can well identify the sample to be tested.

However, most current image recognition models using small-sample learning only focus on how to quickly adapt to new categories, without considering the domain shift problem between the test task and the training domain.

Summary of the invention

The problem addressed by this application is that current small-sample learning image recognition models do not consider the domain shift problem between the test task and the training domain.

To solve the above problems, the first aspect of the present application provides a model training method, comprising:

A plurality of image episodes are randomly obtained in a source domain dataset, each of which includes support samples and query samples of multiple categories, and the support samples and query samples in the image episodes are labeled;

Constructing task-aware adaptive learning network models;

Inputting the image episode into the adaptive learning network model to obtain feature graphs of support samples and query samples in the image episode;

Determine a classification loss according to the feature graphs of the support sample and the query sample, determine an adaptive loss according to the domain shift of the image interlude and a target domain dataset, and determine an overall loss according to the classification loss and the adaptive loss;

The adaptive learning network model is adjusted according to the overall loss until the overall loss converges.

The second aspect of the present application provides an image recognition method, which includes:

Obtaining an image episode to be identified in a target domain dataset, wherein the image episode to be identified includes support samples and query samples of multiple categories, and the support samples have been labeled;

Obtaining a pre-trained adaptive learning network model, wherein the adaptive learning network model is trained by the aforementioned model training method;

Adjusting the adaptive learning network model through labeled support samples;

Determine the feature graphs of the support samples and the query samples in the image episode to be identified by using the adjusted adaptive learning network model;

The category of the query sample is determined according to the feature graph of the query sample and the feature graphs of the labeled support samples.

The third aspect of the present application provides a model training device, which includes:

A training acquisition module, which is used to randomly acquire multiple image episodes in a source domain dataset, each image episode includes support samples and query samples of multiple categories, and the support samples and query samples in the image episode are labeled;

A model building module, which is used to build a task-aware adaptive learning network model;

A feature acquisition module, which is used to input the image episode into the adaptive learning network model to obtain feature graphs of support samples and query samples in the image episode;

a loss determination module, configured to determine a classification loss according to feature maps of the support sample and the query sample, determine an adaptive loss according to a domain shift between the image interlude and a target domain dataset, and determine an overall loss according to the classification loss and the adaptive loss;

A model training module is used to adjust the adaptive learning network model according to the overall loss until the overall loss converges.

A fourth aspect of the present application provides an image recognition device, comprising:

A test acquisition module, which is used to acquire image episodes to be identified in a target domain dataset, wherein the image episodes to be identified include support samples and query samples of multiple categories, and the support samples are labeled;

A model acquisition module, which is used to acquire a pre-trained adaptive learning network model, wherein the adaptive learning network model is trained by the aforementioned model training method;

A model adjustment module, which is used to adjust the adaptive learning network model through labeled support samples;

A model output module, which is used to determine the feature graphs of the supporting samples and the query samples in the image episode to be identified through the adjusted adaptive learning network model;

The category determination module is used to determine the category of the query sample according to the feature graph of the query sample and the feature graphs of the labeled support samples.

A fifth aspect of the present application provides a terminal device, comprising: a memory and a processor;

The memory is used to store programs;

The processor, coupled to the memory, is used to execute the program to execute the model training method described above, or to execute the image recognition method described above.

In a sixth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the above-mentioned model training method, or to implement the above-mentioned image recognition method.

In this application, by introducing domain offset into the loss function, the trained model can take into account target data sets with different domain offsets to achieve more accurate image recognition effects.

In this application, according to the domain offset size between the task to be tested and the source domain, the optimal task-specific parameter strategy is adaptively learned for each test task. At the same time, different optimal inference network structure diagrams are obtained for tasks to be tested with different domain offsets, thereby improving the accuracy of small sample image recognition.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a model training method according to an embodiment of the present application;

FIG2 is a schematic diagram of an adaptive learning network model according to the present application;

FIG3 is a flow chart of a model training method based on an adaptive learning network model according to an embodiment of the present application;

FIG4 is a flow chart of a model training method based on a residual block according to an embodiment of the present application;

FIG5 is a flow chart of a model training method based on an adaptive module layer according to an embodiment of the present application;

FIG6 is a flow chart of a model training method based on a gated network according to an embodiment of the present application;

FIG7 is a flow chart of an image recognition method according to an embodiment of the present application;

FIG8 is a structural block diagram of a model training device according to an embodiment of the present application;

FIG9 is a structural block diagram of an image recognition device according to an embodiment of the present application;

FIG10 is a structural block diagram of a terminal device according to an embodiment of the present application.

DETAILED DESCRIPTION

In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below in conjunction with the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.

Traditional deep learning models have demonstrated excellent generalization performance by training on a large number of labeled samples. However, rich samples and reliable annotations are difficult to obtain in practical applications, such as rare disease diagnosis and fine-grained recognition. Inspired by the ability of humans to quickly learn new knowledge, small sample learning aims to achieve that when there are only a small number of labeled samples in each category, the model can well identify the sample to be tested.

Recently, meta-learning-based few-shot image recognition methods have made great progress when the test data is derived from the source domain. However, more and more studies have shown that these methods have significantly insufficient generalization ability when the test data is heterogeneous with the source domain. This limitation is attributed to the fact that most few-shot image recognition models only focus on how to quickly adapt to new categories, but little or no effort is made to understand and address the domain shift problem between the test task and the training domain.

To address the above problems, the present application provides a new model training scheme, which can solve the problem that the current small-sample learning image recognition model does not consider the domain offset between the test task and the training domain by introducing the domain offset between the image interlude and the target domain dataset to determine the adaptive loss.

For ease of understanding, the following terms that may be used are explained here:

Activation function: Each neuron node in a neural network accepts the output value of the previous layer of neurons as the input value of this neuron, and passes the input value to the next layer. The input layer neuron node will directly pass the input attribute value to the next layer (hidden layer or output layer); in a multi-layer neural network, there is a functional relationship between the output of the upper layer node and the input of the lower layer node. This function is called an activation function.

Source domain: It refers to a domain that is different from the test sample but has rich supervision information.

Target domain: refers to the field where the test samples are located, with no labels or only a small number of labels; generally speaking, the source domain and the target domain belong to the same type of tasks, but have different distributions.

The embodiment of the present application provides a model training method, which can be performed by a model training device, and the model training device can be integrated in electronic devices such as pads, computers, servers, computers, server clusters, data centers, etc. As shown in Figure 1, it is a flow chart of a model training method according to an embodiment of the present application; wherein the model training method includes:

S100, randomly obtaining multiple image episodes in a source domain dataset, each image episode comprising support samples and query samples of multiple categories, and the support samples and query samples in the image episodes are labeled;

In this application, the source domain dataset is used to train the model, and the trained model is used to identify the images to be identified in the target domain dataset. The data in the source domain dataset is the training data, and the data in the target domain dataset is the test data.

The source domain dataset may be an ImageNet image dataset or another source. This application does not limit the method for obtaining the source domain dataset.

It should be noted that the labeling method of the support samples and query samples in the image interlude is not limited in this application.

In one embodiment, the multiple categories of the support samples in the image episode are the same as the multiple categories of the query samples.

In this application, better training effect is achieved by using support samples and query samples of the same category.

In one embodiment, in the image episode, each category contains 1-5 supporting samples.

Each sampled image episode contains a support sample set S and a query sample set Q, namely:

T = S ∪ Q, and

查询样本集记为

The supported sample set is recorded as:

The query sample set is denoted as

S200, building a task-aware adaptive learning network model;

The adaptive learning network model is an adaptive learning network model for small sample image recognition.

S300, inputting the image episode into the adaptive learning network model to obtain feature graphs of support samples and query samples in the image episode;

S400, determining a classification loss according to the feature graphs of the support sample and the query sample, determining an adaptive loss according to a domain shift between the image interlude and a target domain dataset, and determining an overall loss according to the classification loss and the adaptive loss;

The image interlude comes from a source domain dataset, so the domain offset between the image interlude and the target domain dataset is the domain offset between the source domain dataset and the target domain dataset.

In one embodiment, the classification loss is determined based on the feature graphs of the support samples and the query sample. First, the category of the query sample is predicted through the labeled support samples, and then the classification loss is determined based on the predicted category of the query sample and the labeled category.

S500, adjusting the adaptive learning network model according to the overall loss until the overall loss converges.

In this application, the model is adjusted according to the overall loss through back propagation.

In one implementation, the gradient of back propagation is calculated using the gradient of the soft decision to achieve better convergence effect.

In this application, by introducing domain offset into the loss function, the trained model can take into account target data sets with different domain offsets to achieve more accurate image recognition effects.

As shown in FIG2 , it is a schematic diagram of the following adaptive learning network model, and the following description is made in conjunction with the figure.

In one embodiment, the adaptive learning network model includes a plurality of residual blocks and fully connected layers connected in sequence;

As shown in FIG3 , the step of inputting the image episode into the adaptive learning network model to obtain feature graphs of support samples and query samples in the image episode includes:

S301, extracting features from the input image episode through a residual block to obtain an extracted intermediate feature map;

In the present application, every two residual blocks form a group, and multiple groups of residual blocks are connected in sequence; as multiple groups of residual blocks extract the input image interlude layer by layer, the size of the extracted feature map gradually decreases, and the number of feature maps increases exponentially.

S302, performing a linear transformation on the extracted feature map through a fully connected layer to obtain feature maps of the support sample and the query sample.

In this application, the classifier shown in FIG3 is the fully connected layer. The fully connected layers (FC) act as a "classifier" in the entire convolutional neural network.

By using residual blocks to form a residual network to extract the input image episodes layer by layer, the gradient vanishing problem in multi-layer networks can be avoided; through the linear transformation of the fully connected layer, the complexity of the adaptive learning network model is increased, allowing the model to express more complex features.

In one embodiment, the residual block includes an adaptive module layer, a batch normalization layer and a ReLU function;

As shown in FIG4 , the feature extraction of the input image episode by the residual block includes:

S311, extracting features from the output of the previous residual block through an adaptive module layer;

S312, normalizing the output of the adaptive module through a batch normalization layer;

Among them, the role of the batch normalization layer (Batch Normalization, BN) is to standardize the data, thereby accelerating the convergence of the model.

S313, mapping the normalized output result to the output end through the ReLU function.

In this application, the residual block is normalized by BN after convolution, and then ReLU is used as the activation function after adding the direct mapping unit. This residual setting method improves the accuracy of the model.

In one embodiment, the adaptive module layer includes a convolutional layer, a task adapter and a gating network, and the task adapter includes a plurality of task parameter convolutional layers;

As shown in FIG5 , the feature extraction of the output of the previous residual block through the adaptive module layer includes:

S321, performing a first feature extraction on the output of the previous residual block through a convolutional layer;

In one implementation, a first feature extraction is performed on the output of the previous residual block through a 3*3 convolutional layer.

S322, performing second feature extraction on the output of the previous residual block respectively through multiple task parameter convolution layers in the task adapter;

In one implementation, the task parameter convolution layer is a 1*1 convolution layer.

S323, generating a decision result based on the output of the previous residual block through a gating network, and making a decision on whether to execute each task parameter convolution layer in the task adapter based on the decision result;

S324, adding the result of the second feature extraction after the decision to the result of the first feature extraction as the output of the adaptive module layer.

In this application, the task adapter (TA) is parallel to the 3*3 convolution layer, and each adapter contains k specific task parameter 1*1 convolution layers, and whether each specific task parameter layer is executed or not is determined by the gating network.

表示第l个自适应模块的3*3卷积层，当第l个自适应模块的输入为：in,

Represents the 3*3 convolutional layer of the lth adaptive module. When the input of the lth adaptive module is:

任务适配器学习到的特征将与3*3卷积层学习到的特征相结合，即：

The features learned by the task adapter will be combined with the features learned by the 3*3 convolutional layer, namely:

表示特定任务适配器的第i层的特定任务学习函数。

表示门控网络生成的门决策，决定第i层的特定任务函数的执行与否，

其中，1表示执行，0表示不执行。in,

represents the task-specific learning function of the i-th layer of the task-specific adapter.

represents the gate decision generated by the gating network, which determines whether the specific task function of the i-th layer is executed or not.

Among them, 1 means execution, and 0 means no execution.

In one embodiment, the gated network includes a global average pooling layer, a prototype-like layer, a 1*1 convolutional layer, and an activation function;

As shown in FIG6 , the decision result is generated based on the output of the previous residual block through the gating network, including:

S331, compressing the spatial dimension of the output of the previous residual block through a global average pooling layer;

In this application, the input of the gating network is the output h _l-1 of the previous residual block, which first passes through a global average pooling layer to compress the spatial dimension of the feature map, namely:

u _l-1 =GAP(h _l-1 )

并且

in,

and

S332, determining the prototype feature of each category in the image episode at the current layer through the class prototype layer;

Among them, the prototype feature of each category at the current layer can be obtained by the following formula:

表示类别n的原型特征，S_n表示属于类别n的样本集合。并且

表示当前任务所有类别的原型特征。in,

represents the prototype feature of category n, and _Sn represents the sample set belonging to category n. And

Represents the prototype features of all categories of the current task.

S333, based on the prototype features of each category in the current layer, a decision result is generated through a 1*1 convolution layer and an activation function.

The prototype feature generates soft decisions through a 1*1 linear function and a Sigmoid activation function, namely:

σ表示Sigmoid激活函数，

in,

σ represents the Sigmoid activation function,

Then, a discrete decision (decision result) can be generated through a simple threshold algorithm, namely:

As shown in FIG. 2 , 0.6, 0.2, 1, ..., 0.3 are soft decisions, and 1, 0, 1, ..., 0 are discrete decisions/hard decisions/decision results of production.

In one embodiment, S400, a classification loss is determined based on feature maps of the support sample and the query sample, an adaptive loss is determined based on a domain offset between the image interlude and a target domain dataset, and in determining an overall loss based on the classification loss and the adaptive loss, the domain offset between the image interlude and the target domain dataset is quantified by a maximum average difference.

In one implementation, the quantization formula of the domain offset is:

将特征映射到再生希尔伯特空间，P_i为源域数据集中类别i的类原型，N_b代表源域数据集的类别数，P_j为源域数据集中类别j的类原型，k()为核函数，N_s为支持集样本数，

表示MMD度量是在再生希尔伯特空间进行的，等号左侧竖线为范数。in,

Map the features to the regenerated Hilbert space, _Pi is the class prototype of category i in the source domain dataset, _Nb represents the number of categories in the source domain dataset, _Pj is the class prototype of category j in the source domain dataset, k() is the kernel function, _Ns is the number of samples in the support set,

It indicates that the MMD measurement is performed in the regenerated Hilbert space, and the vertical line on the left side of the equal sign is the norm.

In one embodiment, the adaptive loss function is:

为第l层任务适配器的第i层的软决策。Where L is the number of adaptive modules, t is the domain offset, i is the number of layers of the task adapter, and represents the i-th layer of the task adapter.

is the soft decision of the i-th layer of the l-th layer task adapter.

In one embodiment, the overall loss function is:

为自适应损失，

为分类损失。Among them, λ is a hyperparameter used to balance the weights of the two losses.

is the adaptive loss,

is the classification loss.

The embodiment of the present application provides an image recognition method, which can be performed by an image recognition device, and the image recognition device can be integrated in electronic devices such as pads, computers, servers, computers, server clusters, data centers, etc. As shown in Figure 7, it is a flow chart of an image recognition method according to an embodiment of the present application; wherein the image recognition method includes:

S10, obtaining an image episode to be identified in a target domain dataset, wherein the image episode to be identified includes supporting samples and query samples of multiple categories, and the supporting samples have been labeled;

In this step, unlike the model training method, the query sample of the image episode to be identified is not labeled.

Each image episode to be identified is independently subjected to image recognition.

S20, obtaining a pre-trained adaptive learning network model, wherein the adaptive learning network model is trained by the above-mentioned model training method;

S30, adjusting the adaptive learning network model through the labeled support samples;

In one embodiment, the adaptive learning network model is adjusted using labeled support samples, including: inputting the labeled support samples into the adaptive learning network model to obtain a feature map of the support samples; determining the prototype features of each category at the current layer based on the feature map of the support samples and the labeled categories; calculating the cross entropy loss based on the distance between each support sample and the prototype features of the category at the current layer; optimizing the adaptive learning network model based on the cross entropy loss until convergence, to obtain the adjusted adaptive learning network model.

The cross entropy loss is the classification loss in the model training method.

The method for determining the prototype features of each category in the current layer has been described in the model training method and will not be repeated in this step.

The distance between each supporting sample and the prototype feature of the category at the current layer may be a Euclidean distance.

S40, determining feature graphs of supporting samples and query samples in the image episode to be identified by using the adjusted adaptive learning network model;

It should be noted that after the adaptive learning network model is adjusted, the feature graph of the supporting samples needs to be re-determined by the adjusted adaptive learning network model.

S50, determining the category of the query sample according to the feature graph of the query sample and the feature graphs of the labeled support samples.

In one embodiment, determining the category of the query sample based on the feature graph of the query sample and the feature graph of the labeled support samples includes: determining the prototype features of each category at the current layer based on the feature graph of the support samples and the labeled categories; determining the distance between each query sample and the prototype features of the category at the current layer based on the feature graph of each query sample; and selecting the category with the smallest distance as the recognition result of the query sample.

In this application, according to the domain offset size between the task to be tested and the source domain, the optimal task-specific parameter strategy is adaptively learned for each test task. At the same time, different optimal inference network structure diagrams are obtained for tasks to be tested with different domain offsets, thereby improving the accuracy of small sample image recognition.

It should be noted that in the above-mentioned model training method, the prediction of the category of the query sample by the labeled support samples is the same as the determination of the category of the query sample by the labeled support samples in the image recognition method (steps S20-S50). The difference is that in the model training method, the feature graphs of the support samples and the query samples are obtained by the untrained adaptive learning network model, and in the image recognition method, the feature graphs of the support samples and the query samples are obtained by the pre-trained adaptive learning network model. Based on this, the specific process of predicting the category of the query sample by the labeled support samples in the model training method will not be described in detail.

An embodiment of the present application provides a model training device for executing the model training method described in the above content of the present application. The model training device is described in detail below.

As shown in FIG8 , the model training device includes:

A training acquisition module 101 is used to randomly acquire multiple image episodes in a source domain dataset, each image episode includes support samples and query samples of multiple categories, and the support samples and query samples in the image episode are labeled;

A model building module 102, which is used to build a task-aware adaptive learning network model;

A feature acquisition module 103, which is used to input the image episode into the adaptive learning network model to obtain feature graphs of support samples and query samples in the image episode;

a loss determination module 104, configured to determine a classification loss according to the feature graphs of the support sample and the query sample, determine an adaptive loss according to a domain shift between the image interlude and a target domain dataset, and determine an overall loss according to the classification loss and the adaptive loss;

The model training module 105 is used to adjust the adaptive learning network model according to the overall loss until the overall loss converges.

In one embodiment, the adaptive learning network model includes a plurality of residual blocks and fully connected layers connected in sequence;

The feature acquisition module 103 is also used for:

The input image episode is subjected to feature extraction through the residual block to obtain an extracted intermediate feature map;

The extracted feature map is linearly transformed through a fully connected layer to obtain the feature maps of the support sample and the query sample.

In one embodiment, the residual block includes an adaptive module layer, a batch normalization layer and a ReLU function;

The feature acquisition module 103 is also used for:

Perform feature extraction on the output of the previous residual block through the adaptive module layer;

Normalizing the output of the adaptive module through a batch normalization layer;

The normalized output result is mapped to the output end through the ReLU function.

In one embodiment, the adaptive module layer includes a convolutional layer, a task adapter and a gating network, and the task adapter includes a plurality of task parameter convolutional layers;

The feature acquisition module 103 is also used for:

Perform the first feature extraction on the output of the previous residual block through the convolution layer;

Performing second feature extraction on the output of the previous residual block through multiple task parameter convolution layers in the task adapter;

Generate a decision result based on the output of the previous residual block through a gating network, and decide whether to execute each task parameter convolution layer in the task adapter based on the decision result;

The result of the second feature extraction after the decision is added to the result of the first feature extraction as the output of the adaptive module layer.

In one embodiment, the gated network includes a global average pooling layer, a prototype-like layer, a 1*1 convolutional layer, and an activation function;

The feature acquisition module 103 is also used for:

The output of the previous residual block is compressed in spatial dimension through the global average pooling layer;

Determine the prototype feature of each category in the image episode at the current layer through the class prototype layer;

According to the prototype features of each category in the current layer, the decision result is generated through a 1*1 convolution layer and an activation function.

The model training device provided in the above-mentioned embodiments of the present application and the model training method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

An embodiment of the present application provides an image recognition device for executing the image recognition method described in the above content of the present application. The image recognition device is described in detail below.

As shown in FIG9 , the image recognition device includes:

A test acquisition module 201 is used to acquire an image episode to be identified in a target domain dataset, wherein the image episode to be identified includes support samples and query samples of multiple categories, and the support samples are labeled;

A model acquisition module 202, which is used to acquire a pre-trained adaptive learning network model, wherein the adaptive learning network model is trained by the above-mentioned model training method;

A model adjustment module 203, which is used to adjust the adaptive learning network model through the labeled support samples;

A model output module 204, which is used to determine the feature graphs of the support samples and the query samples in the image episode to be identified through the adjusted adaptive learning network model;

The category determination module 205 is used to determine the category of the query sample according to the feature graph of the query sample and the feature graphs of the labeled support samples.

The image recognition device provided in the above-mentioned embodiment of the present application and the image recognition method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

The above describes the internal functions and structure of the model training device/image recognition device. As shown in FIG10 , in practice, the model training device/image recognition device can be implemented as a terminal device, including: a memory 301 and a processor 303 .

The memory 301 may be configured to store programs.

In addition, the memory 301 may also be configured to store various other data to support operations on the terminal device. Examples of such data include instructions for any application or method operating on the terminal device, contact data, phone book data, messages, pictures, videos, etc.

The memory 301 may be implemented by any type of volatile or nonvolatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

The processor 303 is coupled to the memory 301 and is configured to execute a program in the memory 301 to:

A plurality of image episodes are randomly obtained in a source domain dataset, each of which includes support samples and query samples of multiple categories, and the support samples and query samples in the image episodes are labeled;

Constructing task-aware adaptive learning network models;

Inputting the image episode into the adaptive learning network model to obtain feature graphs of support samples and query samples in the image episode;

Determine a classification loss according to the feature graphs of the support sample and the query sample, determine an adaptive loss according to the domain shift of the image interlude and a target domain dataset, and determine an overall loss according to the classification loss and the adaptive loss;

The adaptive learning network model is adjusted according to the overall loss until the overall loss converges.

In one embodiment, the adaptive learning network model includes a plurality of residual blocks and fully connected layers connected in sequence;

The processor 303 is specifically used for:

The input image episode is subjected to feature extraction through the residual block to obtain an extracted intermediate feature map;

The extracted feature map is linearly transformed through a fully connected layer to obtain the feature maps of the support sample and the query sample.

In one embodiment, the residual block includes an adaptive module layer, a batch normalization layer and a ReLU function;

The processor 303 is specifically used for:

Perform feature extraction on the output of the previous residual block through the adaptive module layer;

Normalizing the output of the adaptive module through a batch normalization layer;

The normalized output result is mapped to the output end through the ReLU function.

In one embodiment, the adaptive module layer includes a convolutional layer, a task adapter and a gating network, and the task adapter includes a plurality of task parameter convolutional layers;

The processor 303 is specifically used for:

Perform the first feature extraction on the output of the previous residual block through the convolution layer;

Performing second feature extraction on the output of the previous residual block through multiple task parameter convolution layers in the task adapter;

Generate a decision result based on the output of the previous residual block through a gating network, and decide whether to execute each task parameter convolution layer in the task adapter based on the decision result;

The result of the second feature extraction after the decision is added to the result of the first feature extraction as the output of the adaptive module layer.

In one embodiment, the gated network includes a global average pooling layer, a prototype-like layer, a 1*1 convolutional layer, and an activation function;

The processor 303 is specifically used for:

The output of the previous residual block is compressed in spatial dimension through the global average pooling layer;

Determine the prototype feature of each category in the image episode at the current layer through the class prototype layer;

According to the prototype features of each category in the current layer, the decision result is generated through a 1*1 convolution layer and an activation function.

Alternatively, the processor 303 is coupled to the memory 301 and is configured to execute a program in the memory 301 to:

Obtaining an image episode to be identified in a target domain dataset, wherein the image episode to be identified includes support samples and query samples of multiple categories, and the support samples are labeled;

Obtaining a pre-trained adaptive learning network model, wherein the adaptive learning network model is trained by the above-mentioned model training method;

Adjusting the adaptive learning network model through labeled support samples;

Determine the feature graphs of the support samples and the query samples in the image episode to be identified by using the adjusted adaptive learning network model;

The category of the query sample is determined according to the feature graph of the query sample and the feature graphs of the labeled support samples.

In the present application, only some components are schematically shown in FIG10 , which does not mean that the terminal device only includes the components shown in FIG10 .

The terminal device provided in this embodiment is based on the same inventive concept as the model training method or image recognition method provided in the embodiment of the present application, and has the same beneficial effects as the method adopted, run or implemented by the application stored therein.

Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

The present application is described with reference to the flowchart and/or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each process and/or box in the flowchart and/or block diagram, and the combination of the process and/or box in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one process or multiple processes in the flowchart and/or one box or multiple boxes in the block diagram.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.

In a typical configuration, a computing device includes one or more processors (CPU), input/output interfaces, network interfaces, and memory.

The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and/or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

The present application also provides a computer-readable storage medium corresponding to the model training method or image recognition method provided in the aforementioned embodiment, on which a computer program (i.e., program product) is stored. When the computer program is run by a processor, it will execute the model training method provided in any of the aforementioned embodiments, or execute the image recognition method provided in any of the aforementioned embodiments.

Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the model training method or image recognition method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

It should be noted that in the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims (10)

1. A model training method, characterized by comprising: A plurality of image episodes are randomly obtained in a source domain dataset, each of which includes support samples and query samples of multiple categories, and the support samples and query samples in the image episodes are labeled; Constructing task-aware adaptive learning network models; Inputting the image episode into the adaptive learning network model to obtain feature graphs of support samples and query samples in the image episode; Determine a classification loss according to the feature graphs of the support sample and the query sample, determine an adaptive loss according to the domain shift of the image interlude and a target domain dataset, and determine an overall loss according to the classification loss and the adaptive loss; The adaptive learning network model is adjusted according to the overall loss until the overall loss converges. 2. The method according to claim 1, characterized in that the adaptive learning network model comprises a plurality of residual blocks and fully connected layers connected in sequence; The step of inputting the image episode into the adaptive learning network model to obtain a feature graph of a support sample and a query sample in the image episode includes: Perform feature extraction on the input image episode through the residual block to obtain an extracted intermediate feature map; The extracted feature map is linearly transformed through a fully connected layer to obtain the feature maps of the support sample and the query sample. 3. The method according to claim 2, wherein the residual block comprises an adaptive module layer, a batch normalization layer and a ReLU function; The feature extraction of the input image episode by the residual block includes: Perform feature extraction on the output of the previous residual block through the adaptive module layer; Normalizing the output of the adaptive module through a batch normalization layer; The normalized output result is mapped to the output end through the ReLU function. 4. The method according to claim 3, characterized in that the adaptive module layer includes a convolution layer, a task adapter and a gating network, and the task adapter includes a plurality of task parameter convolution layers; The feature extraction of the output of the previous residual block through the adaptive module layer includes: Perform the first feature extraction on the output of the previous residual block through the convolution layer; Performing second feature extraction on the output of the previous residual block through multiple task parameter convolution layers in the task adapter; Generate a decision result based on the output of the previous residual block through a gating network, and decide whether to execute each task parameter convolution layer in the task adapter based on the decision result; The result of the second feature extraction after the decision is added to the result of the first feature extraction as the output of the adaptive module layer. 5. The method according to claim 4, characterized in that the gated network includes a global average pooling layer, a prototype-like layer, a 1*1 convolutional layer and an activation function; The generating of a decision result based on the output of the previous residual block through the gating network includes: The output of the previous residual block is compressed in spatial dimension through the global average pooling layer; Determine the prototype feature of each category in the image episode at the current layer through the class prototype layer; According to the prototype features of each category in the current layer, the decision result is generated through a 1*1 convolution layer and an activation function. 6. An image recognition method, comprising: Obtaining an image episode to be identified in a target domain dataset, wherein the image episode to be identified includes support samples and query samples of multiple categories, and the support samples are labeled; Obtain a pre-trained adaptive learning network model, wherein the adaptive learning network model is trained by the model training method described in any one of claims 1 to 5; Adjusting the adaptive learning network model through labeled support samples; Determine the feature graphs of the support samples and the query samples in the image episode to be identified by using the adjusted adaptive learning network model; The category of the query sample is determined according to the feature graph of the query sample and the feature graphs of the labeled support samples. 7. A model training device, characterized in that it comprises: A training acquisition module, which is used to randomly acquire multiple image episodes in a source domain dataset, each image episode includes support samples and query samples of multiple categories, and the support samples and query samples in the image episode are labeled; A model building module, which is used to build a task-aware adaptive learning network model; A feature acquisition module, which is used to input the image episode into the adaptive learning network model to obtain feature graphs of support samples and query samples in the image episode; a loss determination module, configured to determine a classification loss according to feature maps of the support sample and the query sample, determine an adaptive loss according to a domain shift between the image interlude and a target domain dataset, and determine an overall loss according to the classification loss and the adaptive loss; A model training module is used to adjust the adaptive learning network model according to the overall loss until the overall loss converges. 8. An image recognition device, comprising: A test acquisition module, which is used to acquire image episodes to be identified in a target domain dataset, wherein the image episodes to be identified include support samples and query samples of multiple categories, and the support samples are labeled; A model acquisition module, which is used to acquire a pre-trained adaptive learning network model, wherein the adaptive learning network model is trained by the model training method described in any one of claims 1 to 5; A model adjustment module, which is used to adjust the adaptive learning network model through labeled support samples; A model output module, which is used to determine the feature graphs of the supporting samples and the query samples in the image episode to be identified through the adjusted adaptive learning network model; The category determination module is used to determine the category of the query sample according to the feature graph of the query sample and the feature graphs of the labeled support samples. 9. A terminal device, comprising: a memory and a processor; The memory is used to store programs; The processor, coupled to the memory, is used to execute the program to execute the model training method described in any one of claims 1-5, or to execute the image recognition method described in claim 6. 10. A computer-readable storage medium having a computer program stored thereon, characterized in that the program is executed by a processor to implement the model training method described in any one of claims 1 to 5, or to implement the image recognition method described in claim 6.

Images