CN116994076B - Small sample image recognition method based on double-branch mutual learning feature generation - Google Patents

Abstract

The invention discloses a small sample image recognition method based on double-branch mutual learning feature generation, which relates to the technical field of small sample image recognition, and comprises the following steps: acquiring a small sample image set to be identified to form a query set to be identified; sending each image in the query set to be identified into a first feature generation module of a pre-constructed global branch to generate a first semantic feature of each image; sending each image in the query set to be identified into a second characteristic generating module of a pre-constructed local branch to generate a second semantic characteristic of each image; adding the first semantic features and the second semantic features, and determining third semantic features of each image in the query set to be identified; and respectively calculating the similarity between the third semantic feature of each image in the query set to be identified and the prototypes of a plurality of categories in the support set to determine the image category of each image in the query set to be identified. The semantic relation between the local features and the global features of the sample is mined, and the technical effect of accurately identifying the small sample image is achieved.

Description

A small-sample image recognition method based on dual-branch mutual learning feature generation

Technical field

The present invention relates to the technical field of small sample image recognition, and more specifically, to a small sample image recognition method based on dual-branch mutual learning feature generation.

Background technique

In recent years, with large-scale data sets and huge computing resources, artificial intelligence algorithms represented by deep learning have made great achievements in image recognition-related fields such as face recognition, autonomous driving, and robots. However, deep learning requires a large number of It is often difficult to obtain data in practical applications. There are both personal privacy issues, such as face data, and problems with very few problem objects, such as the problem of identifying rare and protected animals. In addition, In addition, data annotation work often requires a lot of manpower and material resources, which hinders the development of deep learning technology in some image recognition fields. On the contrary, humans themselves can recognize a new object through a very small number of samples. Inspired by humans' rapid learning ability, researchers hope that after learning a large amount of data of a certain category, the machine learning model only needs to A small number of samples can be learned quickly, and the resulting small sample image recognition problem has gradually become a current research hotspot.

The core problem of small sample image recognition tasks is that the sample size is too small, resulting in low sample diversity. When the amount of data is limited, sample diversity can be improved through data augmentation. Data augmentation refers to using auxiliary data or auxiliary information to expand data or enhance features of the original small sample data set when the amount of data is limited. Data augmentation refers to adding new unlabeled data or synthetic labeled data to the original data set. Feature enhancement refers to adding features that facilitate classification to the feature space of the original sample to improve the feature diversity of the sample.

Existing methods based on feature enhancement mainly rely on the global semantic features of samples when generating new features. These methods generate new global semantic features by analyzing the similarities or differences of global semantic features between different samples. Although this method can increase the size and diversity of the data set, it also has a problem, that is, it ignores the local characteristic information of the sample. In the case of small samples, due to the limited amount of data, each sample may contain some unique or Important local feature information, which is very useful for distinguishing different categories or tasks. If only global semantic features are used to generate new features, these local feature information may be lost or confused, resulting in low-quality or inaccurate generated new features.

Contents of the invention

In view of the shortcomings of the existing technology, the present invention provides a small sample image recognition method based on dual-branch mutual learning feature generation.

According to one aspect of the present invention, a small sample image recognition method based on dual-branch mutual learning feature generation is provided, including:

Obtain a small sample image set to be identified to form a query set to be identified;

Send each image in the query set to be recognized to the first feature generation module of the pre-built global branch to generate the first semantic feature of each image;

Send each image in the query set to be recognized to the second feature generation module of the pre-built local branch to generate the second semantic feature of each image;

Add the first semantic feature and the second semantic feature of each image in the query set to be recognized to determine the third semantic feature of each image in the query set to be recognized;

The similarity between the third semantic feature of each image in the query set to be recognized and the prototypes of multiple categories in the support set is calculated respectively, and the image category of each image in the query set to be recognized is determined.

Optionally, a category prototype for each category in the support set is constructed as follows:

All images in the support set are input into the first feature generation module of the global branch in turn, and the fourth semantic feature of each image in the support set is output;

All images in the support set are input into the second feature generation module of the local branch in turn, and the fifth semantic feature of each image in the support set is output;

The fourth semantic feature and the fifth semantic feature of all images of each category in the support set are added and averaged to determine the category prototype for each category in the support set.

Optionally, the training process of the first feature generation module of the global branch and the second feature generation module of the local branch is as follows:

Based on the small sample image training set, a small sample recognition task is constructed, in which the small sample recognition task includes N categories, and each category includes M sample images;

Use the feature extraction network to extract features from each sample image in the small sample recognition task to determine the global features and local features of each sample image;

The first feature generation module of the global branch is trained through the global features of each sample image in the small sample recognition task;

The second feature generation module of the local branch is trained through the global features and local features of each sample image in the small sample recognition task;

Learn the training information of the global branch and the local branch from each other, and train the first feature generation module and the second feature generation module;

According to the preset training total loss function, the first feature generation module and the second feature generation module are optimized.

Optionally, train the first feature generation module of the global branch through the global features of each sample image in the small sample recognition task, including:

Mask the global features of M sample images of each category to obtain the global features of one sample image;

Replace the global features of the sample images of each category mask with learnable vectors;

Train the first feature generation module of the global branch based on the learnable vectors replaced by each category and the global features retained by the mask;

The first feature generation module is optimized according to the preset global branch loss function, where the global branch loss function includes a global prediction loss function and a global classification loss function.

Optionally, train the second feature generation module of the local branch through the global features and local features of each sample image in the small sample recognition task, including:

Select local features of a sample image in each category;

Train the second feature generation module based on the local features selected for each category and the preset M learnable vectors;

The second feature generation module is optimized according to the preset local branch loss function, where the local branch loss function includes a local prediction loss function and a local classification loss function.

Optionally, it also includes: calculating KL divergence as a mutual learning loss function in which the training information of the global branch and the local branch learn from each other.

Optionally, the total training loss function is the sum of the global branch loss function, the local branch loss function and the mutual learning loss function.

Optionally, separately calculate the similarity between the third semantic feature of each image in the query set to be recognized and the category prototypes of multiple categories in the support set, and determine the image category of each image in the query set to be recognized, including:

Calculate the similarity between the third semantic feature of each image in the query set to be recognized and the category prototypes of multiple categories in the support set, and determine the probability value of each image in the query set to be recognized belonging to each category in the support set;

The category with the largest probability value corresponding to each image in the query set to be recognized is taken as the image category of the image.

According to another aspect of the present invention, a small sample image recognition device based on dual-branch mutual learning feature generation is provided, including:

The acquisition module is used to obtain a small sample image set to be identified to form a query set to be identified;

The first generation module is used to send each image in the query set to be recognized to the first feature generation module of the pre-built global branch to generate the first semantic feature of each image;

The second generation module is used to send each image in the query set to be recognized to the second feature generation module of the pre-constructed local branch to generate the second semantic feature of each image;

The first determination module is used to add the first semantic feature and the second semantic feature of each image in the query set to be recognized, and determine the third semantic feature of each image in the query set to be recognized;

The second determination module is used to respectively calculate the similarity between the third semantic feature of each image in the query set to be recognized and multiple category prototypes in the support set, and determine the image category of each image in the query set to be recognized.

According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium stores a computer program, and the computer program is used to execute the method described in any of the above aspects of the present invention.

According to yet another aspect of the present invention, an electronic device is provided. The electronic device includes: a processor; a memory for storing instructions executable by the processor; and the processor for reading from the memory. Fetch the executable instructions and execute the instructions to implement the method described in any of the above aspects of the present invention.

Therefore, the present invention proposes a small sample image recognition method based on dual-branch mutual learning feature generation, and constructs a feature generation module based on local feature information. This module mines the semantic relationship between the local features and global features of the sample, and utilizes the sample's Local feature information generates diverse global semantic features for feature enhancement. The branch of feature generation based on local feature information and the branch of feature generation based on global semantic features learn from each other, capture complementary information, promote the transfer of implicit knowledge between each other, and enable the model to generate more discriminative features. .

Description of the drawings

A more complete understanding of exemplary embodiments of the invention may be obtained by reference to the following drawings:

Figure 1 is a schematic flowchart of a small-sample image recognition method based on dual-branch mutual learning feature generation provided by an exemplary embodiment of the present invention;

Figure 2 is a schematic structural diagram of a small sample image recognition device based on dual-branch mutual learning feature generation provided by an exemplary embodiment of the present invention;

Figure 3 is a structure of an electronic device provided by an exemplary embodiment of the present invention.

Detailed ways

Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments of the present invention. It should be understood that the present invention is not limited to the example embodiments described here.

It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values set forth in these examples do not limit the scope of the invention unless otherwise specifically stated.

Those skilled in the art can understand that terms such as "first" and "second" in the embodiments of the present invention are only used to distinguish different steps, devices or modules, etc., and do not represent any specific technical meaning, nor do they represent the differences between them. necessary logical sequence.

It should also be understood that in the embodiment of the present invention, "multiple" may refer to two or more than two, and "at least one" may refer to one, two, or more than two.

It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood to mean one or more unless there is an explicit limitation or contrary inspiration is given in the context.

In addition, the term "and/or" in the present invention is only an association relationship describing related objects, indicating that there can be three relationships, for example, A and/or B, which can mean: A alone exists, and A and B exist simultaneously. , there are three situations of B alone. In addition, the character "/" in the present invention generally indicates that the related objects are in an "or" relationship.

It should also be understood that the description of the various embodiments of the present invention focuses on the differences between the various embodiments, and the similarities or similarities between the embodiments can be referred to each other. For the sake of brevity, they will not be described again one by one.

At the same time, it should be understood that, for convenience of description, the dimensions of various parts shown in the drawings are not drawn according to actual proportional relationships.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application or uses.

Techniques, methods and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, the techniques, methods and devices should be considered a part of the specification.

It should be noted that similar reference numerals and letters refer to similar items in the following figures, so that once an item is defined in one figure, it does not require further discussion in subsequent figures.

Embodiments of the present invention may be applied to electronic devices such as terminal devices, computer systems, servers, etc., which may operate with numerous other general or special purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and/or configurations suitable for use with terminal devices, computer systems, servers and other electronic devices include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients Computers, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, small computer systems, mainframe computer systems and distributed cloud computing technology environments including any of the above systems, etc.

Electronic devices such as terminal devices, computer systems, servers, etc. may be described in the general context of computer system executable instructions (such as program modules) being executed by the computer system. Generally, program modules may include routines, programs, object programs, components, logic, data structures, etc., that perform specific tasks or implement specific abstract data types. The computer system/server may be implemented in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communications network. In a distributed cloud computing environment, program modules may be located on local or remote computing system storage media including storage devices.

Example methods

Figure 1 is a schematic flowchart of a small-sample image recognition method based on dual-branch mutual learning feature generation provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices. As shown in Figure 1, the small sample image recognition method 100 based on dual-branch mutual learning feature generation includes the following steps:

Step 101: Obtain a small sample image set to be identified to form a query set to be identified.

Step 102: Send each image in the query set to be recognized to the first feature generation module of the pre-built global branch to generate the first semantic feature of each image.

Step 103: Send each image in the query set to be recognized to the second feature generation module of the pre-built local branch to generate the second semantic feature of each image.

Step 104: Add the first semantic feature and the second semantic feature of each image in the query set to be recognized to determine the third semantic feature of each image in the query set to be recognized.

Step 105: Calculate the similarity between the third semantic feature of each image in the query set to be recognized and multiple category prototypes in the support set, and determine the image category of each image in the query set to be recognized.

Optionally, a category prototype for each category in the support set is constructed as follows:

All images in the support set are input into the first feature generation module of the global branch in turn, and the fourth semantic feature of each image in the support set is output;

All images in the support set are input into the second feature generation module of the local branch in turn, and the fifth semantic feature of each image in the support set is output;

The fourth semantic feature and the fifth semantic feature of all images of each category in the support set are added and averaged to determine the category prototype for each category in the support set.

Optionally, separately calculate the similarity between the third semantic feature of each image in the query set to be recognized and the category prototypes of multiple categories in the support set, and determine the image category of each image in the query set to be recognized, including:

Calculate the similarity between the third semantic feature of each image in the query set to be recognized and the category prototypes of multiple categories in the support set, and determine the probability value of each image in the query set to be recognized belonging to each category in the support set;

The category with the largest probability value corresponding to each image in the query set to be recognized is taken as the image category of the image.

Specifically, N classes are selected as the support set, and then S images are selected from each class to form the support set. , Recognize query set images based on support set images.

Computational support for centralized category prototypes. First, N×S images of the support set are sent to the global branch and the local branch respectively, and through the first feature generation module of the global branch, we get A fourth semantic feature, similarly, through the second feature generation module of the local branch, also obtain/> fifth semantic feature, and then generate the two branches The semantic features are added together to obtain the total semantic features, and finally the // in each category is The semantic features are added and averaged as the category prototype of each category/> Among them/> is the global feature of the image.

Identify the images in the query set. For the query set to be identified Each image x in is sent to the global branch and the local branch respectively. Through the first feature generation module of the global branch, M first semantic features are obtained. Similarly, through the second feature generation module of the local branch, M are also obtained. second semantic features, and then add and average the M semantic features generated by the two branches to obtain the third semantic feature of image x /> . Then, the probability value of the query set sample to be identified belonging to each category is calculated based on the similarity between each query set sample to be identified and the prototype of each category, and the category with the largest probability value is taken as the predicted label of the query set sample to be identified. The probability calculation formula is:

Among them, C _i is the category label of the query set sample to be identified in N categories, is the category label predicted based on the distance between the query set sample to be recognized and N category prototypes.

Optionally, the training process of the first feature generation module of the global branch and the second feature generation module of the local branch is as follows:

Based on the small sample image training set, a small sample recognition task is constructed, in which the small sample recognition task includes N categories, and each category includes M sample images;

Use the feature extraction network to extract features from each sample image in the small sample recognition task to determine the global features and local features of each sample image;

The first feature generation module of the global branch is trained through the global features of each sample image in the small sample recognition task;

The second feature generation module of the local branch is trained through the global features and local features of each sample image in the small sample recognition task;

Learn the training information of the global branch and the local branch from each other, and train the first feature generation module and the second feature generation module;

According to the preset training total loss function, the first feature generation module and the second feature generation module are optimized.

Optionally, train the first feature generation module of the global branch through the global features of each sample image in the small sample recognition task, including:

Mask the global features of M sample images of each category respectively to determine the global features of a sample image;

Replace the global features of the sample images of each category mask with learnable vectors;

Train the first feature generation module of the global branch based on the learnable vectors replaced by each category and the global features retained by the mask;

The first feature generation module is optimized according to the preset global branch loss function, where the global branch loss function includes a global prediction loss function and a global classification loss function.

Optionally, train the second feature generation module of the local branch through the global features and local features of each sample image in the small sample recognition task, including:

Select local features of a sample image in each category;

Train the second feature generation module based on the local features selected for each category and the preset M learnable vectors;

The second feature generation module is optimized according to the preset local branch loss function, where the local branch loss function includes a local prediction loss function and a local classification loss function.

Optionally, it also includes: calculating KL divergence as a mutual learning loss function in which the training information of the global branch and the local branch learn from each other.

Optionally, the total training loss function is the sum of the global branch loss function, the local branch loss function and the mutual learning loss function.

Specifically, the training steps of the first feature generation module of the global branch and the second feature generation module of the local branch are as follows:

Step 1: Construct a small sample recognition task. Construct a series of small sample recognition tasks for training. Specifically, N classes are randomly selected from the training set, and then M samples are randomly sampled from each class to form the task T , with a total of N×M images.

Step 2: Feature extraction. For each image x in task T , the feature extraction network is input to extract features. The feature extraction network uses the visual self-attention model ViT (Vision Transformer), including four basic encoder blocks Transformer Blocks, shared by local branches and global branches. Parameters for the first three Transformer Blocks. For the local branch, extract the global features of the image and local features/> . For the global branch, only the global features of the image are extracted. Where H , W and C are the height, width and number of channels of the feature map respectively.

Step 3: Global branch. Based on step 2, the global features of all images in task T are obtained. For the global features of M images of each class, they are masked, and only the global features of one image are retained. Then the masked global features are used as learnable vector to replace.

is the global feature vector of M images in category i , mask represents the masking operation, A learnable vector representing feature replacement for the mask.

The masked feature vector is then sent to the first feature generation module of the global branch. The first feature generation module consists of a series of Transformer Blocks. The feature generation module uses the retained global feature information to predict the features of the mask, and learns the intra-class changes of samples in a one-to-many manner, so that the model can generate diversified features.

Generate module for the first feature, /> Features generated by the first feature generation module.

Use mean square error (MSE) as the global prediction loss function to measure the difference between the predicted features and the features before masking.

In order to make the features extracted by the global branch discriminative, the global branch is trained under the supervision of the real labels of the input images and learns inter-class relationship mining in the entire class space. The global classification loss is:

where yi is the category label _of xi , h _represents the classifier, which is a fully connected layer.

Finally, the loss of the global branch is:

in is a hyperparameter.

Step 4: Local branching. Based on step 2, the global features and local features of all images in task T are determined. For the local features of M images in each category, only the global features of one image are selected, and M learnable vectors are sent to the local branch. Feature generation module, M learnable vectors are used to generate predicted global features, and M global features extracted by the feature extraction module are used for supervision.

represents M learnable vectors used to generate predicted global features, Represents W×H local features.

Then, M learnable vectors and W×H local features are sent to the second feature generation module of the local branch. By mining the semantic relationship between local features and global features, local feature information is used to generate global semantic features.

Generate module for the second feature, /> M global features generated by the second feature generation module.

As with the global branch, the mean square error (MSE) is used as the local prediction loss function to measure the difference between the predicted features and the original features.

g _i,j represents the generated global feature, and f _i,j represents the j -th global feature of the i- th category.

where yi is the category label _of xi in the selected N classes, h represents the classifier, which is a fully connected layer, and k represents _the number of images in task T.

Finally, the total loss of the local branch is:

in is a hyperparameter.

Step 5: Learn from each other. In order to enable information interaction between the two branches, the two branches learn complementary information from the other branches, and the KL divergence is calculated as the mutual learning loss of the two branches:

in, is the image feature extracted by the global branch, and F _l is the image feature extracted by the local branch.

Step 6: Total loss. The global branch loss, local branch loss and mutual learning loss of the two branches are combined to form the total loss:

in, , /> and/> is a hyperparameter.

Therefore, existing feature enhancement-based methods mainly generate new global semantic features by analyzing the similarities or differences of global semantic features between different samples. Although this method can increase the size and diversity of the data set, it also has a problem, that is, it ignores the local characteristic information of the sample. In the case of small samples, due to the limited amount of data, each sample may contain some unique or Important local feature information, which is very useful for distinguishing different categories or tasks. If only global semantic features are used to generate new features, these local feature information may be lost or confused, resulting in low-quality or inaccurate generated new features. In this regard, the present invention constructs a feature generation module based on local feature information. This module mines the semantic relationship between local features and global features of the sample, and uses the local feature information of the sample to generate diversified global semantic features for feature enhancement. In addition, to address the limitations of using only global features or local features for feature enhancement, the present invention proposes a dual-branch mutual learning feature generation method, including a branch for feature generation based on local feature information and a branch for feature generation based on global semantic features. The two branches learn from each other, capture complementary information, promote the transfer of tacit knowledge between each other, and enable the model to generate more discriminative features.

Exemplary device

Figure 2 is a schematic structural diagram of a small sample image recognition device based on dual-branch mutual learning feature generation provided by an exemplary embodiment of the present invention. As shown in Figure 2, device 200 includes:

The acquisition module 210 is used to acquire a small sample image set to be identified to form a query set to be identified;

The first generation module 220 is used to send each image in the query set to be recognized to the first feature generation module of the pre-built global branch to generate the first semantic feature of each image;

The second generation module 230 is used to send each image in the query set to be recognized to the second feature generation module of the pre-constructed local branch to generate the second semantic feature of each image;

The first determination module 240 is used to add the first semantic feature and the second semantic feature of each image in the query set to be recognized, and determine the third semantic feature of each image in the query set to be recognized;

The second determination module 250 is used to respectively calculate the similarity between the third semantic feature of each image in the query set to be recognized and multiple category prototypes in the support set, and determine the image category of each image in the query set to be recognized.

Optionally, the construction process of the category prototype for each category of the support set in the second determination module 250 is as follows:

The first output sub-module is used to sequentially input all images in the support set into the first feature generation module of the global branch, and output the fourth semantic feature of each image in the support set;

The second output submodule is used to input all the images in the support set into the second feature generation module of the local branch in sequence, and output the fifth semantic feature of each image in the support set;

The determination submodule is used to add and average the fourth semantic features and fifth semantic features of all images of each category in the support set to determine the category prototype of each category in the support set.

Optionally, the training process of the first feature generation module of the global branch and the second feature generation module of the local branch in the first generation module 220 and the second generation module 230 is as follows:

Construct a submodule for constructing a small sample recognition task based on the small sample image training set, where the small sample recognition task includes N categories, and each category includes M sample images;

The extraction submodule is used to use the feature extraction network to extract features from each sample image in the small sample recognition task, and determine the global features and local features of each sample image;

The first training submodule is used to train the first feature generation module of the global branch through the global features of each sample image in the small sample recognition task;

The second training submodule is used to train the second feature generation module of the local branch through the global features and local features of each sample image in the small sample recognition task;

The third training sub-module is used to learn the training information of the global branch and the local branch from each other, and train the first feature generation module and the second feature generation module;

The optimization sub-module is used to optimize the first feature generation module and the second feature generation module according to the preset training total loss function.

Optionally, the first training sub-module includes:

The masking unit is used to mask the global features of M sample images of each category to determine the global features of a sample image;

The replacement unit is used to replace the global features of the sample images of each category mask with learnable vectors;

A first training unit configured to train the first feature generation module of the global branch based on the learnable vectors replaced by each category and the global features retained by the mask;

The first optimization unit is used to optimize the first feature generation module according to a preset global branch loss function, where the global branch loss function includes a global prediction loss function and a global classification loss function.

Optionally, the second training sub-module includes:

The selection unit is used to select local features of a sample image in each category;

The second training unit is used to train the second feature generation module based on the local features selected for each category and the preset M learnable vectors;

The second optimization unit is used to optimize the second feature generation module according to a preset local branch loss function, where the local branch loss function includes a local prediction loss function and a local classification loss function.

Optionally, the apparatus 200 further includes: as a module, a mutual learning loss function for calculating the KL divergence as the training information of the global branch and the local branch learns from each other.

Optionally, the total training loss function is the sum of the global branch loss function, the local branch loss function and the mutual learning loss function.

Optionally, the second determination module 250 includes:

The calculation submodule is used to separately calculate the similarity between the third semantic feature of each image in the query set to be recognized and the category prototypes of multiple categories in the support set, and determine the probability value of each image in the query set to be recognized belonging to each category in the support set. ;

As a sub-module, it is used to take the category with the largest probability value corresponding to each image in the query set to be recognized as the image category of the image.

Example electronic device

Figure 3 is a structure of an electronic device provided by an exemplary embodiment of the present invention. As shown in FIG. 3 , electronic device 30 includes one or more processors 31 and memory 32 .

The processor 31 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

Memory 32 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and/or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 31 may execute the program instructions to implement the methods and/or software programs of various embodiments of the present invention described above. Other desired features. In one example, the electronic device may further include an input device 33 and an output device 34, and these components are interconnected through a bus system and/or other forms of connection mechanisms (not shown).

In addition, the input device 33 may also include, for example, a keyboard, a mouse, and the like.

The output device 34 can output various information to the outside. The output device 34 may include, for example, a display, a speaker, a printer, a communication network and remote output devices connected thereto, and the like.

Of course, for simplicity, only some of the components related to the present invention in the electronic device are shown in FIG. 3 , and components such as buses, input/output interfaces, etc. are omitted. In addition to this, the electronic device may include any other suitable components depending on the specific application.

Example computer program products and computer-readable storage media

In addition to the above methods and devices, embodiments of the present invention may also be a computer program product, which includes computer program instructions that, when executed by a processor, cause the processor to execute the “exemplary method” described above in this specification The steps in methods according to various embodiments of the invention are described in Sec.

The computer program product may be written in any combination of one or more programming languages, including object-oriented programming languages, such as Java, C++, etc., to write program codes for performing operations of embodiments of the present invention. , also includes conventional procedural programming languages, such as the "C" language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server execute on.

In addition, embodiments of the present invention may also be a computer-readable storage medium having computer program instructions stored thereon. The computer program instructions, when executed by a processor, cause the processor to execute the above-mentioned “exemplary method” part of this specification. The steps in methods according to various embodiments of the invention are described in .

The computer-readable storage medium may be any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. Readable storage media may include, for example, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: electrical connection with one or more conductors, portable disk, hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

The basic principles of the present invention have been described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, advantages, effects, etc. mentioned in the present invention are only examples and not limitations. These advantages, advantages, effects, etc. cannot be considered to be Each embodiment of the present invention must have. In addition, the specific details disclosed above are only for the purpose of illustration and to facilitate understanding, and are not limiting. The above details do not limit the present invention to the fact that the invention must be implemented using the above specific details.

Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on its differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple. For relevant details, please refer to the partial description of the method embodiment.

The block diagrams of devices, systems, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, systems, devices, systems may be connected, arranged, and configured in any manner. Words such as "includes," "includes," "having," etc. are open-ended terms that mean "including, but not limited to," and may be used interchangeably therewith. As used herein, the words "or" and "and" refer to the words "and/or" and are used interchangeably therewith unless the context clearly dictates otherwise. As used herein, the word "such as" refers to the phrase "such as, but not limited to," and may be used interchangeably therewith.

The methods and systems of the present invention may be implemented in many ways. For example, the method and system of the present invention can be implemented through software, hardware, firmware, or any combination of software, hardware, and firmware. The above order for the steps of the method is for illustration only, and the steps of the method of the present invention are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, the present invention can also be implemented as programs recorded in recording media, and these programs include machine-readable instructions for implementing the methods according to the present invention. Thus, the present invention also covers recording media storing a program for executing the method according to the present invention.

It should also be noted that in the system, device and method of the present invention, each component or each step can be decomposed and/or recombined. These decompositions and/or recombinations should be regarded as equivalent solutions of the present invention. The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the invention. Thus, the present invention is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for the purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the invention to the form disclosed herein. Although various example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims (8)

1. A small-sample image recognition method based on dual-branch mutual learning feature generation, which is characterized by: Obtain a small sample image set to be identified to form a query set to be identified; Send each image in the query set to be recognized to the first feature generation module of the pre-built global branch to generate the first semantic feature of each image; Send each image in the query set to be recognized to the second feature generation module of the pre-constructed local branch to generate the second semantic feature of each image; Add the first semantic feature and the second semantic feature of each image in the query set to be recognized to determine the third semantic feature of each image in the query set to be recognized; Calculate the similarity between the third semantic feature of each image in the query set to be recognized and multiple category prototypes in the support set respectively, and determine the image category of each image in the query set to be recognized; The construction process of category prototypes for each category of the support set is as follows: Input all images in the support set into the first feature generation module of the global branch in sequence, and output the fourth semantic feature of each image in the support set; Input all images in the support set into the second feature generation module of the local branch in sequence, and output the fifth semantic feature of each image in the support set; Add and average the fourth semantic features and fifth semantic features of all images of each category in the support set to determine the category prototype of each category in the support set; The training process of the first feature generation module of the global branch and the second feature generation module of the local branch is as follows: Construct a small sample recognition task based on the small sample image training set, where the small sample recognition task includes N categories, and each category includes M sample images; Using a feature extraction network to extract features from each sample image in the small sample recognition task, and determine the global features and local features of each sample image; Train the first feature generation module of the global branch through the global features of each sample image in the small sample recognition task; Train the second feature generation module of the local branch through the global features and the local features of each sample image in the small sample recognition task; Learn the training information of the global branch and the local branch from each other, and train the first feature generation module and the second feature generation module; The first feature generation module and the second feature generation module are optimized according to the preset training total loss function. 2. The method according to claim 1, characterized in that training the first feature generation module of the global branch through the global features of each sample image in the small sample recognition task includes: Mask the global features of M sample images of each category respectively to obtain the global features of one sample image; Replace the global features of the sample images of each category mask with learnable vectors; Train the first feature generation module of the global branch based on the learnable vectors replaced by each category and the global features retained by the mask; The first feature generation module is optimized according to a preset global branch loss function, where the loss function of the global branch includes a global prediction loss function and a global classification loss function. 3. The method according to claim 2, characterized in that the second feature generation module of the local branch is trained through the global features and the local features of each sample image in the small sample recognition task. ,include: Select local features of a sample image in each category; Train the second feature generation module according to the local features selected for each category and the preset M learnable vectors; The second feature generation module is optimized according to a preset local branch loss function, where the local branch loss function includes a local prediction loss function and a local classification loss function. 4. The method according to claim 3, further comprising: calculating KL divergence as a mutual learning loss function for the training information of the global branch and the local branch to learn from each other. 5. The method according to claim 4, wherein the total training loss function is the sum of the global branch loss function, the local branch loss function and the mutual learning loss function. 6. The method according to claim 1, characterized by calculating the similarity between the third semantic feature of each image in the query set to be recognized and the category prototypes of multiple categories in the support set, and determining the Identify the image category for each image in the query set, including: Calculate the similarity between the third semantic feature of each image in the query set to be identified and the category prototypes of multiple categories in the support set, and determine that each image in the query set to be identified belongs to each image in the query set to be identified. Probability value of category; The category with the largest probability value corresponding to each image in the query set to be recognized is taken as the image category of the image. 7. A small sample image recognition device based on dual-branch mutual learning feature generation, which is characterized by including: The acquisition module is used to obtain a small sample image set to be identified to form a query set to be identified; A first generation module, configured to send each image in the query set to be recognized to a first feature generation module of a pre-constructed global branch to generate the first semantic feature of each image; A second generation module, configured to send each image in the query set to be recognized to a second feature generation module of a pre-constructed local branch to generate a second semantic feature of each image; A first determination module, configured to add the first semantic feature and the second semantic feature of each image in the query set to be recognized, and determine the third semantic feature of each image in the query set to be recognized; a second determination module, configured to separately calculate the similarity between the third semantic feature of each image in the query set to be recognized and multiple category prototypes in the support set, and determine the image category of each image in the query set to be recognized; The construction process of the category prototype for each category of the support set in the second determination module 250 is as follows: The first output sub-module is used to sequentially input all images in the support set into the first feature generation module of the global branch, and output the fourth semantic feature of each image in the support set; The second output submodule is used to input all the images in the support set into the second feature generation module of the local branch in sequence, and output the fifth semantic feature of each image in the support set; The determination submodule is used to add and average the fourth semantic features and fifth semantic features of all images of each category in the support set to determine the category prototype of each category in the support set; The training process of the first feature generation module of the global branch and the second feature generation module of the local branch in the first generation module 220 and the second generation module 230 is as follows: Construct a submodule for constructing a small sample recognition task based on the small sample image training set, where the small sample recognition task includes N categories, and each category includes M sample images; The extraction submodule is used to use the feature extraction network to extract features from each sample image in the small sample recognition task, and determine the global features and local features of each sample image; The first training submodule is used to train the first feature generation module of the global branch through the global features of each sample image in the small sample recognition task; The second training submodule is used to train the second feature generation module of the local branch through the global features and local features of each sample image in the small sample recognition task; The third training sub-module is used to learn the training information of the global branch and the local branch from each other, and train the first feature generation module and the second feature generation module; The optimization sub-module is used to optimize the first feature generation module and the second feature generation module according to the preset training total loss function. 8. A computer-readable storage medium, characterized in that the storage medium stores a computer program, and the computer program is used to execute the method described in any one of claims 1-6.