CN114648650A - Method and device, device and storage medium for neural network training and target detection - Google Patents

Abstract

The present disclosure provides a method, device, device and storage medium for neural network training and target detection, wherein the method includes: acquiring a first image sample collected in an upstream task, a first neural network to be trained in a downstream task, And the second neural network and the image generation network obtained based on the training of the first image sample; the second neural network is used for feature extraction, and the image generation network is used to generate a new image, and the new image conforms to the overall distribution of the first image sample; respectively; Perform feature extraction on the new image generated based on the image generation network according to the second neural network obtained by training and the first neural network to be trained; train the first neural network to be trained based on the extracted first image features and the second image features , get the trained first neural network. The present disclosure can better guide the training of the first neural network through the two image features, thereby improving the performance in downstream tasks.

Description

Method and device, device and storage medium for neural network training and target detection

technical field

The present disclosure relates to the technical field of artificial intelligence, and in particular, to a method and apparatus, device and storage medium for neural network training and target detection.

Background technique

With the rapid development of artificial intelligence technology, end-to-end deep learning technology is also becoming more and more mature. Using large-scale data sets, a pre-trained neural network (ie, a pre-training model) can be jointly learned upstream for various tasks. The model can directly share the pre-trained early weights and has relatively powerful feature representation capabilities.

However, in the process of migrating a pre-trained model to a specific downstream task, the amount of data that can actually be obtained downstream is relatively small, which leads to whether the model is transferred directly or after fine-tuning the pre-trained model. For migration, the performance of pre-trained models in downstream tasks is poor.

SUMMARY OF THE INVENTION

The embodiments of the present disclosure provide at least a method, apparatus, device, and storage medium for neural network training and target detection.

In a first aspect, an embodiment of the present disclosure provides a method for training a neural network, the method comprising:

Obtain the first image sample collected in the upstream task, the first neural network to be trained in the downstream task, and the second neural network and image generation network obtained by training based on the first image sample; the second neural network is used for performing feature extraction, the image generation network is used to generate a new image, and the new image conforms to the overall distribution of the first image samples;

Perform feature extraction on the new image generated based on the image generation network according to the second neural network obtained by training and the first neural network to be trained, to obtain the first image feature and the second image feature;

The first neural network to be trained is trained based on the first image feature and the second image feature to obtain a trained first neural network.

Using the above neural network training method, the new image generated based on the image generation network can be characterized according to the first neural network to be trained in the downstream task and the second neural network trained based on the first image sample collected in the upstream task. extraction, and then the first neural network can be trained according to the obtained first image features and the second image features. Since the new images generated based on the image generation network are image samples that are more in line with the overall distribution of the first image samples, such image samples are more conducive to adapting to the network environment of the second neural network. The first image feature output by the second neural network and the second image feature output by the first neural network to be trained can better guide the training of the first neural network, thereby further improving the performance in downstream tasks.

In a possible implementation, the image generation network is trained according to the following steps:

obtaining a first image output based on a codebook generation network; the codebook generation network is used to generate a codebook that decomposes the first image sample into a plurality of primitives;

Inputting the first image to the image generation network to be trained to obtain a second image output by the image generation network;

determining a loss function value of the image generation network to be trained based on the image similarity between the second image and the first image;

The image generation network to be trained is trained based on the loss function value to obtain a trained image generation network.

Here, the image generation network to be trained can be trained by the first image output by the codebook encoding method, so that the trained image generation network is more in line with the overall distribution of the first image samples, which is beneficial to improve the subsequent network training performance .

In a possible implementation manner, the inputting the first image to the image generation network to be trained includes:

performing masking processing on part of the image area in the first image to obtain a masked first image;

The masked first image is input to the image generation network to be trained.

In a possible implementation manner, the codebook generation network includes an encoder and a decoder, and the codebook generation network is trained according to the following steps:

The following steps are repeated until the similarity between the image output by the decoder and the first image sample input into the encoder is greater than a preset threshold:

Input the first image sample into the encoder to be trained to obtain the codebook output by the encoder; input the codebook output from the encoder to the decoder to be trained to obtain the image output from the decoder .

The codebook here can be image coding based on an adversarial network composed of an encoder and a decoder, with high accuracy.

In a possible implementation manner, the first image output based on the codebook generation network is obtained according to the following steps:

inputting the first image sample into an encoder included in the codebook generation network to obtain a codebook output by the encoder;

The codebook output by the encoder is input to the decoder included in the codebook generation network to obtain the first image output by the decoder.

Here, the codebook output by the encoder can be used to re-characterize the first image sample, and the characterized first image can be more suitable for subsequent network training.

In a possible implementation, the image generation network includes a first generation sub-network for generating a codebook that decomposes the first image sample into a plurality of primitives, and a first generation sub-network based on the first generation sub-network The output image generates the second generation sub-network of the new image; train the image generation network according to the following steps:

Inputting the first image sample into the trained first generation sub-network to obtain the first image output by the first generation sub-network;

Inputting the first image to the second generation sub-network to be trained to obtain a second image output by the second generation sub-network;

Determine the to-be-trained based on the first image similarity between the first image and the input first image sample, and the second image similarity between the second image and the first image The loss function value of the image generation network;

The image generation network to be trained is trained based on the loss function value to obtain a trained image generation network.

Here, the image generation network can be trained in combination with the first generation sub-network and the second generation sub-network, so that the trained image generation network can better take into account the generation effect and generation efficiency of codebook generation and image generation.

In a possible implementation manner, the first neural network to be trained is trained based on the first image feature and the second image feature to obtain a trained first neural network, including:

determining a loss function value of the first neural network to be trained based on the image similarity between the first image feature and the second image feature;

When the loss function value corresponding to the current round is greater than a preset threshold, adjust the network parameter value of the first neural network based on the loss function value, and perform the next step according to the adjusted first neural network. rounds of training until the loss function value is less than or equal to the preset threshold.

In a possible implementation manner, after the trained first neural network is obtained, the method further includes:

obtain the third image sample collected in the downstream task;

Perform network training again on the trained first neural network based on the third image sample to obtain a final trained first neural network.

Here, the first neural network can be fine-tuned based on the third image sample collected in the downstream task to expand the generalization performance of the network in the downstream task.

In a possible implementation manner, performing network training on the trained first neural network based on the third image sample again to obtain a final trained first neural network, including:

Inputting the third image sample into the first neural network to obtain the task output result of the network;

determining the loss function value of the first neural network based on the comparison between the task output result and the task labeling result labelled for the third image sample;

Perform network training on the first neural network again based on the loss function value to obtain a final trained first neural network.

In a possible implementation, the second neural network is trained according to the following steps:

Obtain an original neural network; the original neural network includes at least a feature extraction layer;

Perform feature extraction on the first image sample based on the feature extraction layer included in the original neural network to obtain image feature information output by the feature extraction layer;

Adjust the network parameter values of the feature extraction layer based on the image feature information to obtain an adjusted feature extraction layer;

The original neural network including the adjusted feature extraction layer is determined as the second neural network obtained by training.

Here, a second neural network can be obtained based on the training of the original neural network including the feature extraction layer, and the network can output relatively general feature information to facilitate subsequent task migration.

In a second aspect, an embodiment of the present disclosure further provides a method for target detection, the method comprising:

Obtain target images collected in downstream tasks;

The target image is input into the first neural network trained by using the neural network training method according to any one of the first aspect and its various embodiments to obtain the detection result of the target object in the target image.

In a third aspect, an embodiment of the present disclosure further provides an apparatus for training a neural network, the apparatus comprising:

an acquisition module, configured to acquire a first image sample collected in an upstream task, a first neural network to be trained in a downstream task, and a second neural network and an image generation network trained based on the first image sample; the Two neural networks are used for feature extraction, and the image generation network is used to generate a new image, and the new image conforms to the overall distribution of the first image samples;

an extraction module, configured to perform feature extraction on a new image generated based on the image generation network according to the second neural network obtained by training and the first neural network to be trained, respectively, to obtain the first image feature and the second image feature;

A training module, configured to train the first neural network to be trained based on the first image feature and the second image feature to obtain a trained first neural network.

In a fourth aspect, an embodiment of the present disclosure further provides an apparatus for target detection, the apparatus comprising:

The acquisition module is used to acquire the target image collected in the downstream task;

The detection module is used to input the target image into the first neural network trained by the neural network training method according to any one of the first aspect and its various embodiments, to obtain the target object in the target image. Test results.

In a fifth aspect, embodiments of the present disclosure further provide an electronic device, including: a processor, a memory, and a bus, where the memory stores machine-readable instructions executable by the processor, and when the electronic device runs, the Communication between the processor and the memory is through a bus, and when the machine-readable instructions are executed by the processor, the steps of the method for training a neural network according to any one of the first aspect and its various embodiments or as The steps of the method for target detection described in the second aspect.

In a sixth aspect, embodiments of the present disclosure further provide a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and the computer program is executed by a processor when the first aspect and various implementations thereof are executed. Any of the steps of the neural network training method or the method of target detection according to the second aspect.

For the description of the effects of the foregoing apparatus, electronic device, and computer-readable storage medium, reference may be made to the description of the foregoing method, and details are not repeated here.

In order to make the above-mentioned objects, features and advantages of the present disclosure more obvious and easy to understand, the preferred embodiments are exemplified below, and are described in detail as follows in conjunction with the accompanying drawings.

Description of drawings

In order to explain the technical solutions of the embodiments of the present disclosure more clearly, the following briefly introduces the accompanying drawings required in the embodiments, which are incorporated into the specification and constitute a part of the specification. The drawings illustrate embodiments consistent with the present disclosure, and together with the description serve to explain the technical solutions of the present disclosure. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as limiting the scope. Other related figures are obtained from these figures.

1 shows a flowchart of a method for training a neural network provided by an embodiment of the present disclosure;

FIG. 2 shows a flowchart of a method for target detection provided by an embodiment of the present disclosure;

FIG. 3 shows a schematic diagram of an apparatus for training a neural network provided by an embodiment of the present disclosure;

FIG. 4 shows a schematic diagram of an apparatus for target detection provided by an embodiment of the present disclosure;

FIG. 5 shows a schematic diagram of an electronic device provided by an embodiment of the present disclosure.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only These are some, but not all, embodiments of the present disclosure. The components of the disclosed embodiments generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the disclosure provided in the accompanying drawings is not intended to limit the scope of the disclosure as claimed, but is merely representative of selected embodiments of the disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present disclosure.

It should be noted that like numerals and letters refer to like items in the following figures, so once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

The term "and/or" in this paper only describes an association relationship, which means that there can be three kinds of relationships, for example, A and/or B, which can mean: the existence of A alone, the existence of A and B at the same time, the existence of B alone. a situation. In addition, the term "at least one" herein refers to any combination of any one of the plurality or at least two of the plurality, for example, including at least one of A, B, and C, and may mean including from A, B, and C. Any one or more elements selected from the set of B and C.

Studies have found that in the process of migrating a pre-trained model to a specific downstream task, the performance of the downstream task is usually improved by model fine-tuning in related technologies.

The existing fine-tuning methods mainly fall into the following two categories: The first category can be to filter and map the features extracted by the backbone network. In specific applications, the above screening process can be implemented by adding an additional network layer after the backbone network, that is, the general features extracted by the backbone network can be filtered and mapped through the additional network layer, and the features required by downstream tasks can be retained and strengthened. Features, here, additional network layers can be convolutional layers, normalization layers, etc. The second category can be manipulating backbone network weight parameters. In specific applications, backpropagation is not directly used to migrate downstream tasks, but weight increments and offset values are predicted for downstream tasks within the specified weight parameter space to help the backbone network adapt to downstream tasks.

However, both of the above two types of methods have shortcomings: the first type of methods may cause overfitting of the feature map layer when the amount of data in the downstream task is small; the weight update range of the second type of methods is limited by the specified weight parameters Space, it is impossible to guarantee that the weights are optimized to the best state. It can be seen that they need to improve the performance of model migration.

In addition, related pre-training models often have specific network structures. In the actual downstream scenarios to be migrated, they may need to be migrated to different network structures, which puts forward higher requirements for model migration performance.

Based on the above research, the present disclosure provides a solution for network migration based on a trained teacher network instructing the student network to be trained, so as to improve the performance of the pre-training model in downstream tasks.

In order to facilitate the understanding of this embodiment, a method for training a neural network disclosed in an embodiment of the present disclosure is first introduced in detail. A device, the electronic device includes, for example, a terminal device or a server or other processing device, and the terminal device may be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, and the like. In some possible implementations, the method for training a neural network may be implemented by a processor invoking computer-readable instructions stored in a memory.

Referring to FIG. 1, which is a flowchart of a method for training a neural network provided by an embodiment of the present disclosure, the method includes steps S101-S103, wherein:

S101: Obtain a first image sample collected in an upstream task, a first neural network to be trained in a downstream task, and a second neural network and an image generation network trained based on the first image sample; the second neural network is used for feature Extracting, the image generation network is used to generate a new image, and the new image conforms to the overall distribution of the first image samples;

S102: Perform feature extraction on the new image generated based on the image generation network according to the second neural network obtained by training and the first neural network to be trained, to obtain the first image feature and the second image feature;

S103: Train the first neural network to be trained based on the first image feature and the second image feature to obtain a trained first neural network.

In order to facilitate the understanding of the neural network training method provided by the embodiments of the present disclosure, the following briefly describes the application scenario of the method. The neural network training method in the embodiments of the present disclosure can be mainly applied to network training in related downstream tasks under visual scene migration. The downstream tasks here may be related tasks based on the currently migrated scene, for example, may be natural The target detection task in the scene can also be the semantic segmentation task in the acquisition scene.

Among them, the number of training samples that can be collected in downstream tasks is relatively small. Corresponding to downstream tasks are upstream tasks, which can be related tasks with more training samples. Taking the target classification task as an example, there is currently a target classification neural network trained by a training database consisting of various target objects. However, for the specific downstream application scenario of automatic driving, the training data corresponding to this scenario is relatively low. Therefore, it is often necessary to use the pre-trained model obtained from the upstream to support the downstream training. For example, the pre-trained model can be fine-tuned and then migrated.

However, due to a series of problems in the fine-tuning scheme in the related art, the performance of the pre-trained model in downstream tasks is not good. At the same time, during the migration process, there is also the limitation that the same model structure must be used when using the upstream pre-trained model to migrate to the downstream data set, which makes it impossible to transfer the pre-trained model well to the existence of different model structures. downstream tasks.

It is in order to solve the above problem that the embodiments of the present disclosure provide a solution for network migration based on a trained teacher network instructing the student network to be trained, so as to improve the performance of the pre-trained model in downstream tasks. performance.

In the embodiment of the present disclosure, the pre-training model here may be a second neural network obtained by training in the upstream task using the first image samples collected in the upstream task. In addition, the image generation network here may be a correlation network that generates new images that conform to the overall distribution of the first image samples.

In a specific application, an upstream data set for upstream tasks and a downstream data set for downstream tasks can be prepared in advance. The upstream data set is used as a large-scale pre-training data set with a large number of first image samples, and the downstream data set is used as a to-be-migrated data set. to the dataset, with a small number of second image samples.

The first image sample may be images collected in multiple tasks under multiple application scenarios, where the application scenarios may be natural scenes, monitoring scenarios, acquisition scenarios, and other scenarios, and the tasks here may be image classification, target detection, etc. , semantic segmentation and other tasks. The second image sample may be a specific scene to be migrated, an image collected in a specific task, such as an image of a pedestrian on a street in a detection task.

The original neural network including the feature extraction layer can be trained based on the above-mentioned first image sample. Here, feature extraction can be performed on the first image sample based on the feature extraction layer, and then the feature extraction layer can be extracted by the image feature information output by the feature extraction layer. The value of the network parameter is adjusted, so that the original neural network obtained by training can be determined as the above-trained second neural network.

Wherein, the above-mentioned original neural network is any network structure with feature providing function, and the second neural network obtained by using large-scale upstream data (corresponding to the first image sample) to train the original neural network, for any image, Its backbone network part (corresponding to the feature extraction layer) can output a general feature representation.

It should be noted that the above-mentioned original neural network can also include a task layer for task processing after the feature extraction layer. In this case, the difference between the task output results of the task layer and the task annotation results for large-scale upstream data can be used. The matching degree is used to train the entire original neural network, which will not be repeated here.

In order to better train the second neural network in the downstream task, here, the trained second neural network can be regarded as the teacher model in the knowledge distillation, and the downstream first neural network to be migrated is regarded as the knowledge distillation. The student model trains the first neural network in a manner of fixing the teacher model and training the student model.

During the training process, the second neural network and the first neural network can be used for feature extraction respectively, and then based on the first image features output by the teacher model of the second neural network and the output of the student model by the first neural network The similarity between the second image features to guide the training of the first neural network, so that the representation of the output of the student model is as close as possible to the teacher model, so that the downstream tasks can be better adapted. , even if the downstream task and the upstream task do not use the same network structure, the task metrics can be done well.

Considering that in practical applications, the first image samples are derived from images in a large-scale pre-training dataset, which makes it possible that different first image samples may be collected based on different application scenarios. There may be some differences in the characteristics of the first image samples collected in different application scenarios, which may interfere with network training to a certain extent. Here, in order to fully mine the feature information contained in the upstream data set, To reduce the interference of irrelevant information, the image generation network can be used to generate new images that conform to the overall distribution of the first image samples, and then the above-mentioned teacher-student models can be trained based on the generated new images, so that the second neural network can be efficiently and accurately generated. The pre-trained model to which the network belongs is migrated to a specific downstream task domain, which can have a good migration effect even in the case of a small amount of downstream data.

Considering the critical role of the training process of the image generation network for model transfer, the following will focus on the scheme for training the image generation network. The image generation network in the embodiment of the present disclosure may be obtained by training on the premise of the codebook generation network, or may be obtained by training synchronously with the codebook generation network. Specifically, it can be expanded in the following two aspects.

The first aspect: the embodiment of the present disclosure can train the image generation network according to the following steps:

Step 1: Obtain the first image output based on the codebook generation network; the codebook generation network is used to generate a codebook that decomposes the first image sample into a plurality of primitives;

Step 2, inputting the first image to the image generation network to be trained to obtain the second image output by the image generation network;

Step 3: Determine the loss function value of the image generation network to be trained based on the image similarity between the second image and the first image;

Step 4: Train the image generation network to be trained based on the loss function value to obtain a trained image generation network.

Here, the first image output by the codebook generation network can be used as the input image of the image generation network to be trained, and then the image generation network is determined based on the image similarity between the second image output by the image generation network and the first image. loss function value. The larger the loss function value, the larger the gap between the output second image and the input first image to a certain extent. At this time, network training needs to be performed again. The smaller the loss function value, the higher the output second image. The gap between the image and the input first image is relatively small. When the gap is small to a certain extent, it can be determined that the output image is basically the same as the input image, and the training can be ended at this time.

In order to better train the image generation network, before the first image is input to the image generation network to be trained, part of the image area in the first image can be masked to obtain the masked first image. When the processed first image is input to the image generation network to be trained, the uncovered part of the image area can guide the generation of the covered part of the image area, which can be realized based on the proximity between the generated image and the original first image. network training.

In the embodiment of the present disclosure, the above-mentioned codebook generation network may also be obtained by training based on the first image sample. The codebook generation network here is mainly to train a codebook that can encode the visual features in the upstream data, and then image restoration can be performed through the multiple primitives contained in the codebook generated in the codebook generation network, and then the codebook generation network can be obtained. The first image of the output.

Next, the training process and application process of the codebook generation network can be introduced in detail.

In the embodiments of the present disclosure, an adversarial network formed by paired encoders and decoders can be used to train the codebook generation network. Here, the first image sample can be input into the encoder to be trained, and the codebook output by the encoder can be obtained; the codebook output by the encoder can be input into the decoder to be trained to obtain the image output by the decoder, and then the decoder can be verified. Whether the similarity between the output image and the first image sample input to the encoder is greater than the preset threshold, if not greater than the preset threshold, then loop the above process of inputting the first image sample to the encoder to be trained until two The similarity of the images is greater than the preset threshold.

Here, by using the trained codebook generation network, an image can be decomposed into a codebook consisting of several primitives through the encoder, and these primitives can be restored to the image through the decoder.

Here, the first image sample can be input into the encoder included in the codebook generation network to obtain the codebook output by the encoder. In the case of inputting the codebook output by the encoder into the decoder included in the codebook generation network, the Image restoration is performed using each primitive included in the codebook, and the re-characterized first image is obtained.

It should be noted that, in practical applications, the determination process of the first image may be determined by the training process of the joint codebook generation network, that is, the input of the first image sample into the encoder to be trained may be performed repeatedly, The steps of obtaining the codebook output by the encoder, and inputting the codebook output by the encoder into the decoder to be trained, and obtaining the image output by the decoder, until the image output by the decoder is the same as the first image sample input into the encoder When the similarity between them is greater than the preset threshold, the image output by the decoder is determined as the first image.

Second aspect: the image generation network includes a first generation sub-network for generating a codebook that decomposes the first image sample into a plurality of primitives, and a second generation sub-network for generating a new image based on an image output by the first generation sub-network In the case of generating a sub-network, the embodiment of the present disclosure can train the image generating network according to the following steps:

Step 1: Input the first image sample into the trained first generation sub-network to obtain the first image output by the first generation sub-network;

Step 2, inputting the first image to the second generation sub-network to be trained to obtain the second image output by the second generation sub-network;

Step 3: Determine the loss function of the image generation network to be trained based on the first image similarity between the first image and the input first image sample, and the second image similarity between the second image and the first image value;

Step 4: Train the image generation network to be trained based on the loss function value to obtain a trained image generation network.

Here, the first image similarity between the first image and the input first image sample and the second image similarity between the second image and the first image can be combined to determine the loss function value of the image generation network to be trained , whether it is the similarity of the first image or the similarity of the second image will affect the adjustment of the relevant network parameter values, that is, the synchronous training of the first generation sub-network and the second generation sub-network is realized here, and the training efficiency is higher .

In the case that the training includes the first generation sub-network and the second generation sub-network, a corresponding new image can be generated for any input first image sample, and the new image contains rich data information of upstream tasks, Thus, it is more suitable for the training needs of the network.

In the case of inputting a large number of new images generated by the image generation network into the trained second neural network (ie, the pre-training model) in the upstream task, a general representation can be obtained, and the general representation can be used for knowledge distillation. knowledge is distilled into the first neural network to be migrated downstream.

In the embodiment of the present disclosure, the image similarity between two image features (ie, the first image feature and the second image feature) extracted by the trained second neural network and the to-be-trained first neural network may be used to determine Instruct the training of the first neural network in the downstream task, which can be achieved by the following steps:

Step 1: Determine the loss function value of the first neural network to be trained based on the image similarity between the first image feature and the second image feature;

Step 2: When the loss function value corresponding to the current round is greater than the preset threshold, adjust the network parameter value of the first neural network based on the loss function value, and perform the next round of training according to the adjusted first neural network, until the loss function value is less than or equal to the preset threshold.

Here, the image similarity between the two image features is negatively correlated with the loss function value of the first neural network, that is, in the case that the image similarity is relatively small, the determined loss function value is larger. When the similarity is relatively large, the determined loss function value is relatively small. The purpose of training the first neural network in the embodiment of the present disclosure is to make the representations output by the two neural networks (the second neural network and the first neural network) as similar as possible.

In order to further expand the generalization performance of the first neural network in the downstream task field, here, the second image sample collected in the downstream task can be used to fine-tune the first neural network, which can be achieved by the following steps:

Step 1. Input the second image sample into the first neural network to obtain the task output result of the network;

Step 2: Determine the loss function value of the first neural network based on the comparison between the task output result and the task labeling result labelled for the second image sample;

Step 3: Perform network training on the first neural network again based on the loss function value to obtain a final trained first neural network.

Here, feature extraction can be performed through the feature extraction layer included in the first neural network. In the case where the feature information output by the feature extraction layer is input into the task layer included in the first neural network, it can be based on the task output result and for the second image. The matching results of the task labeling results of the samples are subjected to multiple rounds of training of the first neural network.

In the embodiment of the present disclosure, in the case where the task output result and the task labeling result do not match, it indicates that the current network performance is not good, and it is necessary to adjust the network parameter values for the next round of training, until the two results match or until the Other network convergence conditions are met, for example, the number of iterations reaches a preset number, and another example, the loss function value is less than a preset threshold, etc.

For different downstream tasks, the task annotation results here are also different. For example, some image samples may be information such as the position and size of the target object marked for the target detection task, and some image samples may be the object semantic information marked for the target semantic segmentation task. Different downstream tasks can be marked here, and there are no specific restrictions on this.

In the process of fine-tuning the network based on the second image sample in the embodiment of the present disclosure, the overall adjustment process of each network layer included in the network can be performed. Here, all parameters of each network layer can be released, and a smaller learning rate can be used to perform The final adjustment of the network can significantly improve the generalization performance of the network in the downstream task domain.

Based on the above-mentioned neural network training method provided by the embodiment of the present disclosure, an embodiment of the present disclosure also provides a method for target detection, as shown in FIG. 2 , which specifically includes the following steps:

S201: Obtain the target image collected in the downstream task;

S202: Input the target image into the first neural network trained by using the neural network training method to obtain the detection result of the target object in the target image.

Here, when the target image collected in the downstream task is acquired, the target object in the target image can be detected based on the trained first neural network for target detection, and the detection of the target object in the target image can be obtained. result.

The detection result of the target object in the target image may be information such as the position and size of the target object in the target image.

In the embodiment of the present disclosure, the target images collected by different downstream tasks are also different. For details, please refer to the collection process of the second image sample, which will not be repeated here. For the training process of the first neural network, refer to the relevant descriptions in the foregoing embodiments, and details are not repeated here.

It should be noted that the neural network training method provided by the embodiments of the present disclosure can be applied not only to the field of target detection, but also to the fields of image classification, semantic segmentation, and the like, which will not be repeated here.

Those skilled in the art can understand that in the above method of the specific implementation, the writing order of each step does not mean a strict execution order but constitutes any limitation on the implementation process, and the specific execution order of each step should be based on its function and possible Internal logic is determined.

Based on the same inventive concept, an apparatus corresponding to the method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem in the apparatus in the embodiment of the present disclosure is similar to the above-mentioned method in the embodiment of the present disclosure, the implementation of the apparatus may refer to the implementation of the method. The repetition will not be repeated.

Referring to FIG. 3 , which is a schematic diagram of an apparatus for training a neural network according to an embodiment of the present disclosure, the apparatus includes: an acquisition module 301 , an extraction module 302 , and a training module 303 ; wherein,

The acquisition module 301 is used to acquire the first image sample collected in the upstream task, the first neural network to be trained in the downstream task, and the second neural network and the image generation network obtained by training based on the first image sample; the second neural network For feature extraction, the image generation network is used to generate a new image, and the new image conforms to the overall distribution of the first image samples;

The extraction module 302 is used to perform feature extraction on the new image generated based on the image generation network according to the second neural network obtained by training and the first neural network to be trained, to obtain the first image feature and the second image feature;

The training module 303 is configured to train the first neural network to be trained based on the first image feature and the second image feature to obtain a trained first neural network.

Using the above-mentioned apparatus for neural network training, the new image generated based on the image generation network can be characterized according to the first neural network to be trained in the downstream task and the second neural network obtained by training based on the first image sample collected in the upstream task. extraction, and then the first neural network can be trained according to the obtained first image features and the second image features. Since the new images generated based on the image generation network are image samples that are more in line with the overall distribution of the first image samples, such image samples are more conducive to adapting to the network environment of the second neural network. The first image feature output by the second neural network and the second image feature output by the first neural network to be trained can better guide the training of the first neural network, thereby further improving the performance in downstream tasks.

In a possible implementation, the acquisition module 301 is used to train the image generation network according to the following steps:

obtaining the first image output based on the codebook generation network; the codebook generation network is used to generate a codebook that decomposes the first image sample into a plurality of primitives;

Inputting the first image to the image generation network to be trained to obtain the second image output by the image generation network;

Determine the loss function value of the image generation network to be trained based on the image similarity between the second image and the first image;

The image generation network to be trained is trained based on the loss function value, and the trained image generation network is obtained.

In a possible implementation manner, the acquisition module 301 is configured to input the first image to the image generation network to be trained:

performing masking processing on part of the image area in the first image to obtain the masked first image;

Input the masked first image to the image generation network to be trained.

In a possible implementation, the codebook generation network includes an encoder and a decoder, and an acquisition module 301 is used to train the codebook generation network according to the following steps:

Repeat the following steps until the similarity between the image output by the decoder and the first image sample input into the encoder is greater than a preset threshold:

The first image sample is input into the encoder to be trained to obtain a codebook output by the encoder; the codebook output from the encoder is input to the decoder to be trained to obtain an image output by the decoder.

In a possible implementation manner, the obtaining module 301 is configured to obtain the first image output based on the codebook generation network according to the following steps:

Inputting the first image sample into the encoder included in the codebook generation network to obtain the codebook output by the encoder;

The codebook output by the encoder is input to the decoder included in the codebook generation network to obtain the first image output by the decoder.

In a possible implementation, the image generation network includes a first generation sub-network for generating a codebook that decomposes the first image sample into a plurality of primitives, and generates a new generation sub-network based on the image output by the first generation sub-network The second generation sub-network of the image; the acquisition module 301 is used to train the image generation network according to the following steps:

Input the first image sample into the trained first generation sub-network to obtain the first image output by the first generation sub-network;

Input the first image to the second generation sub-network to be trained, and obtain the second image output by the second generation sub-network;

Determine the loss function value of the image generation network to be trained based on the first image similarity between the first image and the input first image sample, and the second image similarity between the second image and the first image;

The image generation network to be trained is trained based on the loss function value, and the trained image generation network is obtained.

In a possible implementation, the training module 303 is used to train the first neural network to be trained based on the first image feature and the second image feature according to the following steps, to obtain a trained first neural network:

Determine the loss function value of the first neural network to be trained based on the image similarity between the first image feature and the second image feature;

When the loss function value corresponding to the current round is greater than the preset threshold, the network parameter value of the first neural network is adjusted based on the loss function value, and the next round of training is performed according to the adjusted first neural network until the loss function The value is less than or equal to the preset threshold.

In a possible implementation manner, the training module 303 is further used for:

After the trained first neural network is obtained, a second image sample collected in the downstream task is obtained; network training is performed again on the trained first neural network based on the second image sample to obtain a final trained first neural network.

In a possible implementation, the training module 303 is configured to perform network training again on the trained first neural network based on the second image sample according to the following steps to obtain the final trained first neural network:

Input the second image sample into the first neural network to obtain the task output result of the network;

Determine the loss function value of the first neural network based on the comparison between the task output result and the task labeling result labelled for the second image sample;

Network training is performed on the first neural network again based on the loss function value to obtain a final trained first neural network.

In a possible implementation manner, the acquisition module 301 is configured to train the second neural network according to the following steps:

Obtain the original neural network; the original neural network includes at least a feature extraction layer;

Perform feature extraction on the first image sample based on the feature extraction layer included in the original neural network to obtain image feature information output by the feature extraction layer;

Adjust the network parameter values of the feature extraction layer based on the image feature information to obtain the adjusted feature extraction layer;

The original neural network including the adjusted feature extraction layer is determined as the second neural network obtained by training.

Referring to FIG. 4 , which is a schematic diagram of an apparatus for target detection provided by an embodiment of the present disclosure, the apparatus includes: an acquisition module 401 and a detection module 402 ; wherein,

an acquisition module 401, configured to acquire target images collected in downstream tasks;

The detection module 402 is configured to input the target image into the first neural network trained by the neural network training method to obtain the detection result of the target object in the target image.

For the description of the processing flow of each module in the apparatus and the interaction flow between the modules, reference may be made to the relevant descriptions in the foregoing method embodiments, which will not be described in detail here.

Corresponding to the methods in FIGS. 1 and 2 , an embodiment of the present disclosure further provides an electronic device. As shown in FIG. 5 , a schematic structural diagram of the electronic device provided by an embodiment of the present disclosure includes:

The processor 501, the memory 502, and the bus 503; the memory 502 is used to store the execution instructions, including the memory 5021 and the external memory 5022; the memory 5021 here is also called the internal memory, which is used to temporarily store the operation data in the processor 501, and For the data exchanged by the external memory 5022 such as the hard disk, the processor 501 exchanges data with the external memory 5022 through the memory 5021. When the electronic device is running, the processor 501 and the memory 502 communicate through the bus 503, so that the processor 501 executes the data shown in FIG. 1 . The steps of the method for training a neural network shown in FIG. 2 or the steps for performing the method for target detection shown in FIG.

Embodiments of the present disclosure further provide a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, the steps of the methods described in the foregoing method embodiments are executed. Wherein, the storage medium may be a volatile or non-volatile computer-readable storage medium.

Embodiments of the present disclosure further provide a computer program product, where the computer program product carries program codes, and the instructions included in the program codes can be used to execute the steps of the methods described in the foregoing method embodiments. For details, please refer to the foregoing method embodiments. , and will not be repeated here.

Wherein, the above-mentioned computer program product can be specifically implemented by means of hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Softw first neural network reDevelopment Kit). , SDK) and so on.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the system and device described above, reference may be made to the corresponding process in the foregoing method embodiments, which will not be repeated here. In the several embodiments provided by the present disclosure, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. The apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a processor-executable non-volatile computer-readable storage medium. Based on such understanding, the technical solutions of the present disclosure can be embodied in the form of software products in essence, or the parts that contribute to the prior art or the parts of the technical solutions. The computer software products are stored in a storage medium, including Several instructions are used to cause an electronic device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Re first neural network d-Only Memory, ROM), random access memory (R first neural network ndom first neural network accessMemory, R first neural network accessMemory, R first neural network d-Only Memory, ROM) Various media that can store program codes, such as neural network M), magnetic disk or optical disk.

Finally, it should be noted that the above-mentioned embodiments are only specific implementations of the present disclosure, and are used to illustrate the technical solutions of the present disclosure rather than limit them. The protection scope of the present disclosure is not limited thereto, although referring to the foregoing The embodiments describe the present disclosure in detail, and those skilled in the art should understand that: any person skilled in the art can still modify the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present disclosure. Changes can be easily thought of, or equivalent replacements are made to some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered in the present disclosure. within the scope of protection. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims (15)

1. a method for neural network training, wherein the method comprises: Obtain the first image sample collected in the upstream task, the first neural network to be trained in the downstream task, and the second neural network and image generation network obtained by training based on the first image sample; the second neural network is used for performing feature extraction, the image generation network is used to generate a new image, and the new image conforms to the overall distribution of the first image samples; Perform feature extraction on the new image generated based on the image generation network according to the second neural network obtained by training and the first neural network to be trained, to obtain the first image feature and the second image feature; The first neural network to be trained is trained based on the first image feature and the second image feature to obtain a trained first neural network. 2. The method according to claim 1, wherein the image generation network is trained according to the following steps: obtaining a first image output based on a codebook generation network; the codebook generation network is used to generate a codebook that decomposes the first image sample into a plurality of primitives; Inputting the first image to the image generation network to be trained to obtain a second image output by the image generation network; determining a loss function value of the image generation network to be trained based on the image similarity between the second image and the first image; The image generation network to be trained is trained based on the loss function value to obtain a trained image generation network. 3. The method according to claim 2, wherein the inputting the first image to the image generation network to be trained comprises: performing masking processing on part of the image area in the first image to obtain a masked first image; The masked first image is input to the image generation network to be trained. 4. The method according to claim 2 or 3, wherein the codebook generation network comprises an encoder and a decoder, and the codebook generation network is trained according to the following steps: The following steps are repeated until the similarity between the image output by the decoder and the first image sample input into the encoder is greater than a preset threshold: Input the first image sample into the encoder to be trained to obtain the codebook output by the encoder; input the codebook output from the encoder to the decoder to be trained to obtain the image output from the decoder . 5. method according to claim 4, is characterized in that, obtains the first image based on codebook generation network output according to the following steps: inputting the first image sample into an encoder included in the codebook generation network to obtain a codebook output by the encoder; The codebook output by the encoder is input to the decoder included in the codebook generation network to obtain the first image output by the decoder. 6. The method of claim 1, wherein the image generation network comprises a first generation sub-network for generating a codebook that decomposes the first image sample into a plurality of primitives, and a The image output by the first generation sub-network, and the second generation sub-network of the new image is generated; the image generation network is trained according to the following steps: Inputting the first image sample into the trained first generation sub-network to obtain the first image output by the first generation sub-network; Inputting the first image to the second generation sub-network to be trained to obtain a second image output by the second generation sub-network; Determine the to-be-trained based on the first image similarity between the first image and the input first image sample, and the second image similarity between the second image and the first image The loss function value of the image generation network; The image generation network to be trained is trained based on the loss function value to obtain a trained image generation network. 7. The method according to any one of claims 1 to 6, wherein the first neural network to be trained is trained based on the first image feature and the second image feature to obtain the training Good first neural networks, including: determining a loss function value of the first neural network to be trained based on the image similarity between the first image feature and the second image feature; When the loss function value corresponding to the current round is greater than a preset threshold, adjust the network parameter value of the first neural network based on the loss function value, and perform the next step according to the adjusted first neural network. rounds of training until the loss function value is less than or equal to the preset threshold. 8. The method according to any one of claims 1 to 7, wherein, after the trained first neural network is obtained, the method further comprises: obtaining the second image sample collected in the downstream task; Perform network training again on the trained first neural network based on the second image sample to obtain a final trained first neural network. 9. The method according to claim 8, wherein the network training is performed again on the trained first neural network based on the second image sample to obtain the final trained first neural network, comprising: : Inputting the second image sample into the first neural network to obtain a task output result of the network; determining a loss function value of the first neural network based on the comparison between the task output result and the task labeling result labelled for the second image sample; Perform network training on the first neural network again based on the loss function value to obtain a final trained first neural network. 10. The method according to any one of claims 1 to 9, wherein the second neural network is trained according to the following steps: Obtain an original neural network; the original neural network includes at least a feature extraction layer; Perform feature extraction on the first image sample based on the feature extraction layer included in the original neural network to obtain image feature information output by the feature extraction layer; Adjust the network parameter values of the feature extraction layer based on the image feature information to obtain an adjusted feature extraction layer; The original neural network including the adjusted feature extraction layer is determined as the second neural network obtained by training. 11. A method for target detection, wherein the method comprises: Obtain target images collected in downstream tasks; The target image is input into the first neural network trained by the neural network training method according to any one of claims 1 to 10 to obtain the detection result of the target object in the target image. 12. A device for neural network training, wherein the device comprises: an acquisition module, configured to acquire a first image sample collected in an upstream task, a first neural network to be trained in a downstream task, and a second neural network and an image generation network trained based on the first image sample; the Two neural networks are used for feature extraction, and the image generation network is used to generate a new image, and the new image conforms to the overall distribution of the first image samples; an extraction module, configured to perform feature extraction on a new image generated based on the image generation network according to the second neural network obtained by training and the first neural network to be trained, respectively, to obtain the first image feature and the second image feature; A training module, configured to train the first neural network to be trained based on the first image feature and the second image feature to obtain a trained first neural network. 13. A device for target detection, characterized in that the device comprises: The acquisition module is used to acquire the target image collected in the downstream task; The detection module is configured to input the target image into the first neural network trained by the neural network training method according to any one of claims 1 to 10, to obtain the detection result of the target object in the target image. 14. An electronic device, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the The memory is communicated through a bus, and when the machine-readable instructions are executed by the processor, the steps of the neural network training method according to any one of claims 1 to 10 or the target detection method according to claim 11 are executed. steps of the method. 15. A computer-readable storage medium, characterized in that, a computer program is stored on the computer-readable storage medium, and the computer program is executed by a processor when the neural network training according to any one of claims 1 to 10 is executed. The steps of the method or the steps of the method of object detection as claimed in claim 11 .

Images