JP7267379B2 - Image processing method, pre-trained model training method, device and electronic equipment - Google Patents

Description

The present application relates to the field of image processing technology, specifically to the field of deep learning, computer vision technology, and further to an image processing method, a pre-trained model training method, an apparatus and an electronic device.

Image processing technology based on neural networks has been developing for many years, and according to the needs of image processing, it uses trained image processing models to perform image processing and recognition. Having processing needs and using a fixed image processing model for image processing cannot meet the image processing needs in different scenarios, so how to improve the effect of image processing is an urgent solution. This is a technical issue that should be addressed.

The present application provides an image processing method, a pre-trained model training method, an apparatus and an electronic device for improving the effect of image processing.

According to one aspect of the present application, there is provided an image processing method, the step of obtaining a trained pre-trained model, said pre-trained model having an image feature output from the trained pre-trained model of , is trained using training images of a plurality of frames to satisfy that the difference between the first image feature distance and the second image feature distance is minimal, wherein the first image feature distance is the same distances between image features of training images extracted from video clips, said second image distance being distances between image features of training images extracted from different video clips; generating an image processing model that performs a target image processing task based on the , and using the image processing model to perform the target image processing task on the target image.

According to another aspect of the present application, there is provided a method for training a pre-trained model, obtaining a plurality of video clips; extracting multiple frames of training images from the plurality of video clips to obtain a training set; extracting at least two frames of said training images from each said video clip; and using said training set to perform multi-round training on a pre-trained model for image feature extraction. each round of training comprises selecting from said training set each training image extracted from at least two video clips; and applying each said training image selected in this round to said preliminary Inputting a training model to obtain output image features, and based on the image features of each said training image selected in this round, a first image feature distance between training images belonging to the same video clip. , determining a second image feature distance between training images belonging to different video clips, and based on the first image feature distance and the second image feature distance, the first image feature distance adjusting model parameters of the pre-trained model such that the difference between the distance and the second image feature distance is minimized.

According to another aspect of the present application, an acquisition module for providing an image processing device and acquiring a trained pre-trained model, the pre-trained model output from the trained pre-trained model. is trained using a plurality of frames of training images to satisfy a minimum difference between a first image feature distance and a second image feature distance; an acquisition module, wherein the distances are distances between image features of training images extracted from the same video clip and the second image distances are distances between image features of training images extracted from different video clips; a generation module for generating an image processing model to perform a target image processing task based on said pre-trained model; and using said image processing model to perform a target image processing task on a target image. and a processing module of

According to another aspect of the present application, there is provided a pre-trained model training apparatus, an acquisition module for acquiring a plurality of video clips, and extracting a plurality of frames of training images from the plurality of video clips for training an extraction module for obtaining a set, the extraction module extracting at least two frames of said training images from each said video clip; a training module for performing multi-round training on the training set, each round of training comprising selecting each training image extracted from at least two video clips from said training set; inputting each said training image obtained into said pre-training model to obtain output image features; and based on the image features of each said training image selected in this round, belonging to the same video clip. determining a first image feature distance between training images; determining a second image feature distance between training images belonging to different video clips; combining said first image feature distance and said second image feature distance; and adjusting model parameters of the pre-trained model such that the difference between the first image feature distance and the second image feature distance is minimized based on.

According to another aspect of the present application, an electronic apparatus is provided and includes at least one processor and memory communicatively coupled to the at least one processor, the memory including: are stored with instructions executable by the at least one processor to enable the at least one processor to perform an image processing method according to one aspect, or a pre-trained model training method according to another aspect. executed by one processor.

According to another aspect of the present application, there is provided a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions being stored in a computer to process an image processing method according to one aspect, or an image processing method according to another aspect. Running the pre-trained model training method according to the aspects.
According to another aspect of the present application there is provided a computer program, said computer program causing a computer to perform an image processing method according to one aspect or a pre-trained model training method according to another aspect.

It should be noted that the content described in this section is not intended to limit the key or critical features of the embodiments of the application, nor is it intended to limit the scope of the application. Other features of the present application will become readily comprehensible with the following description.

The drawings are used for better understanding of the present technical solution and are not intended to limit the present application.
1 is a schematic flowchart of an image processing method provided by an embodiment of the present application; 4 is a schematic flowchart of another image processing method provided by an embodiment of the present application; 1 is a schematic block diagram of an image processing model provided by an embodiment of the present application; FIG. 1 is a schematic flow chart of a pre-trained model training method provided by an embodiment of the present application; 1 is a schematic configuration diagram of an image processing device provided by an embodiment of the present application; FIG. 1 is a schematic structural diagram of a pre-trained model training device provided by an embodiment of the present application; FIG. 1 is a block diagram of an electronic device according to an embodiment of the present application; FIG.

Illustrative embodiments of the present application are described below in conjunction with the drawings, and various details of the embodiments of the present application are included therein for ease of understanding and are merely exemplary. should be regarded as Accordingly, those skilled in the art may make various changes and modifications to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for the sake of clarity and brevity, the following description omits descriptions of well-known functions and constructions.

Hereinafter, an image processing method, a pre-trained model training method, an apparatus, and an electronic device according to embodiments of the present application will be described with reference to the drawings.

FIG. 1 is a schematic flowchart of an image processing method provided by an embodiment of the present application.

As shown in FIG. 1, the method includes the following steps 101-103.

Step 101, obtain a trained pre-training model, the pre-training model is such that the image feature output from the trained pre-training model has a minimum difference between the first image feature distance and the second image feature distance; where the first image feature distance is the distance between the image features of the training images extracted from the same video clip, and the second is the distance between image features of training images extracted from different video clips.

The pre-trained model in this example can be trained by deep learning methods when training, and compared to other machine learning methods, deep learning performs better on big data sets. The pre-trained model in this example inputs multiple frames of training images extracted from multiple video clips as a training set to the pre-trained model, until the results output from the pre-trained model meet a preset threshold. , by means of which the parameters of the pre-trained model are continuously adjusted to perform iterative training on the pre-trained model, and then the training is terminated, so based on a large amount of image data, a generic pre-trained model and subsequently improve the generation efficiency of the corresponding target image processing model based on this general pre-trained model.

Here, the training method of the pre-trained model will be described in detail in the subsequent example of the training method of the pre-trained model, but the description will be omitted in this example.

Step 102, generate an image processing model that performs a target image processing task based on the pre-trained model.

Target image processing tasks include image classification tasks, target detection tasks, or object recognition tasks.

In the present application, after the pre-training model is generated, based on the set of images corresponding to the target image processing task, because the pre-training model is a pre-generated general model, the image to perform the corresponding target image processing task To quickly generate a processing model and improve the efficiency of generating an image processing model corresponding to a target image processing task.

Here, the image processing model may be a convolutional neural network model CNN (Convolutional Neural Neural Networks, CNN) or a deep neural network model DNN (Deep Neural Networks, DNN). Not limited.

Step 103, perform a target image processing task on the target image using the image processing model.

The image processing model of this embodiment is an image processing model corresponding to the target image processing task, generated based on a general-purpose pre-trained model obtained by pre-training, which improves model generation efficiency, The image processing model is used to perform a target image processing task on the target image to improve the execution effect and processing efficiency of the target image processing task.

An image processing method according to an embodiment of the present application obtains a trained pre-trained model, wherein the pre-trained model is such that an image feature output from the trained pre-trained model is a first image feature It is trained using multiple frames of training images to satisfy the minimum difference between the distance and the second image feature distance. In addition, a generic pre-training model and a target image processing task generate a corresponding image processing model, improve the generation efficiency of the image processing model corresponding to the target processing task, and use the generated image processing model to , performs the target image processing task on the target image, and the image processing model corresponds to the target image processing task, thus improving the effect and efficiency of image processing.

In the above example, in order to improve the efficiency of image processing, an image processing model corresponding to the target image processing task is generated based on the target image processing task and the pre-trained model. Based on this, the pre-trained model can be trained to generate an image processing model corresponding to the image processing task, thereby improving the efficiency of image processing. As another possible embodiment, after splicing a pre-trained model and a network layer corresponding to the target processing task, training is performed to obtain the corresponding image processing model, thereby improving the generation efficiency of the image processing model and the image processing performance. It can also improve effectiveness.

Therefore, based on the above embodiments, this embodiment provides another image processing method, and FIG. 2 is a schematic flow chart of another image processing method provided by the embodiments of the present application. , the above step 102 includes the following steps 201-203.

Step 201, obtain a network layer corresponding to a target image processing task.

In this application, there is correspondence between the obtained network layer and the target image processing task.

In one scenario, if the target image processing task is an image classification task, the corresponding network layer is the classification layer and is used to classify the target image, e.g. Determine the corresponding vehicle category, eg, passenger car, SUV, and the like.

In another scenario, if the target image processing task is a target detection task, the corresponding network layer is the detection network, which is used to recognize target objects contained in the target image, e.g. On the other hand, it detects whether or not an image contains an obstacle, or detects whether or not a plurality of images contain the same target object.

In yet another scenario, if the target image processing task is an object recognition task, the corresponding network layer is used to recognize objects in the image, e.g., for the target image to be processed, different regions in the image Recognize the object category contained in the image, or recognize the type of object contained in the image.

Step 202, splicing the pre-trained model and the network layer, where the input of the network layer is the image feature output from the pre-trained model, and the output of the network layer is the processing result of the target image task.

In this embodiment, after the general pre-trained model is generated, the pre-trained model and the network layer corresponding to the target image processing task are spliced, and the pre-trained model obtained by training is used as shown in FIG. and splice the network layers to get the image processing model to be trained. Here, the image features output from the pre-trained model are input to the network layer, and the output of the network layer is the processing result of the target image task.

Step 203, using the training set of the target image processing task to train the spliced pre-trained model and network layers to obtain an image processing model.

In this embodiment, for different target image processing tasks, the training set corresponding to the target image processing task is used to quickly acquire the image processing model corresponding to the target image processing task. The pre-trained model and network layers are trained to obtain the image processing model. That is, the image processing model obtained by training and the target image processing task have a corresponding relationship, and the general preprocessing model completed based on the training in advance and the corresponding network layer are spliced and then trained. As a possible embodiment, mainly for the requirements of the target image processing task, the parameters of the network layer can be adjusted to improve the training efficiency of the corresponding image processing model, and for different target image processing tasks. Meet your processing needs at the same time and meet your processing needs in different scenarios.

In the image processing method of this embodiment, the completed universal preprocessing model and the corresponding network layer are spliced based on the pre-training, where the input of the network layer is the image feature output from the pre-training model. , the output of the network layer is the processing result of the target image task, and we train it further, and the training is mainly for the network corresponding to the target image processing task, so the amount of data for training is small, and the corresponding image processing model Improve training efficiency.

In order to implement the above embodiments, this embodiment provides a training method for pre-trained models.

FIG. 4 is a schematic flowchart of a pre-trained model training method provided by an embodiment of the present application, and as shown in FIG. 4, the method includes the following steps 401-403.

Step 401, get a plurality of video clips.

In one possible embodiment of the implementation of the present application, at least one video may be obtained and each video may be randomly split into multiple video clips.

In one possible embodiment, in order to obtain more video clips, multiple videos are obtained, and based on content differences between adjacent image frames in each video, a segmentation process is performed to divide each video. You can get multiple video clips. That is, when video clip division is performed for each video, the content of each frame in the video clip obtained by division changes continuously, improving the continuity of the frames in the video clip.

Another possible embodiment of an embodiment of the present application is to take a video and perform a splitting process based on content differences between adjacent image frames in this video to get multiple video clips. can be done. In other words, when a video is divided into video clips, the content of each frame in the divided video clip changes continuously, improving the continuity of the frames in the video clip.

As shown in FIG. 3, A, B, . . . N are different video clips.

In some scenarios, these different video clips may have been split from a single video clip. In another scenario, these different video clips may have been split and captured from multiple video bands. Specifically, it can be flexibly set according to the needs of the training scenario, and is not limited in this embodiment.

Step 402, extracting multiple frames of training images from multiple video clips to obtain a training set, wherein extracting at least two frames of training images from each video clip.

In this example, the training set consists of multiple frames of training images extracted from multiple video clips. One possible embodiment is to randomly extract a fixed number of training image frame numbers from each video clip and use the extracted video clip frame numbers to construct a training set. Now extract at least two frames of training images from each video clip.

As another possible embodiment, in order to improve the training effect of the model, the homogeneity of the frame number distribution of each video clip in the training set can be improved, since the training images extracted from each video clip have the same number of frames. and train a pre-trained model through this training set so that each video clip has the same proportion of weight in determining model parameters, improving the training effect of subsequent pre-trained models. .

As shown in FIG. 3, A, B, and N are different video clips, and in this embodiment, an example of extracting two frames from each video clip and using them as training images will be described. where A1 and A2 are two frames in video clip A, B1 and B2 are two frames in video clip B, and N1 and N2 are two frames in video clip N.

For example, with one video X, split this video clip to get three video clips, which are video clips A, B, and C, and as shown in Figure 3, N is C and each video clip An example of extracting two frames from .

Here, in video clip A, the extracted two frame images are A1 and A2, and A1 and A2 are two consecutive frames. In video clip B, the extracted two frame images are B1 and B2, and B1 and B2 are two consecutive frames. In video clip C, the extracted two frame images are C1 and C2, and C1 and C2 are two consecutive frames. Further, image frames A1, A2, B1, B2, C1 and C2 are constructed as a training set.

In addition, in practical applications, the number of multi-frame training images included in the training set is not limited to the 6-frame images described in this embodiment, and can be flexibly set according to the training accuracy needs. can be done.

Step 403, using the training set to perform multi-round training on the pre-trained model for image feature extraction, where each round of training is extracted from the training set with at least two video clips. Select each training image selected in this round, input each training image selected in this round into the pre-training model to get the output image features, and then use the image features of each training image selected in this round to determine a first image feature distance between training images belonging to the same video clip and a second image feature distance between training images belonging to different video clips; , the model parameters of the pre-trained model are adjusted so that the difference between the first image feature distance and the second image feature distance is minimized, so that the pre-trained model obtained by training is: Including being a generic pre-trained model that can recognize association relationships between different video clips.

In this example, the training set is used to perform multiple rounds of training on the pre-trained model, and in each round of training, until the model converges, the training effect is determined based on the recognition results, By adjusting the parameters of the pre-trained model, the pre-trained model can accurately generate the image features of the training images. In this example, with the training images in the training set, we pre-train to obtain a generic pre-training model, and the image features output from the pre-training model are used as the generic result of image recognition for subsequent target image recognition. In combination with the task, it can facilitate the rapid acquisition of the image processing model corresponding to the target image recognition task, and improve the generation efficiency of the image processing model.

Note that the training set contains multiple video clips belonging to the same video and also multiple video clips belonging to different videos, so in each round of training, at least two video clips are extracted from the training set. , where the two video clips may belong to the same video or to different videos, and the extracted training images are used to determine the association relationship between the different video clips to make it a generic pre-trained model and improve the robustness of the generic model.

A training method for a pre-trained model according to an embodiment of the present application extracts at least two frames of training images from each of a plurality of captured video clips to obtain a plurality of frames of training images to obtain a training set. , perform multi-round training on the pre-trained model for image feature extraction through the training set, and in each round of training, based on the training images, we get the image features and extract the images of the images belonging to the same video clip Obtaining a first image feature distance between the images based on the features; obtaining a second image feature distance between the images based on the image features of images belonging to different video clips; obtaining the first image feature distances By continuously adjusting the parameters of the pre-trained model so that the difference between the feature distances of the second image and Improve feature reliability.

Based on the above example, this example provides another training method of pre-training model, in order to improve the accuracy of calculating the first image feature distance, between training images belonging to the same video clip We describe how to determine the first image feature distance, which can be specifically achieved by the following steps.

For the training images that were input to the pre-training model in this round of training, we determined the within-class feature distances between the image features of different training images belonging to the same video clip, and the values selected from the training set in this round of training. For at least two video clips, determine the sum of intra-class feature distances to obtain a first image feature distance, and associate between image features of different training images belonging to the same video clip by the first image distance. Realize showing relationships.

In one possible implementation of embodiments of the present application, for example, the selected training images i1 and i2 belong to the same video clip i, and the training images i1 and i2 are input to the pre-training module to obtain the image of each training image The features are obtained and denoted as hi1 and hi2, respectively. In addition, we compute the intra-class feature distance d(i1, i2) between the image features hi1 and hi2 of training images i1 and i2 belonging to the same video clip i, and also For at least two video clips, determine the sum of intra-class feature distances to obtain the first image feature distance dist(in), specifically realized by the following formula:

Here, i is a video clip, that is, a video clip is a natural number from 1 to n, where n is 2 or more.

In another possible embodiment of the examples of the present application, to meet the needs of different scenarios, classify the image features of different training images with respect to the image features of different training images belonging to the same video clip, i.e. The image features of different training images are divided into different categories to achieve refined feature recognition. For example, determine which image features belong to the People category, which belong to Buildings, or which belong to the Nose category, and furthermore, for different training images, we determine which image features correspond to the same category in the image features of any two training images. Compute the inter-category feature distance for each feature, and sum all the inter-category feature distances to get the intra-class feature distance belonging to the same video clip. Further, for at least two video clips selected from the training set in this round of training, determine the sum of the within-class feature distances to obtain the first image feature distance, Calculation precision is achieved to improve the accuracy of the calculation of the first image feature distance.

Note that the image feature distance can be calculated based on Euclidean distance or cosine distance.

Based on the above example, this example provides another training method of pre-training model, in order to improve the accuracy of the calculation of the second image feature distance, the first difference between the training images belonging to different video clips is provided. 2, which can be specifically realized by the following steps.

For the training images that were input to the pre-trained model in this round of training, we determined the inter-class feature distances between the image features of different training images belonging to different video clips and selected from the training set in this round of training. determining a sum of inter-class feature distances for at least two video clips to obtain a second image feature distance, and association between image features of different training images belonging to different video clips by the second image distance; Realize showing relationships.

In one possible embodiment of an embodiment of the present application, for example, selected training images i1 and i2 belong to the same video clip i, training images j1 and j2 belong to the same video clip j, and training images i1 and i2 are Input a pre-training module to obtain image features for each training image, denoted as hi1 and hi2, respectively, and input training images j1 and j2 to the pre-training module to obtain corresponding image features, hj1 and hj1, respectively. denoted as hj2. Further, compute the inter-class feature distance between the image features of the training images belonging to different video clips i and j; Determine the sum of the distances to obtain a second image feature distance dist(between). Specifically, it can be realized by the following formula.

where i and j are different video clips, n is greater than or equal to 2, and d(hi1, hj1) is the interclass feature distance between image features hi1 and hj1 of training images in different video clips i and j. where d(hi1, hj2), d(hi2, hj1) and d(hi2, hj2) are interclass feature distances between image features of training images in different video clips i and j.

It should be noted that in this embodiment, the selection of two training images from each video clip is taken as an example, and in actual application, the number of selected training images in each video clip is flexible according to the training needs. , and is not limited in this embodiment.

In another possible embodiment of the embodiments of the present application, image features of different training images can be classified for image features of different training images belonging to different video clips to meet the needs of different scenarios. That is, the image features of different training images can be divided into different categories to achieve refined feature recognition. For example, determine which image features belong to the person category, which belong to buildings, or which belong to the nose category, and furthermore, for training images belonging to different video clips, for any two training images, the same Calculate the inter-category feature distances respectively for the features corresponding to the categories, and sum all the inter-category feature distances to obtain the inter-class feature distances of different training images belonging to different video clips. Further, for at least two video clips selected from the training set in this round of training, determine the sum of the inter-class feature distances to obtain a second image feature distance, Calculation precision is achieved to improve the accuracy of the calculation of the second image feature distance.

Note that the image feature distance can be calculated based on Euclidean distance or cosine distance.

In order to implement the above embodiments, the present application further provides an image processing device.

FIG. 5 is a schematic configuration diagram of an image processing device provided by an embodiment of the present application.

As shown in FIG. 5, it includes an acquisition module 51 , a generation module 52 and a processing module 53 .

Acquisition module 51 acquires a trained pre-trained model, wherein the image features output from the trained pre-trained model are the first image feature distance and the second image feature distance. trained using multiple frames of training images to satisfy the minimum difference, wherein the first image feature distance is the distance between the image features of the training images extracted from the same video clip; The second image feature distance is the distance between image features of training images extracted from different video clips.

Generation module 52 generates an image processing model that performs a target image processing task based on the pre-trained model.

The processing module 53 uses the image processing model to perform target image processing tasks on the target image.

Further, in one possible embodiment of the examples of the present application, the generation module 52 specifically obtains the network layer corresponding to the target image processing task, splices the pre-trained model and the network layer, where , the input of the network layer is the image features output from the pre-trained model, the output of the network layer is the processing result of the target image task, and the training set of the target image processing task is used to generate the spliced pre- The training model and network layers are trained to obtain an image processing model.

In one possible embodiment of the examples of the present application, the target image processing task comprises an image classification task, a target detection task or an object recognition task.

The description of the embodiment of the image processing method is also applied to the image processing apparatus of this embodiment, and the principle is the same, so the description is omitted here.

An image processing apparatus according to an embodiment of the present application obtains a trained pre-trained model, wherein the pre-trained model is such that the image features output from the trained pre-trained model are first image features trained using multiple frames of training images to satisfy the minimum difference between the distance and the second image feature distance, and based on a generic pre-trained model and a target image processing task , generate a corresponding image processing model, improve the generation efficiency of the image processing model corresponding to the target processing task, use the generated image processing model to perform the target image processing task on the target image, Improving the effectiveness and efficiency of image processing because the image processing model corresponds to the target image processing task.

In order to implement the above embodiments, this embodiment provides a pre-trained model training device.

FIG. 6 is a schematic structural diagram of a pre-trained model training device provided by an embodiment of the present application. As shown in FIG. 6, the device includes an acquisition module 61 , an extraction module 62 and a training module 63 .

Acquisition module 61 acquires a plurality of video clips.

Extraction module 62 extracts multiple frames of training images from multiple video clips to obtain a training set, where it extracts at least two frames of training images from each video clip.

Training module 63 uses the training set to perform multiple rounds of training on the pre-trained model for image feature extraction, where each round of training consists of at least two video clips from the training set. and inputting each training image selected in this round into the pre-training model to obtain the output image features, and each training image selected in this round Based on the image features of the images, determine a first image feature distance between training images belonging to the same video clip and a second image feature distance between training images belonging to different video clips; and adjusting model parameters of the pre-trained model based on the second image feature distance such that the difference between the first image feature distance and the second image feature distance is minimized.

In one possible embodiment of the examples of the present application, the training module 63 specifically stores different training images belonging to the same video clip for training images input to said pre-trained model in this round of training. and for at least two video clips selected from the training set in this round of training, determining the sum of the intra-class feature distances between the image features of the first Get the image feature distance.

In one possible embodiment of the examples of the present application, the training module 63 specifically stores different training images belonging to different video clips for the training images input to said pre-trained model in this round of training. and for at least two video clips selected from the training set in this round of training, determining the sum of the interclass feature distances between the image features of the second Get the image feature distance.

In one possible embodiment of the implementation of the present application, the number of frames of training images extracted from each said video clip is the same.

In one possible embodiment of the present application, the acquisition module 61 specifically acquires a plurality of videos and performs a segmentation process based on content differences between adjacent image frames in each said video to obtain each said video. Get multiple video clips of a video.

The pre-trained model training device according to an embodiment of the present application extracts at least two frames of training images from each of the captured video clips to obtain a plurality of frames of training images to obtain a training set. , perform multi-round training on the pre-trained model for image feature extraction through the training set, and in each round of training, based on the training images, we get the image features and extract the images of the images belonging to the same video clip Obtaining a first image feature distance between the images based on the features; obtaining a second image feature distance between the images based on the image features of images belonging to different video clips; obtaining the first image feature distances Continuously adjust the parameters of the pre-trained model so that the difference between the distance between the and the second image feature distance is minimized, to realize the training of the general pre-trained model, and the image feature recognized by the pre-trained model Improve reliability.

To implement the above embodiments, an embodiment of the present application provides an electronic device, including at least one processor and memory communicatively coupled to the at least one processor, the memory comprising , instructions executable by said at least one processor are stored, said instructions directing said at least one processor to perform an image processing method according to said method embodiment or a training method according to said method embodiment. preferably executed by said at least one processor.

To implement the above embodiments, embodiments of the present application further provide a non-transitory computer-readable storage medium having computer instructions stored thereon, said computer instructions being transmitted to a computer by the method embodiments described above. or the training method described in the embodiment of said method.

According to embodiments of the present application, the present application further provides an electronic device and a readable storage medium.
According to an embodiment of the present application, the present application provides a computer program that causes a computer to perform the image processing method or the pre-trained model training method provided by the present application.

As shown in FIG. 7, it is a block diagram of an electronic device according to an embodiment of the present application. Electronic equipment is intended to represent various forms of digital computers such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronics can also represent various forms of mobile devices such as personal digital processors, mobile phones, smart phones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functionality illustrated herein are merely examples and are not intended to limit the description and/or required implementation of the application herein.

As shown in FIG. 7, the electronic device includes one or more processors 701, memory 702, and interfaces for connecting components, including high speed and low speed interfaces. Each component is interconnected by a different bus and can be mounted on a common motherboard or otherwise mounted as desired. The processor executes instructions executed within the electronic device, including instructions in or stored in memory to display graphical information of the GUI on an external input/output device (such as a display device coupled to the interface). can be processed. In other embodiments, multiple processors and/or multiple buses can be used along with multiple memories, if desired. Similarly, multiple electronic devices can be connected, each providing a partial required operation (eg, being a server array, blade server, or multi-processor system). In FIG. 7, one processor 701 is taken as an example.

Memory 702 is a non-transitory computer-readable storage medium provided by the present application. Here, the memory stores instructions executable by the at least one processor so that the at least one processor performs the image processing method provided by the present application. A non-transitory computer-readable storage medium of the present application stores computer instructions for causing a computer to perform the image processing method provided by the present application.

The memory 702, as a non-transitory computer-readable storage medium, stores program instructions/modules (for example, the acquisition module 51, the generation module 52, the processing module 53 shown in FIG. 5) corresponding to the image processing method in the embodiments of the present application. , ), non-transitory computer-executable programs and modules. Processor 701 performs the various functional applications and data processing of the server by executing non-transitory software programs, instructions and modules stored in memory 702, i.e. image processing methods in the above method embodiments. Realize

The memory 702 can include a program storage area and a data storage area, where the program storage area can store an operating system, application programs required for at least one function, and the data storage area can: Data created by the use of image processing method electronics and the like can be stored. Memory 702 may also include high speed random access memory and may further include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state memory device. A storage device. In some embodiments, memory 702 can optionally include memory configured remotely to processor 701, and these remote memories can be connected to the electronic device via a network. can. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

The image processing method electronics can further include an input device 703 and an output device 704 . The processor 701, memory 702, input device 703, and output device 704 can be connected via a bus or other manner, and FIG. 8 takes the connection by bus as an example.

The input device 703 is capable of receiving entered numeric or character information and generating key signal inputs for user settings and functional control of this electronic device, such as touch screen, keypad, mouse, trackpad, touch screen, etc. Input devices such as pads, pointing sticks, one or more mouse buttons, trackballs, joysticks, and the like. Output devices 804 may include display devices, supplemental lighting devices (eg, LEDs), tactile feedback devices (eg, vibration motors), and the like. Such display devices can include, but are not limited to, liquid crystal displays (LCD), light emitting diode (LED) displays, and plasma displays. In some embodiments, the display device may be a touchscreen.

Various embodiments of the systems and techniques described herein may be digital electronic circuit systems, integrated circuit systems, application specific integrated circuits (ASICs), computer hardware, firmware, software, and/or can be realized by a combination of These various embodiments are embodied in one or more computer programs, which can be executed and/or interpreted in a programmable system including at least one programmable processor. The programmable processor may be a dedicated or general-purpose programmable processor that receives data and instructions from the storage system, at least one input device, and at least one output device, and transmits data and instructions to the storage system, It can be transmitted to the at least one input device and the at least one output device.

These computing programs (also called programs, software, software applications, or code) contain machine instructions for programmable processors, and are written in high-level process and/or object-oriented programming languages, and/or assembly/machine language. Able to implement programs. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program that is used to provide machine instructions and/or data to a programmable processor. Refers to a product, apparatus, and/or apparatus (eg, magnetic disk, optical disk, memory, programmable logic device (PLD)) and includes a machine-readable medium for receiving machine instructions, which are machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and/or data to a programmable processor.

To provide interaction with a user, the systems and techniques described herein can be implemented on a computer that includes a display device (e.g., cathode ray tube (CRT)) for displaying information to the user. ) or LCD (liquid crystal display) monitor), and a keyboard and pointing device (e.g., mouse or trackball) through which a user can provide input to the computer. Other types of devices can also provide interaction with a user, e.g., the feedback provided to the user can be any form of sensing feedback (e.g., visual, auditory, or tactile feedback). may receive input from the user in any form (including acoustic, speech, and tactile input).

The systems and techniques described herein may be computing systems that include back-end components (e.g., data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include front-end components. A system (e.g., a user computer having a graphical user interface or web browser, through which users interact with embodiments of the systems and techniques described herein), or such a backend component , middleware components, and front-end components in any combination. The components of the system can be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server is created by computer programs running on corresponding computers and having a client-server relationship to each other.

According to the technical solution of the embodiments of the present application, a trained pre-trained model is obtained, wherein the pre-trained model is such that the image feature output from the trained pre-trained model is the first image feature distance is trained using multiple frames of training images to satisfy the minimum difference between Generate a corresponding image processing model, improve the generation efficiency of the image processing model corresponding to the target processing task, use the generated image processing model to perform the target image processing task on the target image, and perform the image Improving the effectiveness and efficiency of image processing because the processing model corresponds to the target image processing task.

It should be noted that this electronic device can also implement the training method of the pre-trained model of the present application, the principle is the same, and the description is omitted here.

It should be appreciated that steps may be reordered, added, or deleted using the various forms of flow shown above. For example, each step described in this application may be performed in parallel, sequentially, or in a different order, but the technology disclosed in this application There is no limitation herein as long as the desired result of the scheme can be achieved.

The above specific embodiments do not limit the protection scope of the present application. Those skilled in the art can make various modifications, combinations, subcombinations, and substitutions depending on design requirements and other factors. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall all fall within the protection scope of this application.

Claims (17)

An image processing method comprising:
obtaining a trained pre-trained model, wherein the image feature output from the trained pre-trained model is a first image feature distance and a second image feature distance; trained using a plurality of frames of training images to satisfy a minimum difference, wherein said first image feature distance is the distance between image features of training images extracted from the same video clip; , wherein the second image feature distance is a distance between image features of training images extracted from different video clips;
generating an image processing model that performs a target image processing task based on the pre-trained model;
using the image processing model to perform a target image processing task on the target image ;
the number of frames of training images extracted from each said video clip is the same;
An image processing method characterized by: generating an image processing model that performs a target image processing task based on the pre-trained model,
obtaining a network layer corresponding to the target image processing task;
splicing the pre-trained model and the network layer, wherein the input of the network layer is the image features output from the pre-trained model, and the output of the network layer is the image feature of the target image task; a step that is a processing result;
training the spliced pre-trained model and the network layers using the training set for the target image processing task to obtain the image processing model;
2. The image processing method according to claim 1, wherein: the target image processing task comprises an image classification task, a target detection task, or an object recognition task;
3. The image processing method according to claim 1, wherein: A method of training a pre-trained model, comprising:
obtaining a plurality of video clips;
extracting multiple frames of training images from the plurality of video clips to obtain a training set, extracting at least two frames of the training images from each of the video clips;
performing multi-round training on a pre-trained model for image feature extraction using the training set;
each round of training comprises selecting from said training set each training image extracted from at least two video clips; inputting each said training image selected in this round into said pre-trained model; obtaining the output image features, and based on the image features of each said training image selected in this round, determining a first image feature distance between training images belonging to the same video clip, different video clips; and determining the first image feature distance and the second image feature distance based on the first image feature distance and the second image feature distance adjusting the model parameters of the pre-trained model such that the difference between
the number of frames of training images extracted from each said video clip is the same ;
A training method for a pre-trained model, characterized by: determining a first image feature distance between training images belonging to the same video clip;
for the training images input to the pre-trained model in this round of training, determining intra-class feature distances between image features of different training images belonging to the same video clip;
determining the sum of the within-class feature distances for at least two video clips selected from the training set in this round of training to obtain the first image feature distance;
5. The training method according to claim 4, characterized in that: determining a second image feature distance between training images belonging to said different video clips;
for the training images input to the pre-trained model in this round of training, determining inter-class feature distances between image features of different training images belonging to different video clips;
determining the sum of the inter-class feature distances for at least two video clips selected from the training set in this round of training to obtain the second image feature distance;
5. The training method according to claim 4, characterized in that: Obtaining the plurality of video clips comprises:
obtaining a plurality of videos;
performing a splitting process based on content differences between adjacent image frames in each said video to obtain a plurality of video clips of each said video;
The training method according to any one of claims 4 to 6, characterized in that: An image processing device,
An acquisition module for acquiring a trained pre-trained model, wherein the image features output from the trained pre-trained model are a first image feature distance and a second image feature trained using a plurality of frames of training images such that the first image feature distance is the distance between the image features of the training images extracted from the same video clip. distance, wherein the second image feature distance is a distance between image features of training images extracted from different video clips;
a generation module for generating an image processing model that performs a target image processing task based on the pre-trained model;
a processing module for performing a target image processing task on the target image using the image processing model ;
the number of frames of training images extracted from each said video clip is the same ;
An image processing apparatus characterized by: The generation module is
obtaining a network layer corresponding to the target image processing task;
splicing the pre-trained model and a network layer, wherein the input of the network layer is the image feature output from the pre-trained model, the output of the network layer is the processing result of the target image task;
training the spliced pre-trained model and the network layer using the training set for the target image processing task to obtain the image processing model;
9. The image processing apparatus according to claim 8 , characterized by: the target image processing task comprises an image classification task, a target detection task, or an object recognition task;
10. The image processing apparatus according to claim 8 , wherein: A training device for a pre-trained model, comprising:
an acquisition module for acquiring multiple video clips;
an extraction module for extracting a plurality of frames of training images from the plurality of video clips to obtain a training set, the extraction module extracting at least two frames of the training images from each of the video clips;
a training module for performing multi-round training on a pre-trained model for image feature extraction using the training set;
each round of training comprises selecting from said training set each training image extracted from at least two video clips; inputting each said training image selected in this round into said pre-trained model; obtaining the output image features, and based on the image features of each said training image selected in this round, determining a first image feature distance between training images belonging to the same video clip, different video clips; determining a second image feature distance between training images belonging to the first image feature distance and the second image feature distance based on the first image feature distance and the second image feature distance adjusting the model parameters of the pre-trained model such that the difference between
the number of frames of training images extracted from each said video clip is the same ;
A pre-trained model training device characterized by: The training module comprises:
for the training images input to said pre-trained model in this round of training, determining intra-class feature distances between image features of different training images belonging to the same video clip;
determining the sum of the within-class feature distances for at least two video clips selected from the training set in this round of training to obtain the first image feature distance;
The training device according to claim 11 , characterized in that: The training module comprises:
for training images input to said pre-trained model in this round of training, determining inter-class feature distances between image features of different training images belonging to different video clips;
determining the sum of the inter-class feature distances for at least two video clips selected from the training set in this round of training to obtain the second image feature distance;
The training device according to claim 11 , characterized in that: the acquisition module,
obtaining multiple videos and performing a splitting process based on content differences between adjacent image frames in each said video to obtain multiple video clips of each said video;
The training device according to any one of claims 11 to 13 , characterized in that: at least one processor;
a memory communicatively coupled to the at least one processor;
The memory stores instructions executable by the at least one processor, and the instructions are stored in the image processing method according to any one of claims 1 to 3, or claims 4 to 7 . performed by the at least one processor to enable execution of the pre-trained model training method of any of
An electronic device characterized by: A non-transitory computer-readable storage medium having computer instructions stored thereon,
The computer instructions cause a computer to perform the image processing method according to any one of claims 1-3 or the training method of a pre-trained model according to any one of claims 4-7 ,
A non-transitory computer-readable storage medium characterized by: A computer program,
The computer program causes a computer to execute the image processing method according to any one of claims 1 to 3 or the pre-trained model training method according to any one of claims 4 to 7 ,
A computer program characterized by:

Images