CN116109966A - A video large-scale model construction method for remote sensing scenes - Google Patents

Abstract

The application relates to the technical field of computer model construction, in particular to a remote sensing scene-oriented video large model construction method. The method comprises the following steps: acquiring a remote sensing image set A and a target video set B, wherein A= { a ₁ ,a ₂ ,…,a _N }，a _n The method comprises the steps that N is the number of the remote sensing images in A, wherein the value range of N is 1 to N, and N is the number of the remote sensing images in A; b= { B ₁ ,b ₂ ,…,b _M }，b _m Is the B thM target videos, wherein the value range of M is 1 to M, M is the number of target videos in B, B _m ＝(b _m,1 ,b _m,2 ,…,b _m,Q )，b _m,q B is _m A q-th frame target image; training a neural network model using a and B, the neural network model comprising a first neural network sub-model and a second neural network sub-model. The invention constructs the remote sensing scene-oriented video large model with strong feature extraction capability and feature rule discovery capability.

Description

A video large model construction method for remote sensing scenes

technical field

The invention relates to the technical field of computer model building, in particular to a remote sensing scene-oriented large video model building method.

Background technique

Since the remote sensing video has dual characteristics in time and space, and the remote sensing scene itself has a complex texture background, the model required for the video interpretation task in the remote sensing scene needs to have strong feature extraction capabilities, and at the same time, it is necessary to explore the space of the video. Characteristic regularity and temporal characteristic regularity. How to construct a large video model for remote sensing scenes with strong feature extraction ability and feature rule discovery ability is an urgent problem to be solved.

Contents of the invention

The purpose of the present invention is to provide a method for constructing a large video model for remote sensing scenes, and construct a large video model for remote sensing scenes with strong feature extraction capabilities and feature rule discovery capabilities.

According to the present invention, there is provided a method for building a large video model for remote sensing scenes, comprising the following steps:

Obtain remote sensing image set A and target video set B, A={a ₁ ,a ₂ ,…,a _N }, a _n is the nth frame of remote sensing image in A, the value range of n is from 1 to N, and N is A The number of remote sensing images; B={b ₁ ,b ₂ ,…,b _M }, b _m is the mth target video in B, the value of m ranges from 1 to M, and M is the number of target videos in B , b _m ＝(b _m,1 ,b _m,2 ,…,b _m,Q ), b _m,q is the target image of the qth frame in b _m , the value range of q is from 1 to Q, and Q is the target The number of target images in the video, b _m,1 , b _m,2 ,..., b _m,Q are Q frames of target images shot continuously; the target video in B is the video taken by the satellite-equipped remote sensing equipment or the UAV-equipped remote sensing The video taken by the equipment, the remote sensing image is the image taken by the remote sensing equipment carried by the satellite.

Utilize A and B to train the neural network model, the neural network model includes the first neural network sub-model and the second neural network sub-model, and the training process includes:

Traverse A, block a _n , and randomly mask k*C blocks in a _n ; C is the number of blocks obtained by dividing a _n into blocks, and k is the preset mask ratio; using the mask The code-processed a _n trains the first neural network sub-model, the first neural network sub-model is a 2D swin-transformer structure, and the first neural network sub-model includes a first encoder and a first decoder.

Traversing B, masking the [i _m ,i _m +L]th frame image in b _m , i _m +L≤Q, i _m ≥1, L is the number of preset mask frames, and i _m is b The initial mask frame in _m ; the second neural network sub-model is trained by b _m after mask processing, and the second sub-model is a 3D swin-transformer structure, and the second neural network sub-model includes the second neural network sub-model Two encoders and a second decoder; the training of the first neural network sub-model is performed simultaneously with the training of the second neural network sub-model, and the second encoder and the first encoder are training There is weight sharing in the process.

Compared with the prior art, the present invention has obvious beneficial effects. By means of the above-mentioned technical solutions, the method provided by the present invention can achieve considerable technical advancement and practicality, and has wide industrial application value. It has at least the following benefits Effect:

The remote sensing scene-oriented large video model of the present invention includes two branches, the first branch corresponds to the first neural network sub-model, and the training samples corresponding to the branch are remote sensing image sets; the second branch corresponds to the second neural network sub-model. The network sub-model, the training sample corresponding to this branch is a target video set, and the target video set of the present invention not only includes remote sensing video (that is, the video taken by the satellite-mounted remote sensing equipment), but also includes UAV video (the unmanned aerial vehicle is equipped with remote sensing equipment). shot video), because the remote sensing video is not easy to obtain, so the number of remote sensing videos that can be used as training samples is relatively small; the present invention expands the number of video samples by introducing drone videos, and utilizes the expanded video samples for the second The training of the neural network sub-model can improve the feature extraction and rule discovery capabilities of the second neural network sub-model, and also improve the generalization ability of the trained second neural network sub-model, which can be applied to the downstream of different partial space-time predictions Task.

Moreover, the masking strategy adopted by the present invention for the remote sensing image samples corresponding to the first neural network sub-model is to randomly mask a part of the pixels, and the ability of the first neural network model to extract the spatial information of the remote sensing image is improved through the random masking strategy ; The masking strategy adopted for the target video sample corresponding to the second neural network sub-model is to use a certain frame in the target video as the starting frame, and mask all the fixed-length frames after the starting frame. Masking strategy to increase the difficulty of video prediction, improve the ability of the second neural network sub-model to extract the temporal and spatial continuous information of objects in the video; the present invention is similar to the training process of the first neural network sub-model The training process is carried out at the same time, which speeds up the training process of the large video model, and there is weight sharing between the first encoder in the first neural network sub-model and the second encoder in the second neural network sub-model during the training process, Thus, the second neural network sub-model can obtain the ability of the first neural network sub-model to extract the spatial information of remote sensing images, thereby improving the ability of the second neural network sub-model itself to extract the spatial information of remote sensing images, which is conducive to speeding up The training process of the second neural network sub-model.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained based on these drawings without creative effort.

FIG. 1 is a flowchart of a method for constructing a large video model for a remote sensing scene provided by an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts fall within the protection scope of the present invention.

According to the present invention, there is provided a kind of method for building a large video model for remote sensing scenes, as shown in Figure 1, comprising the following steps:

S100, acquire remote sensing image set A and target video set B, A={a ₁ ,a ₂ ,...,a _N }, a _n is the nth frame of remote sensing image in A, and the value range of n is from 1 to N, N is the number of remote sensing images in A; B={b ₁ ,b ₂ ,…,b _M }, b _m is the mth target video in B, the value range of m is from 1 to M, and M is the target video in B b _m ＝(b _m,1 ,b _m,2 ,…,b _m,Q ), b _m,q is the target image of the qth frame in b _m , the value range of q is from 1 to Q, and Q is the number of target images in the target video, b _m,1 , b _m,2 ,..., b _m,Q are Q frames of target images captured continuously; the target video in B is the video taken by the satellite equipped with remote sensing equipment or the UAV A video taken with remote sensing equipment, and the remote sensing image is an image taken by a satellite equipped with remote sensing equipment.

The remote sensing scene-oriented large video model of the present invention includes two branches, the first branch corresponds to the first neural network sub-model, and the training samples corresponding to the branch are remote sensing image sets; the second branch corresponds to the second neural network sub-model. The network sub-model, the training sample corresponding to this branch is a target video set, and the target video set of the present invention not only includes remote sensing video (that is, the video taken by the satellite-mounted remote sensing equipment), but also includes UAV video (the unmanned aerial vehicle is equipped with remote sensing equipment). captured video).

Preferably, the number of videos taken by remote sensing equipment carried by drones in B is greater than the number of videos taken by remote sensing equipment carried by satellites in B. In the present invention, the video taken by the remote sensing equipment carried by the UAV is used as a kind of target video, and the number of target videos can be expanded to solve the problem that the number of target videos is not enough to meet the subsequent training requirements of the neural network model due to the difficulty in obtaining remote sensing videos. Question; moreover, the video taken by the UAV equipped with remote sensing equipment and the video taken by the satellite equipped with remote sensing equipment are all taken from the angle of view of the aerial remote sensing equipment, so the video taken by the UAV equipped with remote sensing equipment is taken as the target video The subsequent training of the neural network model can also take into account the effect of training the neural network model.

Preferably, both N and M are on the order of millions. The number of training samples in the present invention is one million, and the trained large video model for remote sensing scenes has powerful feature extraction capabilities, rule discovery capabilities, and generalization capabilities. The trained large video model for remote sensing scenes The model parameters are used as the initial model parameters of the models corresponding to different downstream tasks, which can speed up the training process of the models corresponding to the downstream tasks and improve the accuracy of the models corresponding to the downstream tasks; the above downstream tasks can be video prediction tasks, target detection tasks, single target tracking tasks and video segmentation tasks, etc.

S200, using A and B to train the neural network model, the neural network model includes a first neural network sub-model and a second neural network sub-model, and the training process includes:

S210, traversing A, performing block processing on a _n , and randomly performing mask processing on k*C blocks in a _n ; C is the number of blocks obtained by performing block a _n , and k is a preset mask ratio; Using masked a _n to train the first neural network submodel, the first neural network submodel is a 2D swin-transformer structure, and the first neural network submodel includes a first encoder and a first decoder device.

The structure of the 2D swin-transformer in the present invention is the prior art, and will not be repeated here. In the present invention, the function of the first encoder is to extract the features of the masked a _n , and the function of the first decoder is to predict the original pixel value corresponding to the mask block according to the output of the first encoder.

The masking strategy adopted by the present invention for the remote sensing image samples corresponding to the first neural network sub-model is to randomly mask a part of pixels, and the ability of the first neural network model to extract the spatial information of the remote sensing image is improved through the random masking strategy. Preferably, 40%≤k≤60%. Small-scale experiments show that when the value of k is set in the range of 40%≤k≤60%, the first neural network sub-model can not only extract the spatial information of remote sensing images better, but also take into account the training of the first neural network sub-model duration. Optionally, k=50%.

As an embodiment, a _n is an image with a resolution of 224*224, and a _n is divided into blocks to obtain 56*56 blocks, and each block has 4*4=16 pixels; randomly extract 56* Half of the 56 blocks are masked out to obtain the masked a _n .

S220 traverses B, and performs mask processing on the [i _m , i _m +L]th frame image in b _m , where i _m + L ≤ Q, i _m ≥ 1, L is the number of preset mask frames, and i _m is The initial mask frame in b _m ; the b _m after mask processing is used to train the second neural network submodel, the second submodel is a 3D swin-transformer structure, and the second neural network submodel includes A second encoder and a second decoder; the training of the first neural network sub-model is performed simultaneously with the training of the second neural network sub-model, and the second encoder and the first encoder are at the same time There is weight sharing during training.

The biggest difference between the 3D swin-transformer in the present invention and the 2D swin-transformer is that it has changed from 2D to 3D, and has one more dimension. The structure of the 3D swin-transformer is also an existing technology, and will not be described here. In the present invention, the function of the second encoder is to extract the features of the masked b _m , and the function of the second decoder is to predict the masked target image according to the output of the second encoder.

In the present invention, the training process of the first neural network sub-model is carried out simultaneously with the training process of the second neural network sub-model, which speeds up the training process of the large video model, and the first neural network sub-model in the training process There is weight sharing between the encoder and the second encoder in the second neural network sub-model, so that the weights corresponding to the modules with the same structure in the second encoder and the first encoder are the same, such as the attention in the second encoder The (attention) module has the same weight as the attention module in the first encoder. Thus, the second neural network sub-model can obtain the ability of the first neural network sub-model to extract the spatial information of remote sensing images, thereby improving the ability of the second neural network sub-model itself to extract the spatial information of remote sensing images, which is conducive to speeding up The training process of the second neural network sub-model.

The masking strategy adopted by the present invention for the target video sample corresponding to the second neural network sub-model is to use a certain frame in the target video as the starting frame, and mask all the fixed-length frames after the starting frame. The masking strategy is used to increase the difficulty of video prediction and improve the ability of the second neural network sub-model to extract the temporal and spatial continuous information of objects in the video.

Preferably, Q=16, 5≤L≤9. Small-scale experiments show that when Q=16 and the value of L is set in the range of 5≤L≤9, the second neural network sub-model can not only extract the temporal and spatial continuous information of objects in the video, but also take into account the second The training time of the neural network submodel. Optionally, L=7.

The present invention adopts a random continuous frame mask strategy for b _m , that is, the initial mask frames corresponding to different target videos may be different or the same, but the number of masked frames is equal. As an embodiment, b _m includes 16 frames of target images shot continuously, each frame is a 224*224 image, the number of mask frames is preset to 7, and a starting point is randomly selected in the 16 frames of target images, and then Mask all the starting point and the subsequent 7 frames of images to obtain the masked b _m . It should be understood that the selection of the starting point should ensure that there are 7 or more images after the starting point.

According to the present invention, the trained neural network model is the remote sensing scene-oriented video large model of the present invention, and the remote sensing scene-oriented video large model has strong feature extraction capabilities and feature rule discovery capabilities.

As a specific implementation, the remote sensing image set A includes more than 1.09 million remote sensing images, the target video set B includes more than 1.01 million target videos, and more than half of the target videos in B are videos taken by drones equipped with remote sensing equipment ; The above-mentioned remote sensing images are processed in blocks, and half of the blocks in the remote sensing images are randomly masked; each target video is set to include continuous 16 frames of target images, and the initial mask frame in the target video is randomly selected, and the The initial mask frame and the following 7 frames of target images are masked; the remote sensing image after mask processing is used to train the first neural network sub-model in the neural network model, and the target video after mask processing is used to train the neural network model. The second neural network sub-model in the network model is trained, and the encoder in the first neural network sub-model is weight-shared with the encoder in the second neural network sub-model during the training until the training ends.

Experiments show that compared with random initialization of model parameters, the model parameters of the trained neural network model are used as the initial model parameters of the models corresponding to different downstream tasks, and the accuracy of the models corresponding to the downstream tasks with the same training time is higher. : When the downstream task is a target detection task, the corresponding mean average precision (mAP) index rises from 0.3629 to 0.3718; when the downstream task is a video prediction task, the corresponding structural similarity (SSIM) index rises from 0.7018 to 0.7152. It can be seen that the large video model for remote sensing scenes constructed by the present invention is suitable for different downstream tasks, has strong generalization ability, and the corresponding feature extraction ability and feature law discovery ability are strong, which can improve the accuracy of models corresponding to different downstream tasks.

Although some specific embodiments of the present invention have been described in detail through examples, those skilled in the art should understand that the above examples are for illustration only, rather than limiting the scope of the present invention. Those skilled in the art will also appreciate that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.

Claims (7)

1. The method for constructing the large video model for the remote sensing scene is characterized by comprising the following steps of:

acquiring a remote sensing image set A and a target video set B, wherein A= { a ₁ ,a ₂ ,…,a _N }，a _n The method comprises the steps that N is the number of the remote sensing images in A, wherein the value range of N is 1 to N, and N is the number of the remote sensing images in A; b= { B ₁ ,b ₂ ,…,b _M }，b _m For the mth target video in B, the value range of M is 1 to M, M is the number of the target videos in B, B _m ＝(b _m,1 ,b _m,2 ,…,b _m,Q )，b _m,q B is _m In the Q-th frame of target image, the value range of Q is 1 to Q, Q is the number of target images in the target video, b _m,1 、b _m,2 、…、b _m,Q Q frames of target images are continuously shot; b, the target video is a video shot by a satellite carried remote sensing device or a remote carried by an unmanned aerial vehicleThe remote sensing image is an image shot by the satellite carried remote sensing equipment;

training a neural network model using a and B, the neural network model comprising a first neural network sub-model and a second neural network sub-model, the training comprising:

traversing A, pair a _n Performing block processing, and randomly performing block processing on the a _n The k x C blocks in (a) are subjected to mask processing; c is a pair a _n The number of blocks obtained by partitioning is k, which is a preset mask proportion; a processed by mask _n Training a first neural network sub-model, the first neural network sub-model being of a 2D swin-transformer structure, the first neural network sub-model comprising a first encoder and a first decoder;

traversal B, pair B _m I of [ i ] _m ,i _m +L]Masking the frame image, i _m +L≤Q，i _m Not less than 1, L is the number of preset mask frames, i _m B is _m A start mask frame of (a); b processed by mask _m Training a second neural network sub-model, the second sub-model being a 3D swin-transformer structure, the second neural network sub-model comprising a second encoder and a second decoder; the training of the first neural network sub-model is performed simultaneously with the training of the second neural network sub-model, and the second encoder and the first encoder have weight sharing in the training process.

2. The method for constructing a large video model for a remote sensing scene according to claim 1, wherein k is more than or equal to 40% and less than or equal to 60%.

3. The method for constructing a large video model for a remote sensing scene according to claim 2, wherein k=50%.

4. The method for constructing a large video model for a remote sensing scene according to claim 1, wherein q=16, and 5.ltoreq.l.ltoreq.9.

5. The method for constructing a large video model for a remote sensing scene as claimed in claim 4, wherein l=7.

6. The method for constructing the large video model for the remote sensing scene according to claim 1, wherein the number of videos shot by the remote sensing equipment carried by the unmanned aerial vehicle in the B is larger than the number of videos shot by the remote sensing equipment carried by the satellite in the B.

7. The method for constructing a large video model for a remote sensing scene as claimed in claim 1, wherein N and M are each in the order of millions.

Images