CN117097853A - Real-time image matting method and system based on deep learning - Google Patents

Abstract

The invention provides a real-time image matting method and a system based on deep learning, wherein the method comprises the following steps: s1: acquiring a matting data set; s2: constructing a matting network model based on a ViT and CNN mixed structure; s3: training the model by using the data set, and correcting the model by using the loss function to obtain a trained model; s4: and inputting the video frame obtained by the image file to be scratched or the camera and the cyclic feature image from the last moment into a training completed model, and acquiring the scratched alpha image and the cyclic feature image at the moment in real time. Aiming at the problem of unstable image matting results under a complex background, the invention integrates a self-attention mechanism to strengthen the global information extraction capability, reduces the possibility of semantic misjudgment of foreground and background pixels, and ensures the precision of the image matting results. Meanwhile, the invention can process video data in real time without additional constraint, has low use cost and can be used for various non-professional scenes.

Description

A real-time keying method and system based on deep learning

Technical field

The invention belongs to the field of image processing technology, in particular to computer vision technology, and specifically to a real-time matting method and system based on deep learning.

Background technique

Keying is a popular technology in the field of computer vision. It can effectively separate foreground objects that people are interested in from pictures or videos. It is widely used in various commercial fields such as live television, movie special effects, and advertising. The mathematical model of this technology is shown in formula (1):

I＝αF+(1-α)B (1)

Among them, I is a given picture or video frame, F is the foreground image, B is the background image, and α is the alpha map, which is the opacity of the foreground image pixels. In the case of only known quantity I, the values of the other three unknown quantities cannot be obtained through this formula, so the problem is an under-constrained problem.

Traditional keying methods are based on sampling and propagation, and artificially add constraints to formula (1) by assuming that the colors of different pixels in the image have a certain functional relationship. This type of method does not make full use of the contextual information of the image. When the foreground pixels and background pixels are similar in color, misjudgments are prone to occur. The keying accuracy is low and the results are unstable.

Due to various defects in traditional methods, two deep learning-based methods are currently used for natural background matting: the matting method based on Convolution Neural Network (CNN) and the matting method based on VisionTransformer (ViT). Among them, the CNN-based method constructs a convolutional neural network model by using convolution layers, pooling layers, and activation layers, which is a more traditional method in deep learning; the ViT-based method uses the ViT module to construct a neural network model, which constructs The model can be a pure ViT structure or a hybrid structure of CNN and ViT. ViT has a self-attention mechanism that can capture the correlation of long-range pixels in the image and model the global information of the image. It has higher accuracy than CNN and is an emerging technology in the field of computer vision.

After searching the existing literature, the following relevant literature was found:

Robust Video Matting (RVM) method (Lin Shanchuan, Yang Linjie, Saleemi I, et al. "Robust High-Resolution Video Matting with Temporal Guidance (High-resolution stable video matting based on time guidance)". Proc of IEEE/CVF Winter Conference onApplications of Computer Vision.Waikoloa, HI, USA: IEEE Press, 2022:3132-3141), using MobileNet as the backbone network to achieve real-time keying of natural backgrounds. This method has high real-time performance and good keying accuracy on simple backgrounds. However, because this method still uses the traditional CNN structure and has weak global information processing capabilities, it is easy to confuse foreground and background pixels in complex backgrounds.

VMFormer (Video Matting with Transformer) method (Li Jiachen, Goel V, Ohanyan M, et al.

"VMFormer: End-to-End Video Matting with Transformer (End-to-End Video Matting based on Transformer)", https://arxiv.org/abs/2208.12801), which improves the shortcomings of previous CNN structures in image processing, Vision Transformer is introduced to implement image feature extraction and feature map decoding. This method uses a large number of ordinary Vision Transformer structures in both the encoder and decoder, resulting in a large number of network model parameters, which is about 2 times that of the RVM method. Experiments show that the network model proposed by the VMFormer method can only process 1080p images at a speed of 3 frames per second on the Nvidia GeForce RTX 4060 GPU, making it difficult to achieve real-time processing of images. Currently, this method is only suitable for processing ready-made video files and cannot be applied to fields such as live broadcasts. It is subject to certain limitations in usage scenarios.

Contents of the invention

Aiming at the real-time matting task of natural backgrounds, the existing CNN model has poor accuracy when processing complex backgrounds. The present invention proposes a method that uses the self-attention mechanism in the ViT model to strengthen global relationship modeling and reduce image pixel semantic recognition errors. frequency, and then achieve a technical solution for high-resolution and high-precision video keying while ensuring real-time performance.

In order to solve the above problems, the technical solutions adopted by the present invention are as follows:

A real-time keying method based on deep learning, including the following steps:

S1: Obtain the keying data set;

S2: Construct a matting network model based on the hybrid structure of ViT and CNN;

S3: Use the data set to train the model, and correct it through the loss function to obtain the trained model;

S4: Input the image file to be matted or the video frame obtained by the camera and the cyclic feature map from the previous moment into the trained model, and obtain the matting alpha map and the cyclic feature map at this moment in real time.

Preferably, in step S1, the keying data set specifically includes a video keying foreground data set, a video background data set, a picture keying foreground data set, a picture background data set and a portrait segmentation data set, all of which are 360p, 720p or 1080p images. Among them, the keyed foreground data set contains foreground images and corresponding alpha maps.

Preferably, in step S2, when constructing the matting network model based on the hybrid structure of ViT and CNN, the following method is used:

S2.1: Construct the original image resampling sub-network, down-sample the higher-resolution original image and then send it to the encoder sub-network for processing, and restore the low-resolution alpha image generated by the decoder network to the original resolution alpha picture;

S2.2: Construct a feature extraction encoder subnetwork based on the hybrid structure of ViT and CNN to extract multi-level features from the downsampled original image;

S2.3: Construct a bottleneck block subnetwork to connect the encoder and decoder subnetworks;

S2.4: Construct a recurrent decoder subnetwork based on attention and content awareness, perform spatiotemporal modeling of feature maps, and generate low-resolution alpha maps.

Preferably, in step S2.1, construct the original image resampling subnetwork, which specifically includes the following steps:

S2.1.1: Downsample the high-resolution original image F1.1 through the average pooling operation to obtain the low-resolution original image F1.2, and send it to the encoder subnetwork described in step S2.2;

S2.1.2: Splice the low-resolution alpha image F1.3 generated by the encoder subnetwork described in step S2.4 with the high-resolution original image F1.1 described in step S2.1.1, and input it into the depth-guided filter (Deep Guided Filter, DGF), thereby restoring the alpha image F1.4 of the original resolution.

Preferably, in step S2.2, a feature extraction encoder subnetwork based on the hybrid structure of ViT and CNN is constructed, specifically:

Three Mobile ViT V3 modules are embedded into the 17 inverted residual blocks of MobileNet V3 Large to form an encoder subnetwork, and three skip connection feature maps F2.1, F2.2, and F2 are derived from the subnetwork. 3. The end of the encoder network outputs feature map F2.4.

Preferably, in step S2.3, a bottleneck block subnetwork is constructed, specifically:

Use the Convolutional Block Attention Module (CBAM), LR-ASPP, Conv-GRU and Content-Aware Reassembly of FEatures (CARAFE) upsampling operators to connect in turn to form bottleneck blocks. network. This sub-network accepts feature map F2.4 as input and outputs feature map F3.

Preferably, in step S2.4, a recurrent decoder subnetwork based on attention and content awareness is constructed. The specific module structure is:

Construct three encoder modules D1, D2, and D3. The construction method of each module is: use the convolution layer, normalization layer, activation layer, Conv-GRU and CARAFE upsampling operator to connect back and forth to combine into a decoder module.

Preferably, in step S2.4, a recurrent decoder subnetwork based on attention and content awareness is constructed. The specific steps are:

S2.4.1: The low-resolution original image F1.2 is downsampled 8 times to obtain the image F4.1.1, and the skip-connected feature map F2.1 is passed through a layer of CBAM to obtain the feature map F4.1.2, and then F4.1.1, F4.1.2 and F3 are sent to the decoder module D1 to obtain the output feature map F4.1.3;

S2.4.2: The low-resolution original image F1.2 is downsampled 4 times to obtain the image F4.2.1, and the skip-connected feature map F2.2 is passed through a layer of CBAM to obtain the feature map F4.2.2, and then F4.2.1, F4.2.2 and F4.1.3 are sent to the decoder module D2 to obtain the output feature map F4.2.3;

S2.4.3: The low-resolution original image F1.2 is downsampled by 2 times to obtain the image F4.3.1, and the skip-connected feature map F2.3 is passed through a layer of CBAM to obtain the feature map F4.3.2, and then F4.3.1, F4.3.2 and F4.2.3 are sent to the decoder module D3 to obtain the output feature map F4.3.3;

S2.4.4: Splice the low-resolution original image F1.2 and the feature map F4.3.3, and send them to the module processing composed of two sets of convolution layers, normalization layers, and activation layers to obtain the low-resolution alpha image F1.3.

Preferably, in step S3, the loss function used is specifically: the sum of L1 loss, Laplacian pyramid loss and temporal coherence loss. The calculation formula is as follows:

Among them, α _t is the real alpha map at time t, Alpha map predicted for time t,/> Represents the value of the alpha map at the i-th level of the Laplacian pyramid.

Preferably, in step S4, the inputs of the model are: the original image F1.1 and the cyclic feature maps T1.1, T1.2, and T1.3 output from the Conv-GRU layer at the previous moment, where the cyclic feature maps This is an optional input and is not necessary when processing a single image.

Preferably, in step S4, the output of the model is specifically: the predicted alpha map F1.4 and the cyclic feature maps T2.1, T2.2, and T2.3 output by the Conv-GRU layer at this time.

The invention also discloses a real-time matting system based on deep learning for executing the above method, which includes the following modules:

Data set acquisition module: obtains the keying data set, including video keying foreground data set, video background data set, picture keying foreground data set, picture background data set and portrait segmentation data set;

Network model building module: Construct a matting network model based on the hybrid structure of ViT and CNN;

Model training module: Use the data set to train the matting network model, and correct it through the loss function to obtain the trained model;

Keying alpha map acquisition module: Input the image file to be keyed or the video frame obtained by the camera and the cycle feature map from the previous moment into the trained model, and obtain the keying alpha map and the cycle feature map at this moment in real time.

Aiming at the problem of unstable matting results under complex backgrounds, this invention incorporates a self-attention mechanism to enhance global information extraction capabilities, reduce the possibility of semantic misjudgments of foreground and background pixels, and improve the accuracy of matting results; at the same time, this invention The invention can process video data in real time without additional constraints, has low cost of use, and can be used in a variety of non-professional scenarios.

Description of the drawings

The present invention will be further described below in conjunction with the accompanying drawings and examples:

Figure 1 is a flow chart of a real-time keying method based on deep learning according to a preferred embodiment of the present invention;

Figure 2 is the network framework of the present invention;

Figure 3 is a structural diagram of the encoder subnetwork of the present invention;

Figure 4 is a structural diagram of the decoder subnetwork of the present invention;

Figure 5 is a block diagram of a real-time matting system based on deep learning according to a preferred embodiment of the present invention.

Detailed ways

The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

Aiming at the real-time matting task of natural backgrounds and the problem of poor accuracy of ordinary CNN models when processing complex backgrounds, this invention proposes a method that uses the self-attention mechanism in the ViT model to strengthen global relationship modeling and reduce the frequency of image pixel semantic recognition errors. , and then achieve a technical solution for high-resolution and high-precision video keying while ensuring real-time performance.

As shown in Figure 1-4, this embodiment is a real-time keying method based on deep learning, which specifically includes the following steps:

S1: Obtain the keying data set, including the foreground, the alpha image of the foreground and the background, and divide the data set into a training set, a verification set and a test set;

S2: Construct a matting network model based on the hybrid structure of ViT and CNN;

S3: Use the data set to train the model in step S2, and correct it through the loss function to obtain the trained model;

S4: Input the image file to be matted or the video frame obtained by the camera and the cyclic feature map from the previous moment into the trained model, and obtain the matting alpha map and the cyclic feature map at this moment in real time.

Each step is detailed below.

In step S1, the foreground data set for image and video keying consists of the foreground image and the corresponding real alpha map. The video matting data set used in this embodiment is Video Matte 240K; the image matting data set is AIM-500, Adobe Image Matting Datase, Distinctions-646, PPM-100 and P3M-10K; the background data set is DVM, Indoor CVPR 09; The portrait segmentation data sets are COCO, Supervisely Person Dataset and YoutubeVIS2021.

Since the ViT structure relies more on data enhancement of the data set to break through the limitation of no inductive bias characteristics, the data enhancement operations performed on the image in this embodiment mainly include the following:

(1) Rotation: Rotate the image 90 degrees, 180 degrees or 270 degrees according to its own center;

(2) Translation: Translate the foreground of the image away from its original position;

(3) Stretching: Stretch the image at an angle;

(4) Zoom: Reduce or enlarge the foreground of the image by random multiples;

(5) Cropping: Crop out part of the foreground of the image;

(6) Color change: Convert the image from original color to grayscale;

(7) Noise: Randomly add noise of varying densities to the image;

(8) Truncation: For the video matting data set, crop a part of the complete video clip at a random length;

(9) Reverse playback: For the video matting data set, reverse the order of the video frame sequence for training;

(10) Frame extraction: For the video matting data set, in a video clip, a video frame is deleted at certain intervals.

In step S2, a matting network model based on the hybrid structure of ViT and CNN is constructed, which also includes the following sub-steps:

S2.1: Construct the original image resampling sub-network, down-sample the higher-resolution original image and then send it to the encoder sub-network for processing, and restore the low-resolution alpha image generated by the decoder network to the original resolution alpha picture;

S2.2: Construct a feature extraction encoder subnetwork based on the hybrid structure of ViT and CNN to extract multi-level features from the downsampled original image;

S2.3: Construct a bottleneck block subnetwork to connect the encoder and decoder subnetworks;

S2.4: Construct a recurrent decoder subnetwork based on attention and content awareness, perform spatiotemporal modeling of feature maps, and generate low-resolution alpha maps.

In step S2.1, the function of constructing the original image resampling sub-network is to process high-resolution images faster. If there is no requirement for real-time performance or the resolution of the processed image is low, this sub-network is not necessary.

In step S2.1, construct the original image resampling subnetwork, which specifically includes the following sub-steps:

S2.1.1: Downsample the high-resolution original image F1.1 through the average pooling operation to obtain the low-resolution original image F1.2, and send it to the encoder subnetwork described in step S2.2;

S2.1.2: Splice the low-resolution alpha image F1.3 generated by the encoder subnetwork described in step S2.4 with the high-resolution original image F1.1 described in step S2.1.1, and input it into the depth-guided filter (Deep Guided Filter, DGF), thereby restoring the alpha image F1.4 of the original resolution.

In step S2.2, construct a feature extraction encoder subnetwork based on the hybrid structure of ViT and CNN, specifically: embed 3 Mobile ViT V3 modules into 17 inverted residual blocks of MobileNet V3 Large to form the encoder subnetwork network.

Specifically, in step S2.2, a structure composed of 17 inverted residual blocks of MobileNet V3 Large is used, and the Mobile ViT V3 module is embedded behind the 4th, 6th and 9th inverted residual blocks of the structure. form a hybrid structure. In addition, three skip connection feature maps F2.1, F2.2, and F2.3 are derived at the positions of the second inverted residual block, the first Mobile ViT block, and the second Mobile ViT block. The encoder accepts the downsampled original image F1.2 as input and outputs the feature map F2.4.

In step S2.3, the bottleneck block subnetwork is constructed by using CBAM, LR-ASPP, Conv-GRU and CARAFE upsampling operators to connect in sequence to form the bottleneck block subnetwork.

Specifically, in step S2.3, the bottleneck block subnetwork accepts feature map F2.4 as input and outputs feature map F3. In addition, as far as the Conv-GRU layer in this structure is concerned, it accepts the cyclic feature map T1.1 of the previous moment as a constraint input, and outputs a cyclic feature map T2.1 of this moment, which is used as itself at the next moment. constraint input.

In step S2.4, construct a cyclic decoder subnetwork based on attention and content awareness, specifically: construct a total of 3 encoder modules D1, D2, and D3, connect the three before and after, and add two sets of volumes at the end The module composed of accumulation layer, normalization layer and activation layer forms a decoder sub-network.

Specifically, in step S2.4, a recurrent decoder subnetwork based on attention and content awareness is constructed. The steps are as follows:

S2.4.1: The low-resolution original image F1.2 is downsampled 8 times to obtain the image F4.1.1, and the skip-connected feature map F2.1 is passed through a layer of CBAM to obtain the feature map F4.1.2, and then F4.1.1, F4.1.2 and F3 are sent to the decoder module D1 to obtain the output feature map F4.1.3;

S2.4.2: The low-resolution original image F1.2 is downsampled 4 times to obtain the image F4.2.1, and the skip-connected feature map F2.2 is passed through a layer of CBAM to obtain the feature map F4.2.2, and then F4.2.1, F4.2.2 and F4.1.3 are sent to the decoder module D2 to obtain the output feature map F4.2.3;

S2.4.3: The low-resolution original image F1.2 is downsampled by 2 times to obtain the image F4.3.1. The skip-connected feature map F2.3 is passed through a layer of CBAM to obtain the feature map F4.3.2, and then F4.3.1, F4.3.2 and F4.2.3 are sent to the decoder module D3 to obtain the output feature map F4.3.3;

S2.4.4: Splice the low-resolution original image F1.2 and the feature map F4.3.3, and send them to the module processing composed of two sets of convolution layers, normalization layers, and activation layers to obtain the low-resolution alpha image F1.3.

Specifically, in step S2.4, for each Conv-GRU layer in the decoder modules D1, D2, and D3, the cyclic feature maps T1.2, T1.3, and T1.4 from the previous moment are respectively accepted as Constrain the input, and output the cyclic feature maps T2.2, T2.3, and T2.4 at this time, which are used for the constraint input of the Conv-GRU layer at the corresponding position at the next time.

Table 1 Ablation experiments

The ablation experiment was performed a total of 3 times. Among them, ablation model 1 eliminates the Mobile ViT module in the encoder, and ablation model 2 not only eliminates Mobile ViT, but also eliminates the CBAM and CARAFE operators in the decoder. The results show that both the Mobile ViT module in the encoder and the CBAM and CARAFE operators in the decoder have a significant impact on the accuracy of the predicted alpha map of the network model. The more complete the model mechanism, the higher the keying accuracy.

In step S3, the data set is used to train the model, specifically: first train the 720p video keying data set, then train the 1080p video keying data set, and finally train the picture keying data set. Among them, portrait segmentation training is interspersed in the above training steps, and portrait segmentation training is performed every few times of keying training.

Specifically, in step S3, the loss function is used to correct the training. The loss function used is the sum of L1 loss, Laplacian pyramid loss and temporal coherence loss. The calculation formula is as follows:

Among them, α _t is the real alpha map at time t, Alpha map predicted for time t,/> Represents the value of the alpha map at the i-th level of the Laplacian pyramid.

In step S4, the input to the model is specifically: the original image F1.1 and the cyclic feature maps T1.1, T1.2, T1.3, and T1.4 output from the Conv-GRU layer at the previous moment, where the cyclic feature Image is an optional input and is not required when processing a single image.

In step S4, the output of the model is specifically: the predicted alpha map F1.4 and the cyclic feature maps T2.1, T2.2, T2.3, and T2.4 output by the Conv-GRU layer at this time.

This model makes full use of the self-attention mechanism's ability to globally extract image features and the content-aware mechanism to improve the overall accuracy of the model. It solves the shortcoming of traditional CNN networks that are insensitive to long-range pixel relationships, and can accurately distinguish foregrounds in complex backgrounds. and the semantics of background pixels; the decoder uses attention and content-aware mechanisms to improve the accuracy of reconstructed images and make details clearer; the invention also improves the speed of processing high-resolution images through depth-guided filters. This invention has good application value in keying tasks that require both real-time performance and accuracy.

As shown in Figure 5, this embodiment discloses a real-time matting system based on deep learning for executing the above method embodiment, which includes the following modules:

Data set acquisition module: obtains the keying data set, including video keying foreground data set, video background data set, picture keying foreground data set, picture background data set and portrait segmentation data set;

Network model building module: Construct a matting network model based on the hybrid structure of ViT and CNN;

Model training module: Use the data set to train the matting network model, and correct it through the loss function to obtain the trained model;

Keying alpha map acquisition module: Input the image file to be keyed or the video frame obtained by the camera and the cycle feature map from the previous moment into the trained model, and obtain the keying alpha map and the cycle feature map at this moment in real time.

For other contents of this embodiment, please refer to the above method embodiment.

The above is only a detailed description of the preferred embodiments and principles of the present invention. For those of ordinary skill in the art, there will be changes in the specific implementation methods based on the ideas provided by the present invention, and these changes should also be made. regarded as the protection scope of the present invention.

Claims (10)

1. The real-time image matting method based on deep learning is characterized by comprising the following steps of:

s1: acquiring a matting data set;

s2: constructing a matting network model based on a ViT and CNN mixed structure;

s3: training the model in the step S2 by utilizing a data set, and correcting by a loss function to obtain a trained model;

s4: and inputting the video frame obtained by the image file to be scratched or the camera and the cyclic feature image from the last moment into a training completed model, and acquiring the scratched alpha image and the cyclic feature image at the moment in real time.

2. The method of claim 1 wherein the data set in step S1 comprises a video matting foreground data set, a video background data set, a picture matting data set, and a portrait segmentation data set.

3. The method of claim 1, wherein in step S2, a matting network model based on a ViT and CNN hybrid structure is constructed, specifically as follows:

s2.1: an original image resampling sub-network is constructed, the original image with higher resolution is subjected to downsampling and then sent to an encoder sub-network for processing, and a low resolution alpha image generated by a decoder network is restored to an original resolution alpha image;

s2.2: constructing a characteristic extraction encoder sub-network based on a ViT and CNN mixed structure, and extracting multi-level characteristics from the original image after downsampling;

s2.3: constructing a bottleneck block sub-network, and connecting an encoder sub-network and a decoder sub-network;

s2.4: a cyclic decoder sub-network based on attention and content perception is constructed, the feature map is subjected to space-time modeling, and a low-resolution alpha map is generated.

4. A method according to claim 3, characterized in that in step S2.1, an original image resampling sub-network is built, comprising in particular the steps of:

s2.1.1: downsampling the high-resolution original image F1.1 through an average pooling operation to obtain a low-resolution original image F1.2, and sending the low-resolution original image F1.2 into the encoder subnetwork in the step S2.2;

s2.1.2: the low resolution alpha map F1.3 generated by the encoder sub-network in step S2.4 is spliced with the high resolution original image F1.1 in step S2.1.1 and input into a depth-oriented filter, so that the alpha map F1.4 with the original resolution is restored.

5. A method according to claim 3, characterized in that in step S2.2, a feature extraction encoder sub-network based on a ViT and CNN hybrid structure is constructed, in particular as follows: respectively embedding 3 Mobile ViT V3 modules behind the 4 th, 6 th and 9 th reverse residual blocks of the Mobile Net V3 Larget to form an encoder sub-network, and leading out 3 jump connection characteristic diagrams F2.1, F2.2 and F2.3 from the positions of the 2 nd reverse residual block, the 1 st Mobile ViT block and the 2 nd Mobile ViT block of the sub-network; the end of the encoder network outputs a profile F2.4.

6. A method according to claim 3, characterized in that in step S2.3 a bottleneck block sub-network is built, in particular: the convolution block attention module, the LR-ASPP, the Conv-GRU and the characteristic reconstruction upsampling operator of the content perception are sequentially connected to form a bottleneck block sub-network; the subnetwork accepts as input the profile F2.4 and outputs the profile F3.

7. Method according to any of claims 3-6, characterized in that in step S2.4 a cyclic decoder sub-network based on attention and content perception is built, in particular: the convolution layer, the standardization layer, the activation layer, the Conv-GRU and the CARAFE up-sampling operator are connected back and forth to form encoder modules, and 3 encoder modules D1, D2 and D3 are constructed;

and/or, in step S2.4, constructing a cyclic decoder sub-network based on attention and content perception, specifically as follows:

s2.4.1: the method comprises the steps of downsampling a low-resolution original image F1.2 by 8 times to obtain a graph F4.1.1, obtaining a feature graph F4.1.2 by a layer of CBAM through a jump connection feature graph F2.1, and then sending F4.1.1, F4.1.2 and F3 to a decoder module D1 to obtain an output feature graph F4.1.3;

s2.4.2: downsampling the low-resolution original image F1.2 by 4 times to obtain a graph F4.2.1, obtaining a feature graph F4.2.2 by a layer of CBAM through the jump connected feature graph F2.2, and then sending F4.2.1, F4.2.2 and F4.1.3 to a decoder module D2 to obtain an output feature graph F4.2.3;

s2.4.3: downsampling the low-resolution original image F1.2 by 2 times to obtain a graph F4.3.1, obtaining a feature graph F4.3.2 by a layer of CBAM through the jump connection feature graph F2.3, and then sending F4.3.1, F4.3.2 and F4.2.3 to a decoder module D3 to obtain an output feature graph F4.3.3;

s2.4.4: and splicing the low-resolution original image F1.2 and the feature map F4.3.3, and sending the spliced images into a module formed by two groups of convolution layers, a normalization layer and an activation layer for processing to obtain the low-resolution alpha map F1.3.

8. The method according to any one of claims 1-6, wherein in step S3, the loss function is specifically: the sum of the L1 loss, the Laplacian pyramid loss and the time continuity loss is calculated as follows:

wherein alpha is _t Is an alpha map of the reality at the time t,alpha-map predicted for time t +.>Representing the value of the alpha map at the ith layer of the laplacian pyramid.

9. The method according to any one of claims 1-6, wherein in step S4, the content of the model input is specifically: the method comprises the steps that an image file to be scratched or a video frame F1.1 obtained by a camera and cyclic feature images T1.1, T1.2, T1.3 and T1.4 output by a Conv-GRU layer at the last moment are obtained, and if the moment is the starting moment, the cyclic feature images do not need to be input;

and/or, in step S4, the content of the output of the trained model is specifically: predicted alpha map F1.4 and cycle characteristic maps T2.1, T2.2, T2.3, T2.4 output by Conv-GRU layers at the moment.

10. A deep learning based real-time matting system for performing a method as claimed in any one of claims 1 to 9, comprising the following modules:

a data set acquisition module: acquiring a matting data set;

the network model building module: constructing a matting network model based on a ViT and CNN mixed structure;

model training module: training the image matting network model by utilizing the data set, and correcting by a loss function to obtain a trained model;

and an image matting alpha image acquisition module: and inputting the video frame obtained by the image file to be scratched or the camera and the cyclic feature image from the last moment into a training completed model, and acquiring the scratched alpha image and the cyclic feature image at the moment in real time.