WO2023116632A1

WO2023116632A1 - Video instance segmentation method and apparatus based on spatio-temporal memory information

Info

Publication number: WO2023116632A1
Application number: PCT/CN2022/140070
Authority: WO
Inventors: 周翊民; 马壮
Original assignee: 中国科学院深圳先进技术研究院
Priority date: 2021-12-22
Filing date: 2022-12-19
Publication date: 2023-06-29
Also published as: CN114241388A

Abstract

Disclosed in the present invention are a video instance segmentation method and apparatus based on spatio-temporal memory information. The video instance segmentation method comprises: acquiring a query key feature map and a query value feature map of an instance to be segmented of the current frame of image in a video, and a memory key feature map and a memory value feature map of an instance to be segmented of each memory frame image; calculating a weight value of the memory key feature map of the instance to be segmented of each memory frame image when attention matching is performed; selecting a predetermined number of weight values from among all the weight values in a descending order, and taking same as weight coefficients; according to the weight coefficients, and memory key feature maps and memory value feature maps of memory frame images corresponding to the weight coefficients, obtaining a global feature map having weighted spatio-temporal information; and obtaining an instance segmentation result of the current frame of image according to the global feature map, the query key feature map and the query value feature map. By means of the method, full use can be made of historical information of a memory frame image, thereby improving the robustness of a segmentation result.

Description

Video instance segmentation method and segmentation device based on spatio-temporal memory information

technical field

The invention belongs to the technical field of video processing, and in particular relates to a method for segmenting video instances based on spatio-temporal memory information, a segmenting device, a computer-readable storage medium, and computer equipment.

Background technique

The goal of video instance segmentation is to segment some specific categories of objects and obtain their segmentation masks without any human intervention. Different from unsupervised video object segmentation, video instance segmentation needs to identify specific objects, not just salient objects. In order to complete detection, segmentation and tracking tasks at the same time, most video instance segmentation methods are often based on object detection methods. The framework extends segmentation modules into one-stage and two-stage methods.

The two-stage method is the method of "detect first and then segment". First, the frame of the target object is located, and then the target object is segmented within the frame. The typical representative is Mask R-CNN. Mask R-CNN is based on Faster R-CNN, which adds a branch of predictive segmentation mask, which relies heavily on ROI features and operations. It first generates a set of candidate solutions, and then predicts the foreground mask on each ROI. , the two-stage problem is that features cannot be shared between segmentation and detection, so that end-to-end backpropagation cannot be performed; secondly, ROI features are cropped to a fixed resolution size, so that some large objects will lose segmentation accuracy; finally, The problem of ROI itself, the ROI candidate area is much larger than the final prediction, which limits the operating efficiency of the algorithm.

Single-stage methods treat detection, segmentation, and tracking in video instance segmentation as problems that can be solved simultaneously. The early single-stage method does not perform target detection, but directly performs segmentation, which loses the category information of the object and has a low accuracy rate. The later single-stage method is mainly to design the combination relationship between the prototype mask and the target instance. By learning a set of coefficients, the target position and the semantic segmentation result are matched. In 2019, YOLACT decomposed the instance segmentation into two parallel tasks to generate a Group prototype masks and predict mask coefficients for each instance, further improving accuracy. The SG-Net and ST-Mask proposed in 2021 refine the segmentation module based on the aforementioned method, and add the segmentation result information of the previous frame to guide the segmentation process of the current frame. However, the historical segmentation results contain many frames, including the segmentation results of different states of the target instance, which have important guiding significance for the segmentation branch to resist physical deformation and occlusion.

Existing single-stage video instance segmentation methods often do not consider historical segmentation results, and are not robust to drastic object appearance changes and occlusions.

Contents of the invention

(1) technical problem to be solved by the present invention

The technical problem solved by the invention is: how to make full use of the historical segmentation results in video instance segmentation to improve the robustness of the segmentation.

(2) The technical scheme adopted in the present invention

A video instance segmentation method based on spatiotemporal memory information, the video instance segmentation method comprising:

Obtain the query key feature map and query value feature map of the instance to be segmented in the current frame image in the video and the memory key feature map and memory value feature map of the instance to be segmented in each memory frame image, wherein the memory frame image is The historical frame image containing the instance to be segmented before the current frame image in the video;

Calculate the weight value of the memory key feature map of the instance to be divided in each piece of the memory frame image when performing attention matching;

Select a predetermined number of weight values from all weight values in descending order as weight coefficients;

Obtain a global feature map with weighted spatiotemporal information according to each of the weight coefficients, the memory key feature map and the memory value feature map of the memory frame image corresponding to each of the weight coefficients;

The instance segmentation result of the current frame image is obtained according to the global feature map, the query key feature map and the query value feature map.

Preferably, the method for obtaining the query key feature map and the query value feature map of the instance to be segmented of the current frame image in the video includes:

performing feature extraction on the current frame image to obtain several feature maps of different scales;

Obtaining the central point and bounding box of the instance to be segmented and the overall query key feature map and the overall query value feature map of the current frame image according to the feature maps of several different scales;

Binarize the overall query key feature map and the overall query value feature map of the current frame image according to the center point and the bounding box of the instance to be segmented, and obtain the query key feature map and query value of the instance to be segmented. value feature map.

Preferably, the method for obtaining the memory key feature map and memory value feature map of the instance to be segmented in each memory frame image in the video includes:

Obtain the segmentation results, original images, center points and bounding boxes of all the historical frame images before the current frame image in the video;

Obtain the overall memory key feature map and the overall memory value feature map of the historical frame image according to the segmentation result corresponding to the historical frame image and the original image;

According to the center point and the bounding box of each instance in the historical frame image, the overall memory key feature map and the overall memory value feature map of the historical frame image are binarized to obtain the individual instance in the historical frame image. Memory key feature map and memory value feature map;

According to the category of the instance to be segmented, from the memory key feature map and the memory value feature map of each instance in each of the history frame images, the memory key feature map and the memory value of the instance to be segmented in each memory frame image are screened out. feature map.

Preferably, the method for calculating the weight value of the memory key feature map of the instance to be segmented in each of the memory frame images when performing attention matching includes:

performing spatial connection on the memory key feature maps of the examples to be segmented in each of the memory frame images to obtain feature maps connected by channel dimensions;

A global pooling process is performed on the feature maps connected by the channel dimensions to obtain each weight value.

Preferably, the video instance segmentation method also includes:

Determining the prediction area of the instance to be segmented in the current frame image and the history area of the instance to be segmented in the memory frame image adjacent to the current frame image;

calculating a cosine similarity between the predicted area and the historical area, wherein the value of the cosine similarity is greater than 0 and less than 1;

The predetermined number is determined according to the cosine similarity and the total number of memory frame images containing the instance to be segmented.

Preferably, the method for obtaining a global feature map with weighted spatio-temporal information according to each of the weight coefficients, the memory key feature map and the memory value feature map of the memory frame image corresponding to each of the weight coefficients includes:

Calculate the weighted memory key feature map KMW _u,n of the memory frame image corresponding to each of the weight coefficients W _n [u]:

KMW _u,n = KM _u,n *W _n [u]

Perform matrix multiplication calculation on the weighted memory key feature map KMW _u,n and memory value feature map VM _u,n of the memory frame image:

The global feature map G _u,n with weighted spatio-temporal information is calculated according to the following formula:

Wherein, 1≤u≤g _q,n , g _q,n represents a predetermined number, and KM _u,n represents a memory key feature map of a memory frame image.

Preferably, the method for obtaining the instance segmentation result of the current frame image according to the global feature map, the query key feature map and the query value feature map includes:

performing matrix dot product on the query key feature map of the instance to be segmented in the global feature map to obtain an attention matrix;

The attention matrix is connected with the channel dimension of the instance to be segmented, and the result of the connection operation is sent to the decoder for deconvolution and upsampling to obtain the instance segmentation result.

The present application also discloses a video instance segmentation device based on a spatiotemporal memory weighted network, and the video instance segmentation device includes:

The feature map acquisition module is used to obtain the query key feature map and the query value feature map of the instance to be segmented in the current frame image in the video and the memory key feature map and the memory value feature map of the instance to be segmented in each memory frame image, Wherein the memory frame image is the historical frame image containing the instance to be segmented before the current frame image in the video

Weight value calculation module, used to calculate the weight value of the memory key feature map of the instance to be segmented in each of the memory frame images when performing attention matching

A weight coefficient screening module, configured to select a predetermined number of weight values from all weight values as weight coefficients in descending order;

A weighted value calculation module, used to obtain a global feature map with weighted spatiotemporal information according to each of the weight coefficients, the memory key feature map and the memory value feature map of the memory frame image corresponding to each of the weight coefficients;

An attention matching module, configured to obtain an instance segmentation result of the current frame image according to the global feature map, the query key feature map and the query value feature map.

The present application also discloses a computer-readable storage medium, the computer-readable storage medium stores a video instance segmentation program based on a spatiotemporal memory weighted network, and when the video instance segmentation program based on a spatiotemporal memory weighted network is executed by a processor Realize the above-mentioned video instance segmentation method based on spatiotemporal memory information.

The present application also discloses a computer device, which includes a computer-readable storage medium, a processor, and a video instance segmentation program based on a spatiotemporal memory weighted network stored in the computer-readable storage medium. When the video instance segmentation program of the memory weighted network is executed by the processor, the above-mentioned video instance segmentation method based on spatiotemporal memory information is realized.

(3) Beneficial effects

The invention discloses a video instance segmentation method and segmentation device based on spatio-temporal memory information. Compared with the existing method, it has the following technical effects:

This method can make full use of the historical information of memory frame images and improve the robustness of the segmentation results. At the same time, by screening out high-weight memory frame images for weighted matching, it avoids directly using all memory frame images for calculation, and reduces the amount of calculation. At the same time, let the network learn the attention matching weight of the memory frame to optimize the attention matching effect.

At the same time, the feature map is binarized, so that the spatial attention matching is only performed locally, which reduces the influence of similar objects on the segmentation results.

Description of drawings

Fig. 1 is the overall flowchart of the video instance segmentation method based on spatio-temporal memory information of Embodiment 1 of the present invention;

Fig. 2 is the detailed flowchart of the video instance segmentation method based on spatio-temporal memory information of Embodiment 1 of the present invention;

3 is a schematic diagram of the calculation process of the weight value of the memory key feature map according to Embodiment 1 of the present invention;

4 is a functional block diagram of a video instance segmentation device based on a spatiotemporal memory weighted network according to Embodiment 2 of the present invention;

FIG. 5 is a schematic diagram of computer equipment according to Embodiment 4 of the present invention.

Detailed ways

In order to make the object, technical solution and advantages of the present invention more clear, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

Before describing the various embodiments of the present application in detail, first briefly describe the inventive concept of the present application: when performing video instance segmentation in the prior art, the historical frame information is often not effectively used, when the sharp object appearance changes and occlusion , which greatly weakens the robustness of the segmentation results. To this end, this application provides a video instance segmentation method based on spatiotemporal memory information. First, obtain the key feature map and value feature map of the instance to be segmented in the current frame image and each memory frame image, and then calculate each memory key feature The weight value of the graph in the attention matching, and further screen out the key feature map with a larger weight value for weighted matching, and use the selected memory key feature map and memory value according to each weight value and the corresponding memory frame image The feature map performs spatio-temporal weighted matching on the key feature map and value feature map of the instance to be segmented in the current frame image to obtain the final video instance segmentation result. This method can make full use of the historical information of memory frame images and improve the robustness of the segmentation results. At the same time, by filtering out memory frame images with high weights for weighted matching, it avoids directly using all memory frame images for calculation and reduces the amount of calculation.

Specifically, as shown in FIG. 1 and FIG. 2, the video instance segmentation method based on spatiotemporal memory information in the first embodiment includes the following steps:

Step S10: Obtain the query key feature map and query value feature map of the instance to be segmented in the current frame image in the video, and the memory key feature map and memory value feature map of the instance to be segmented in each memory frame image, wherein the memory frame image is the historical frame image containing the instance to be segmented before the current frame image in the video;

Step S20: Calculate the weight value of the memory key feature map of the instance to be segmented in each memory frame image when performing attention matching;

Step S30: Select a predetermined number of weight values from all weight values in descending order as weight coefficients;

Step S40: Obtain a global feature map with weighted spatiotemporal information according to each weight coefficient, the memory key feature map and the memory value feature map of the memory frame image corresponding to each weight coefficient;

Step S50: Obtain an instance segmentation result of the current frame image according to the global feature map, the query key feature map and the query value feature map.

In step S10, the method for obtaining the query key feature map and the query value feature map of the instance to be segmented of the current frame image in the video includes the following steps:

Step S101 , perform feature extraction on the current frame image, and obtain several feature maps of different scales.

Exemplarily, the instance segmentation framework adopts the FCOS single-stage instance segmentation framework, and adds segmentation branches on the basis of center point and frame prediction. The backbone network is the same as FCOS, and selects ResNet and FPNs to extract input features. Among them, ResNet is used to extract the features of the current frame image, and five different scale convolution feature maps r1, r2, r3, r4, r5 are obtained, and three of the convolution feature maps r3, r4, r5 are horizontally connected to FPN The network obtains three different scales of pyramid feature maps P ₃ , P ₄ , P ₅ , and then performs pooling operation on the pyramid feature map P ₅ to obtain pyramid feature maps P ₆ , P ₇ , and the pyramid feature maps of different scales P ₃ , P ₄ , P ₅ , P ₆ , P ₇ as subsequent input.

Step S102. Obtain the center point and bounding box of the instance to be segmented and the overall query key feature map and overall query value feature map of the current frame image according to several feature maps of different scales.

On the one hand, the pyramid feature maps P ₃ , P ₄ , P ₅ , P ₆ , and P ₇ of different scales are respectively input into the center point regression branch network and the frame regression classification prediction branch network to obtain the center point CE _{t of the instance to be segmented, i} , bounding box B _t,i , class CL _t,i .

On the other hand, the fourth convolutional block of ResNet is combined with a convolutional layer as a query frame encoder, the original image of the current frame is input to the query frame encoder, and the overall query key feature map K _q of the current frame image is output. And the overall query value feature map V _q , q stands for query, which means query, and the query frame is equivalent to the current frame.

Step S103: Binarize the overall query key feature map and the overall query value feature map of the current frame image according to the center point and bounding box of the instance to be segmented, and obtain the query key feature map and query value feature map of the instance to be segmented .

According to the obtained center point and bounding box, the bounding box of each instance to be segmented is binarized at 1.5 times the size, and each to-be-segmented instance is divided into the bounding box of the overall query key feature map and the overall query value feature map of the current frame image. The pixel gray value of the area where the bounding box of the segmentation instance is located is set to 1, and the pixel gray value of other areas is set to 0. This can get N query key feature maps K _q,n and query value feature maps V _q,n , where N is the number of instances to be segmented, n∈[1,N]. The binarization process is to prevent similar instances from affecting the segmentation results, effectively increasing the segmentation accuracy.

Further, in step S10, the method for obtaining the memory key feature map and memory value feature map of the instance to be segmented in each memory frame image in the video includes:

Step S111 , acquiring the segmentation results, original images, center points and bounding boxes of each instance corresponding to all historical frame images in the video before the current frame image.

Segmentation results, original images, center points and bounding boxes of each instance, and each instance have been stored in advance for all historical frame images I ₁ -I _t-1 .

Step S112 , according to the corresponding segmentation results and original images of the historical frame images, the overall memory key feature map and memory query value feature map of the historical frame images are obtained.

Use the fourth convolution block of ResNet as a memory frame encoder, spatially connect the segmentation results corresponding to the historical frame images and the original image, input the spatial connection results to the memory frame encoder, and the memory frame encoder outputs the historical frame images The overall memory key feature map KM _T and the overall memory query value feature map VM _T , T∈[1,t-1].

Step S113, according to the center point and bounding box of each instance in the historical frame image, perform binarization operation on the overall memory key feature map and the overall memory value feature map of the historical frame image, and obtain the memory key feature of each instance in the historical frame image graph and memory value feature map.

According to the obtained center point and bounding box, the bounding box of each instance is binarized at 1.5 times the size, and the boundary of each instance is placed inside the bounding box of the overall memory key feature map and the overall memory value feature map of the historical frame image The pixel gray value of the area where the box is located is set to 1, and the pixel gray value of other areas is set to 0. This can get N memory key feature maps KM _T,n and query value feature maps VM _T,n , where N is the instance quantity. The binarization process is to prevent similar instances from affecting the segmentation results, effectively increasing the segmentation accuracy.

Step S114: Filter out the memory key feature map and memory value feature map of each instance to be segmented in each memory frame image from the memory key feature map and memory value feature map of each instance in each frame image according to the category of the instance to be segmented.

After the binarization process, it is beneficial to accurately segment each instance, find out the same instance category as the instance to be segmented from each historical frame image, and use the corresponding memory key feature map and memory value feature map as each The memory key feature map and memory value feature map of the instance to be segmented in the memory frame image.

Further, in step S20, the method for calculating the weight value of the memory key feature map of the instance to be segmented in each memory frame image when performing attention matching includes: the memory key feature of the instance to be segmented in each memory frame image The graphs are spatially connected to obtain feature maps connected by channel dimensions; global pooling is performed on feature maps connected by channel dimensions to obtain each weight value.

As shown in Figure 3, suppose the memory key feature map KM _T,n of the instance to be segmented has a dimension of H×W×C, and the total number of memory frame images is L. After spatial connection, the feature map C _n connected by channel dimensions is obtained. The dimension of is HWC×L, and the global pooling is performed through the convolution kernel of H×W×C to obtain the weight vector W _n , _which contains L weight values.

Further, the video instance segmentation method also includes:

Determine the prediction area of the instance to be segmented in the current frame image and the history area of the instance to be segmented in the memory frame image adjacent to the current frame image.

Calculate the cosine similarity P _gate between the predicted area and the historical area, where the value of the cosine similarity is greater than 0 and less than 1.

The predetermined number is determined according to the cosine similarity and the total number L _n of memory frame images containing instances to be segmented, specifically, the predetermined number _gq,n is calculated according to the following formula,

g _q,n =[P _gate *L _n ]

Among them, the closer the cosine similarity P _gate is to 1, the more similar the predicted area of the instance is to the historical area, and [] indicates rounding.

After the predetermined number g _q,n is calculated, the weight value W _n of the top g _q,n position is selected from the L weight values W _n as the weight coefficient W _n [u], that is, the memory frame with a high degree of correlation is used The feature map of the image is used for subsequent attention matching calculations. While making full use of historical information, it reduces the amount of calculation, computing time and memory usage.

Further, in step S40, according to each weight coefficient W _n [u], the memory key feature map KM _u,n and the memory value feature map VM _u,n of the memory frame image corresponding to each weight coefficient W _n [u] are obtained Methods for global feature maps G _u,n with weighted spatiotemporal information include:

Calculate the weighted memory key feature map KMW _u,n of the memory frame image corresponding to each weight coefficient W _n [u]:

KMW _u,n = KM _u,n *W _n [u]

Among them, 1≤u≤g _q,n , g _q,n represents a predetermined number, KM _u,n represents the memory key feature map of the memory frame image, D _u,n is the number u that contains attention information and historical segmentation details The feature map corresponding to the memory frame frame, the dimension of the global feature map G _u,n is a fixed dimension.

In step S50, the method for obtaining the instance segmentation result of the current frame image according to the global feature map G _u,n , the query key feature map K _q,n and the query value feature map V _q,n includes:

Perform matrix dot product of the global feature map G _u,n and the query key feature map K _q,n of the instance to be segmented to obtain the attention matrix;

The attention matrix and the query value feature map V _{q, n} of the instance to be segmented are connected in the channel dimension, and the result of the connection operation is sent to the decoder for deconvolution and upsampling to obtain the instance segmentation result.

Finally, store the instance segmentation results of the current frame image, the center point CE _t,i , the bounding box B _t,i , and the category CL _t,i into the memory module for the segmentation of subsequent frame images. The above process is repeated until the image segmentation of each frame in the video is completed.

Further, as shown in FIG. 4 , the video instance segmentation device based on the spatiotemporal memory weighted network of the second embodiment includes a feature map acquisition module 10, a weight value calculation module 20, a weight coefficient screening module 30, a weight value calculation module 40 and attention matching module 50 . Feature map acquisition module 10 is used to obtain the query key feature map and the query value feature map of the instance to be segmented in the current frame image in the video and the memory key feature map and the memory value feature map of the instance to be segmented in each memory frame image; weight value The calculation module 20 is used to calculate the weight value of the memory key feature map of the example to be segmented in each piece of memory frame images when performing attention matching; the weight coefficient screening module 30 is used to select from all weight values in order from large to small A predetermined number of weight values are used as weight coefficients; weight value calculation module 40 is used to obtain a global feature map with weighted spatio-temporal information according to each weight coefficient, the memory key feature map and the memory value feature map of the memory frame image corresponding to each weight coefficient; note The force matching module 50 is used to obtain the instance segmentation result of the current frame image according to the global feature map, the query key feature map and the query value feature map.

Specifically, the feature map acquisition module 10 is used to: perform feature extraction on the current frame image to obtain feature maps of several different scales; obtain the center point and bounding box of the instance to be segmented and the current frame image according to the feature maps of several different scales The overall query key feature map and the overall query value feature map; according to the center point and bounding box of the instance to be segmented, the overall query key feature map and the overall query value feature map of the current frame image are binarized to obtain the instance to be segmented The query key feature map and query value feature map of . And, the feature map acquisition module 10 is used to: acquire the segmentation results corresponding to all historical frame images before the current frame image in the video, the original image, the center point and the bounding box of each instance; according to the segmentation results corresponding to the historical frame images, the original The overall memory key feature map and the overall memory value feature map of the historical frame image are obtained from the figure; according to the center point and the bounding box of each instance in the historical frame image, the overall memory key feature map and the overall memory value feature map of the historical frame image are processed twice. value operation to obtain the memory key feature map and memory value feature map of each instance in the historical frame image; according to the category of the instance to be segmented, filter out the memory key feature map and memory value feature map of each instance in each historical frame image The memory key feature map and memory value feature map of the instance to be segmented in the memory frame image. For specific processing details of the feature map acquisition module 10, reference may be made to the relevant description in Embodiment 1, and details are not repeated here.

Further, the weight value calculation module 20 is used to spatially connect the memory key feature maps of the instances to be segmented in each memory frame image to obtain feature maps connected by channel dimensions; perform global pooling processing on the feature maps connected by channel dimensions, Get each weight value.

Further, for specific processing details of the weight coefficient screening module 30 , the weight value calculation module 40 and the attention matching module 50 , reference may be made to the relevant description in Embodiment 1, and details are not repeated here.

Embodiment 3 of the present application also discloses a computer-readable storage medium. The computer-readable storage medium stores a video instance segmentation program based on a spatiotemporal memory weighted network. When the video instance segmentation program based on a spatiotemporal memory weighted network is executed by a processor, Implement the above-mentioned video instance segmentation method based on spatio-temporal memory weighted network.

Embodiment 4 also discloses a computer device. At the hardware level, as shown in FIG. 5 , the computer device includes a processor 12 , an internal bus 13 , a network interface 14 , and a computer-readable storage medium 11 . The processor 12 reads the corresponding computer program from the computer-readable storage medium and executes it, forming a request processing device on a logical level. Of course, in addition to software implementations, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each A logic unit, which can also be a hardware or logic device. The computer-readable storage medium 11 stores a video instance segmentation program based on a spatiotemporal memory weighted network. When the video instance segmentation program based on a spatiotemporal memory weighted network is executed by a processor, the above-mentioned video instance segmentation method based on a spatiotemporal memory weighted network is realized.

Computer-readable storage media includes both volatile and non-permanent, removable and non-removable media by any method or technology for storage of information. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage , magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by computing devices.

The specific embodiments of the present invention have been described in detail above. Although some embodiments have been shown and described, those skilled in the art should understand that without departing from the principle and spirit of the present invention whose scope is defined by the claims and their equivalents Under the circumstances, these embodiments can be modified and improved, and these modifications and improvements should also be within the protection scope of the present invention.

Claims

A video instance segmentation method based on spatiotemporal memory information, characterized in that the video instance segmentation method comprises:

Obtain the query key feature map and query value feature map of the instance to be segmented in the current frame image in the video and the memory key feature map and memory value feature map of the instance to be segmented in each memory frame image, wherein the memory frame image is The historical frame image containing the instance to be segmented before the current frame image in the video;

Calculate the weight value of the memory key feature map of the instance to be divided in each piece of the memory frame image when performing attention matching;

Select a predetermined number of weight values from all weight values in descending order as weight coefficients;

Obtain a global feature map with weighted spatiotemporal information according to each of the weight coefficients, the memory key feature map and the memory value feature map of the memory frame image corresponding to each of the weight coefficients;

The instance segmentation result of the current frame image is obtained according to the global feature map, the query key feature map and the query value feature map.
The video instance segmentation method based on spatiotemporal memory information according to claim 1, wherein the method for obtaining the query key feature map and the query value feature map of the instance to be segmented in the current frame image in the video comprises:

performing feature extraction on the current frame image to obtain several feature maps of different scales;

Obtaining the central point and bounding box of the instance to be segmented and the overall query key feature map and the overall query value feature map of the current frame image according to the feature maps of several different scales;

Binarize the overall query key feature map and the overall query value feature map of the current frame image according to the center point and the bounding box of the instance to be segmented, and obtain the query key feature map and query value of the instance to be segmented. value feature map.
The video instance segmentation method based on spatiotemporal memory information according to claim 1, wherein the method for obtaining the memory key feature map and the memory value feature map of the instance to be segmented in each memory frame image in the video comprises:

Obtain the segmentation results, original images, center points and bounding boxes of all the historical frame images before the current frame image in the video;

Obtain the overall memory key feature map and the overall memory value feature map of the historical frame image according to the segmentation result corresponding to the historical frame image and the original image;

According to the center point and the bounding box of each instance in the historical frame image, the overall memory key feature map and the overall memory value feature map of the historical frame image are binarized to obtain the individual instance in the historical frame image. Memory key feature map and memory value feature map;

According to the category of the instance to be segmented, from the memory key feature map and the memory value feature map of each instance in each of the history frame images, the memory key feature map and the memory value of the instance to be segmented in each memory frame image are screened out. feature map.
The video instance segmentation method based on spatiotemporal memory information according to claim 1, characterized in that, calculating the weight value of the memory key feature map of the instance to be segmented in each of the memory frame images when performing attention matching Methods include:

performing spatial connection on the memory key feature maps of the examples to be segmented in each of the memory frame images to obtain feature maps connected by channel dimensions;

A global pooling process is performed on the feature maps connected by the channel dimensions to obtain each weight value.
The video instance segmentation method based on spatiotemporal memory information according to claim 4, wherein the video instance segmentation method further comprises:

Determining the prediction area of the instance to be segmented in the current frame image and the history area of the instance to be segmented in the memory frame image adjacent to the current frame image;

calculating a cosine similarity between the predicted area and the historical area, wherein the value of the cosine similarity is greater than 0 and less than 1;

The predetermined number is determined according to the cosine similarity and the total number of memory frame images containing the instance to be segmented.
The video instance segmentation method based on spatio-temporal memory information according to claim 1, wherein, according to each of the weight coefficients, the memory key feature map and the memory value feature map of the memory frame image corresponding to each of the weight coefficients are obtained with Methods for weighting global feature maps with spatiotemporal information include:

Calculate the weighted memory key feature map KMW u,n of the memory frame image corresponding to each of the weight coefficients W n [u]:

KMW u,n = KM u,n *W n [u]

Perform matrix multiplication calculation on the weighted memory key feature map KMW u,n and memory value feature map VM u,n of the memory frame image:

The global feature map G u,n with weighted spatio-temporal information is calculated according to the following formula:

Wherein, 1≤u≤g q,n , g q,n represents a predetermined number, and KM u,n represents a memory key feature map of a memory frame image.
The video instance segmentation method based on spatiotemporal memory information according to claim 6, wherein the result of instance segmentation of the current frame image is obtained according to the global feature map, the query key feature map and the query value feature map Methods include:

performing matrix dot product on the query key feature map of the instance to be segmented in the global feature map to obtain an attention matrix;

The attention matrix is connected with the channel dimension of the instance to be segmented, and the result of the connection operation is sent to the decoder for deconvolution and upsampling to obtain the instance segmentation result.
A video instance segmentation device based on spatio-temporal memory weighted network, characterized in that, the video instance segmentation device includes:

The feature map acquisition module is used to obtain the query key feature map and the query value feature map of the instance to be segmented in the current frame image in the video and the memory key feature map and the memory value feature map of the instance to be segmented in each memory frame image, Wherein the memory frame image is the historical frame image containing the instance to be segmented before the current frame image in the video

Weight value calculation module, used to calculate the weight value of the memory key feature map of the instance to be segmented in each of the memory frame images when performing attention matching

A weight coefficient screening module, configured to select a predetermined number of weight values from all weight values as weight coefficients in descending order;

A weighted value calculation module, used to obtain a global feature map with weighted spatiotemporal information according to each of the weight coefficients, the memory key feature map and the memory value feature map of the memory frame image corresponding to each of the weight coefficients;

An attention matching module, configured to obtain an instance segmentation result of the current frame image according to the global feature map, the query key feature map and the query value feature map.
A computer-readable storage medium, characterized in that the computer-readable storage medium stores a video instance segmentation program based on a spatio-temporal memory weighted network, and the video instance segmentation program based on a spatio-temporal memory weighted network is implemented when the processor executes The video instance segmentation method based on spatiotemporal memory information according to any one of claims 1 to 7.
A computer device, characterized in that the computer device includes a computer-readable storage medium, a processor, and a video instance segmentation program based on a spatio-temporal memory weighted network stored in the computer-readable storage medium, the spatio-temporal memory-based When the video instance segmentation program of the weighted network is executed by the processor, the video instance segmentation method based on spatiotemporal memory information according to any one of claims 1 to 7 is realized.