CN114627292A - Industrial occlusion target detection method - Google Patents

Abstract

The present application relates to an industrial occlusion target detection method. In three consecutive Wiggle Transformer Block operations, different types of W‑MSA modules, WWL‑MSA modules and WWR modules are used when performing the window segmentation step in each Wiggle Transformer Block operation. ‑MSA module performs window segmentation, and performs attention calculation in each window formed after segmentation. Different MSA modules can extract different window positions, realize more cross-window connections, increase the interaction between windows, retain the necessity of image edge information interaction, and improve globality and robustness. Awesome. The attention calculation is performed separately in each window. The advantage of this is that it can not only introduce the limitations of the CNN convolution operation, but also save the calculation amount to meet the needs of industrial applications. The present application can not only reduce the influence of the occluder on the detection target, but also can increase the detail features at multiple levels, so as to improve the accuracy of the detection of the occluded target.

Description

Industrial occlusion target detection method

technical field

The present application relates to the technical field of computer vision, and in particular, to an industrial occlusion target detection method.

Background technique

In recent years, with the rapid development of deep learning technology, related technologies have been widely used in the field of object detection. As one of the more basic recognition technologies, target detection provides technical support for intelligent transportation, social security, industrial Internet and other fields, and has a wide range of application scenarios.

Occlusion is common in target detection in various fields. Therefore, occlusion target detection is a difficult problem in the field of target detection, and it is also one of the most widely concerned problems in the industrial application of target detection methods. According to the different occluders, the occlusion targets in the real scene are mainly divided into two cases, that is, the target to be detected is occluded by the interfering object, which often causes the loss of target information, resulting in the missed detection of the target; the other is The occlusion caused by the targets to be detected usually leads to a large amount of interference information, which in turn leads to the situation of target misdetection.

Object detection has been dominated by convolutional neural networks for a relatively long time. Although the detection algorithm of the convolutional neural network has achieved good results, most methods only use the features of the last layer of the convolutional neural network, which cannot achieve the diversity of features with different receptive fields in terms of scale, and the amount of system computation will vary with As the number of convolutional layers increases, the computation is very time-consuming. In particular, convolution operations are good at extracting local features but have difficulty capturing global representations. Many people combine Self-Attention with ResNet in the task of target detection. Compared to pure convolution, the combination of the two achieves a trade-off between accuracy and speed than the corresponding ResNet architecture. However, their expensive memory access causes their actual latency to be significantly larger than convolutional nets and the problem of how to precisely embed local features and global representations remains. The next Vision Transformer has made some progress in image classification, but its structure is not suitable as a general backbone network for dense vision tasks or input images. Limited by the matrix nature of the image, a picture needs at least several hundred pixels to express the picture information, and modeling such hundreds of long sequences of data results in a large amount of computation for the Transformer, which cannot achieve accuracy and access memory. And the trade-off before image processing speed.

To sum up, for industrial occlusion target detection, the main problem at present is the lack of detection accuracy, and the detection speed is difficult to meet the requirements.

SUMMARY OF THE INVENTION

Based on this, it is necessary to provide an industrial occlusion target detection method to solve the problems that the detection accuracy of the traditional industrial occlusion target detection method is insufficient and the detection speed is difficult to meet the requirements. The method provided in this application can be widely used in various industrial detection fields.

The present application provides an industrial occlusion target detection method, the method comprising:

Input the image to be processed, perform patch segmentation on the to-be-processed image and apply a linear embedding layer for dimensionality reduction to obtain the first feature map;

The first feature map is carried out three consecutive WiggleTransformer Block operations to generate a second feature map;

The second feature map is input to the Patch Merging module and performs the Restructure operation, and the second feature map after the Restructure operation is carried out three consecutive WiggleTransformer Block operations to generate the third feature map;

The 3rd feature map is input to Patch Merging module and carries out Restructure operation, the 3rd feature map after Restructure operation is carried out six times of continuous WiggleTransformer Block operation, generates the 4th feature map;

The 4th feature map is input to Patch Merging module and performs Restructure operation, the 4th feature map after the Restructure operation is carried out three consecutive WiggleTransformer Block operations, and the 5th feature map is generated;

The fifth feature map is sent to the RPN network, and the occlusion target detection is performed on the fifth feature map through the RPN network, and the occlusion target detection result is output;

In three consecutive WiggleTransformer Block operations, the W-MSA module, the WWL-MSA module and the WWR-MSA module are respectively used for window segmentation.

The present application relates to an industrial occlusion target detection method, which improves the SwinTransformer block module in the Swin Transformer network. In three consecutive WiggleTransformer Block operations, different types of MSA module. In three consecutive WiggleTransformer Block operations, the W-MSA module, the WWL-MSA module and the WWR-MSA module are respectively used to perform window segmentation, and the attention calculation is performed in each window formed after segmentation. Different MSA modules can extract different window positions, realize more cross-window connections, increase the interaction between windows, retain the necessity of image edge information interaction, and improve globality and robustness. Awesome. The attention calculation is performed separately in each window. The advantage of this is that it can not only introduce the limitations of the CNN convolution operation, but also save the amount of calculation and improve the detection speed of the model to meet the needs of industrial applications. And by combining patches, the resolution can be reduced, the number of channels can be adjusted to form a hierarchical design, and a certain amount of computation can be saved. At the same time, it can not only reduce the influence of occluders on the detection target, but also increase the detail features at multiple levels to improve the accuracy of occluded target detection and meet the high-precision requirements of industrial applications.

Description of drawings

FIG. 1 is a schematic flowchart of an industrial occlusion target detection method provided by an embodiment of the present application.

FIG. 2 is a network structure diagram of an industrial occlusion target detection method provided by an embodiment of the present application.

FIG. 3 is a flowchart block diagram of three consecutive WiggleTransformer Block operations in an industrial occlusion target detection method provided by an embodiment of the present application.

FIG. 4 is a schematic diagram of window cutting in an industrial occlusion target detection method provided by an embodiment of the present application.

5 is a schematic diagram of a W-MSA module window reorganization mode in an industrial occlusion target detection method provided by an embodiment of the application.

6 is a schematic diagram of a window reorganization mode of a WWL-MSA module in an industrial occlusion target detection method provided by an embodiment of the present application.

7 is a schematic diagram of a WWR-MSA module window reorganization mode in an industrial occlusion target detection method provided by an embodiment of the application.

Fig. 8 is a schematic diagram of selecting 2 × 2 patch blocks at intervals for splicing in the height direction and width direction of the second feature map through the Patch Merging module in S320 in the industrial occlusion target detection method provided by an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

The present application provides an industrial occlusion target detection method. It should be noted that the industrial occlusion target detection method provided in this application is applied to pictures in which objects occlude each other.

In addition, the industrial occlusion target detection method provided by this application does not limit its execution subject. Optionally, the execution subject of the industrial occlusion target detection method provided by the present application may be an industrial occlusion target detection terminal.

As shown in Figure 1, in an embodiment of the present application, the industrial occlusion target detection method includes the following S100 to S600:

S100, inputting a picture to be processed, performing patch segmentation on the picture to be processed and applying a linear embedding layer to reduce the dimension to obtain a first feature map.

S200, perform three consecutive Wiggle Transformer Block operations on the first feature map to generate a second feature map.

S300, the second feature map is input to the Patch Merging module to perform a Restructure operation, and the second feature map after the Restructure operation is subjected to three consecutive Wiggle Transformer Block operations to generate a third feature map.

S400: Input the third feature map to the Patch Merging module to perform a Restructure operation, and perform six consecutive Wiggle Transformer Block operations on the third feature map after the Restructure operation to generate a fourth feature map.

S500, the fourth feature map is input to the Patch Merging module to perform a Restructure operation, and the fourth feature map after the Restructure operation is subjected to three consecutive Wiggle Transformer Block operations to generate a fifth feature map.

S600, the fifth feature map is sent to the RPN network, and the occlusion target detection is performed on the fifth feature map through the RPN network, and the occlusion target detection result is output.

Among them, in three consecutive Wiggle Transformer Block operations, the W-MSA module, the WWL-MSA module and the WWR-MSA module are respectively used to perform window segmentation, and the attention calculation is performed in each window formed after segmentation.

Specifically, the to-be-processed picture is a picture in which objects that occlude each other appear.

In three consecutive WiggleTransformer Block operations, the present embodiment adopts the W-MSA module, the WWL-MSA module and the WWR-MSA module to perform window segmentation, and performs attention calculation in the windows formed after each segmentation.

As shown in Figure 2, Figure 2 is a network structure diagram of an industrial occlusion target detection method provided by an embodiment of the application, which briefly describes the flow of the entire industrial occlusion target detection method, that is, S100 to S600 are briefly described.

The Image in Figure 2 is the image to be processed. stage1 is S100 to S200. stage2 is S300. stage3 is S400. stage4 is S500. The last RPN part is S600, that is, the occlusion target detection is performed on the fifth feature map through the RPN network, and the occlusion target detection result is output. There are 2 target detection results, one is the class loss value, and the other is the bbox_loss value.

In the present embodiment, the Swin Transformer block module in the Swin Transformer network is improved, and in three consecutive WiggleTransformer Block operations, different types of MSA modules are used when performing the window segmentation step in each WiggleTransformerBlock operation. In three consecutive WiggleTransformer Block operations, the W-MSA module, the WWL-MSA module and the WWR-MSA module are used for window segmentation. Different MSA modules can extract different window positions, realize more cross-window connections, increase the interaction between windows, retain the necessity of image edge information interaction, and improve globality and robustness. Awesome. The attention calculation is performed separately in each window. The advantage of this is that it can not only introduce the limitations of the CNN convolution operation, but also save the calculation amount and improve the detection speed of the model to meet the needs of industrial applications. And by merging patches, the resolution can be reduced, the number of channels can be adjusted to form a hierarchical design, and a certain amount of computation can be saved. At the same time, it can not only reduce the influence of occluders on the detection target, but also increase the detailed features at multiple levels to improve the accuracy of occluded target detection and meet the needs of industrial applications for high precision.

In addition, we added three convolution operations in series in the last step of each WiggleTransformer Block operation, and integrated the local features extracted by the convolution operation into the transformer to enhance representation learning and better preserve the representation capabilities of local features and global representations .

In an embodiment of the present application, S100 includes the following S110 to S130:

S110, input a picture to be processed whose height is H, width is W, and the number of channels is 3.

Specifically, for example, H takes 224 pixels and W takes 224 pixels, then the input to-be-processed picture is a 224×224×3 picture.

个尺寸相同的patch块，每个patch块的 高度为4像素，宽度为4像素。S120: Divide the picture to be processed into

Patch blocks of the same size, each patch block is 4 pixels high and 4 pixels wide.

承接上述的例子，如果H取224像 素，W取224像素，那么本步骤中划分后的patch块的数量为56×56＝3136，其 中每个patch大小为4×4，即高度为4像素，宽度为4像素。那么这多个尺寸 相同的patch块可以输出为一个56×56×48的特征图(即feature map)，这 个特征图在本实施例中称为原始特征图。48成为原始特征图的特征维度，你可 以把它看成是由多个二维图片(即patch块)叠放在一起产生的结构，其特征 维度为4×4×3＝48。Specifically, as shown in Figure 2, the to-be-processed picture is divided into non-overlapping patch sets by Patch Partition, and the number of patch blocks is

Following the above example, if H takes 224 pixels and W takes 224 pixels, then the number of patch blocks divided in this step is 56×56=3136, where the size of each patch is 4×4, that is, the height is 4 pixels, The width is 4 pixels. Then, the multiple patch blocks with the same size can be output as a 56×56×48 feature map (ie, feature map), and this feature map is called the original feature map in this embodiment. 48 becomes the feature dimension of the original feature map, you can think of it as a structure produced by stacking multiple two-dimensional images (ie, patch blocks) together, and its feature dimension is 4×4×3=48.

S130: Input multiple patch blocks with the same size as original feature maps into the linear embedding layer, and change the feature dimension of the original feature map to a preset dimension C, so as to convert the 2-dimensional original feature map into a 1-dimensional patch slice layer , take the converted 1-dimensional patch slice as the first feature map.

Specifically, following the above example, the original feature map of 56×56×48 is input into the linear embedding layer. The English name of the linear embedding layer is Linear Embedding. The linear embedding layer can map the 56×56×48 original feature map into a preset dimension C. The specific principle is to expand the H dimension and W dimension in the 2-dimensional original feature map, so that the 2-dimensional original feature map is converted into a 1-dimensional patch slice, and the final feature structure (or called feature vector) is generated as the first feature map.

Then it can be understood that the original feature map of 56×56×48 can be transformed into the first feature map of 56×56×96.

In an embodiment of the present application, the S200 includes the following S210 to S230:

S210: Input the first feature map into the Wiggle Transformer Block, and perform the first Wiggle Transformer Block operation. In the process of performing the first Wiggle Transformer Block operation, the W-MSA module is used to perform window segmentation to generate a plurality of first windows, and the attention calculation is performed separately in the first windows formed after each segmentation. S220: Input the first feature map after the first Wiggle Transformer Block operation into the Wiggle Transformer Block, and perform the second Wiggle Transformer Block operation. In the process of performing the second Wiggle Transformer Block operation, the WWL-MSA module is used to perform window segmentation to generate a plurality of second windows, and the attention calculation is performed separately in the second windows formed after each segmentation.

S230, input the first feature map after the second Wiggle Transformer Block operation into the Wiggle Transformer Block, perform the third Wiggle Transformer Block operation, and use the finally obtained feature map as the second feature map. In the process of performing the third Wiggle Transformer Block operation, the WWR-MSA module is used to perform window segmentation, multiple third windows are generated, and attention calculation is performed in each of the third windows formed after segmentation.

Specifically, S210 to S230 include three consecutive Wiggle Transformer Block operations. As shown in Fig. 3, Fig. 3 is a flow block diagram of three consecutive Wiggle Transformer Block operations. The window segmentation module used in each WiggleTransformer Block operation is different, which means that the window segmentation method is different. The W-MSA module has a window-type multi-head attention module without window-to-window interaction. The WWL-MSA module has a multi-head attention module that focuses on the left window interaction. The WWR-MSA module has a multi-head attention module that focuses on the right window interaction. Their respective focuses are different.

Specifically, the window generated by window segmentation using the W-MSA module is named the first window. The window generated by window segmentation using the WWL-MSA module is named the second window. The window generated by window segmentation using the WWR-MSA module is named the third window. The meanings of "the first window", "the second window" and "the third window" appearing later are all explained here, and the explanation will not be repeated hereafter.

S210, S220, S230, each step includes two steps, one step is window segmentation, and the other step is attention calculation in the window.

Window segmentation is also divided into two steps, the first step is window cutting, and the second step is window reorganization.

FIG. 4 is a schematic diagram of window cutting in an industrial occlusion target detection method provided by an embodiment of the present application, which shows an example of window cutting, and window cutting can indicate how the window comes from. In Fig. 4, the minimum unit is the smallest grid, and the patch block framed by the dotted range of the 4×4 minimum unit is a window. It can be understood that Fig. 4(a) has 3×3=9 windows.

Window cuts can be cut in different ways. The method of Figure 4(a) is to take a quarter of the window size at the center point of each 2*2 adjacent window to form a new window, and repeat this 4 times to obtain window 1, window 2, window 3 and window 4 . The method in Figure 4(b) is the two outermost lines in the horizontal direction, and half of the window size is taken between each two adjacent windows to form a new window. Repeat this 4 times to obtain window 5, window 6, window 9 and window 11. The method in Figure 4(c) is the two outermost columns in the vertical direction, and half of the window size is taken between each two adjacent window blocks to form a new window, and this is repeated 4 times to obtain window 7, window 8, and window 8. 11 and window 12.

The three different window reorganization methods of W-MSA module, WWL-MSA module and WWR-MSA are shown in Figure 5-Figure 7.

The window reorganization method of the W-MSA module is shown in Figure 5, which mainly focuses on the interaction between windowless and windows. Following the above embodiment, 9 windows are randomly selected from the multiple windows obtained by cutting the window in FIG. 4 and arranged into a 3 × 3 matrix, and then, among the 9 windows, attention calculation is performed in each window.

The window reorganization method of the WWL-MSA module is shown in Figure 6, which mainly focuses on the interaction with the left window. Following the above embodiment, 9 windows are randomly selected from the multiple windows obtained by cutting the window in FIG. 4 and arranged into a 3×3 matrix, but window 1, window 2, window 3, and window 4 are in the lower right corner, and other windows are surrounded by windows. 1, the upper left corner of window 2, window 3, and window 4 can indicate that the focus is on the interaction with the left window. Then in the 9 windows, the attention calculation is performed in each window.

The window reorganization method of the WWR-MSA module is shown in Figure 7, which mainly focuses on the interaction with the right window. Randomly select 9 of the multiple windows obtained by cutting the window in Figure 4 and arrange them into a 3×3 matrix, but window 1, window 2, window 3, and window 4 are in the upper left corner, and other windows are surrounded by window 1, window 2, Window 3, the lower right corner of Window 4, can indicate a focus on interaction with the right window. Then in the 9 windows, the attention calculation is performed in each window.

In this embodiment, in three consecutive WiggleTransformer Block operations, window segmentation is performed by using the W-MSA module, the WWL-MSA module and the WWR-MSA module respectively. Different MSA modules can extract different window positions, realize more connections across windows, increase the interaction between windows, and improve globality and robustness.

As shown in FIG. 3, in an embodiment of the present application, the S210 includes the following S211 to S217:

S211 , input the first feature map into the Wiggle Transformer Block, and perform a LayerNormalization operation.

Specifically, S211 to S217 specifically describe the process of the first Wiggle Transformer Block operation, which is the first column operation in FIG. 3 . First, in S211, a Layer Normalization operation, which is referred to as an LN operation for short, is performed on the first feature map. For the convenience of description, the first feature map is denoted as Z ^x-1 here.

The role of the Layer Normalization operation is to normalize Z ^x-1 .

S212, perform the W-MSA module window segmentation on the feature map after the Layer Normalization operation, generate a plurality of first windows, and perform attention calculation in the first windows formed after each segmentation to generate the W-MSA Feature map after module window segmentation and attention calculation.

Specifically, the feature map after the Layer Normalization operation is the first feature map after the LayerNormalization operation in S211.

The window segmentation method of the W-MSA module is shown in Figure 5.

When the W-MSA module window is divided, the number of heads of Multi-Head Attention is 3, K, Q, V are respectively a Tensor of length 3, and the dimension of each Tensor is (56, 56, 96), and the window size is 7×7. The size of each window shown in Figure 4 is 4x4 instead of 7x7 due to canvas size limitations. Then, following the above embodiment, the number of windows is 56/7×56/7=8×8=64.

In the feature map obtained by window segmentation, each 7×7 window performs Self-Attention (that is, the attention calculation mentioned in the previous content) within itself, and is added to Q and K in the original formula for calculating Attention The relative position encoding B is used to further improve the performance of the model.

The formula for the Attention calculation of this application is as follows:

Among them, Q is the Query vector, K is the Key vector, V is the Value vector, and d is the parameter of score normalization. B is the relative position code newly added in this application, which is a self-learning parameter.

即将W-MSA模块窗口切 分后得到的特征图，它是第一特征图先进行Layer Normalization操作，再执 行W-MSA模块窗口切分后得到的结果。After each 7×7 window performs Self-Attention within itself, the entire process of window segmentation is completed, and the generated feature map is marked as

That is, the feature map obtained after the W-MSA module window is segmented, it is the result obtained after the first feature map performs the Layer Normalization operation, and then performs the W-MSA module window segmentation.

S213, perform residual connection between the feature map after the W-MSA module window segmentation and the attention calculation and the first feature map.

和Z^x-1进行残差连接，残差连接的作用是探 寻W-MSA模块窗口切分后得到的特征图和W-MSA模块窗口切分前的特征图之间 的联系。Specifically, this step is to

Perform residual connection with Z ^x-1 . The function of residual connection is to explore the connection between the feature map obtained after the W-MSA module window is segmented and the feature map before the W-MSA module window segment.

S214, perform a Layer Normalization operation on the feature map obtained after residual connection.

Specifically, the feature map obtained after residual connection is again subjected to an LN operation.

S215, the feature map after the Layer Normalization operation is input into a 2-layer MLP neural network module for neural network processing.

Specifically, the "feature map after Layer Normalization operation" mentioned in this step is the output result of S214. This step is to input the output result of S214 into a 2-layer MLP neural network module for neural network processing.

The 2-layer MLP neural network module is a 2-layer multilayer perceptron.

S216, perform three convolutions on the feature map processed by the neural network.

Specifically, the feature map processed by the neural network is input into a 3-layer convolution structure for cubic convolution. The convolution kernels of the cubic convolution are 3×3, 5×5 and 1×1, respectively. Convolution can make the feature map pay more attention to the extraction of local details.

S217, perform residual connection again on the feature map obtained after three convolutions and the feature map obtained by residual connection, and finally obtain the first feature map after the operation of the first Wiggle Transformer Block. The convolution kernels of the three convolutions are 3×3, 5×5 and 1×1, respectively.

Specifically, this residual connection is equivalent to performing residual connection between the output result of S213 and the output result of S216, and finally obtains the first feature map after the first Wiggle Transformer Block operation, which is denoted as Z ^x .

In this embodiment, a 3-layer convolution module is added after the MLP module of each block, so that it pays attention to the global features and overcomes the situation that the target is occluded by occluders and confused in the background. The interaction of global features in the larger receptive field also pays more attention to local features, thereby improving the accuracy of occluded target detection. It has the characteristics of fully considered the displacement invariance of CNN, the relationship between the receptive field and the level, etc., which effectively combines the advantages of CNN and transformer.

As shown in FIG. 3, in an embodiment of the present application, the S220 includes the following S221 to S227:

S221 , input the first feature map after the operation of the first Wiggle Transformer Block into the Wiggle Transformer Block, and perform a Layer Normalization operation.

S222, perform the WWL-MSA module window segmentation on the feature map after the Layer Normalization operation, generate multiple second windows, and perform attention calculation in each of the second windows formed after segmentation to generate the WWL-MSA Feature map after module window segmentation and attention calculation.

S223, perform residual connection between the feature map after window segmentation and attention calculation by the WWL-MSA module and the first feature map after the first WiggleTransformer Block operation.

S224 , performing a Layer Normalization operation on the feature map obtained after the residuals are connected.

S225, the feature map after the Layer Normalization operation is input into a 2-layer MLP neural network module for neural network processing.

S226, perform three convolutions on the feature map processed by the neural network.

S227: Perform residual connection again on the feature map obtained after the three convolutions and the feature map obtained by residual connection, and finally obtain the first feature map after the second Wiggle Transformer Block operation. The convolution kernels of the three convolutions are 3×3, 5×5 and 1×1, respectively.

Specifically, S221 to S227 are the operations in the second column in FIG. 3 , and their working principles are mostly the same as those of S211 to S217 , which will not be repeated here. The difference lies in the modules used in window segmentation. Different, that is, the segmentation form adopted is different (actually it belongs to the different window reorganization methods). The S222 adopts the WWL-MSA module.

Figure 6 shows the window segmentation method of the WWL-MSA module.

The W-MSA module realizes the interaction between windows, and the window after masking contains the elements of the original adjacent windows, and pays more attention to the communication between the left windows.

After performing S221 to S227 for Z ^x , a first feature map after the second Wiggle Transformer Block operation is generated, which is denoted as Z ^x+1 .

In an embodiment of the present application, the S230 includes the following S231 to S237:

S231: Input the first feature map after the second Wiggle Transformer Block operation into the Wiggle Transformer Block, and perform a Layer Normalization operation.

S232, perform the WWR-MSA module window segmentation on the feature map after the Layer Normalization operation, generate a plurality of third windows, and perform attention calculation in the third windows formed after each segmentation to generate the WWR-MSA Feature map after module window segmentation and attention calculation.

S233, perform residual connection between the feature map after window segmentation and attention calculation by the WWR-MSA module and the first feature map after the second WiggleTransformer Block operation.

S234 , performing a Layer Normalization operation on the feature map obtained after residual connection.

S235, the feature map after the Layer Normalization operation is input into a 2-layer MLP neural network module for neural network processing.

S236, perform three convolutions on the feature map processed by the neural network.

S237, perform residual connection again on the feature map obtained after the three convolutions and the feature map obtained after residual connection, and finally obtain a second feature map. The convolution kernels of the cubic convolution are 3×3, 5×5 and 1×1, respectively.

Specifically, S231 to S237 are the operations in the third column in FIG. 3 . Its working principle is the same as that of S211 to S217 , and most of the working principles of S221 to S227 are the same, so I will not repeat them here. The difference is that The modules used in window segmentation are different, that is, the segmentation forms used are different. The S232 uses the WWR-MSA module. Based on the division of the W-MSA module, the WWR-MSA module masks the feature map at different positions to obtain the “shifted” window. The size of the mask is M/2, and the number of windows can be kept unchanged. .

The window segmentation method of the WWR-MSA module is shown in Figure 7.

The WWR-MSA module realizes the interaction between windows. The window after masking contains the elements of the original adjacent windows, and pays more attention to the communication between the right windows.

After executing S231 to S237 for Z ^x+1 , a second feature map is generated, which is denoted as Z ^x+2 .

In summary, after performing three consecutive Wiggle Transformer Block operations, Z ^x-1 becomes Z ^x+2 , but the resolution remains at 224/4 for H and 224/4 for W.

Subsequent S300, S400, and S600 all have the same steps as S200, that is, they all have the same “perform three consecutive WiggleTransformer Block operations”. Note that their working principles are exactly the same. It's just that S300, S400, and S600 all need to perform a "Restructure operation" before performing three consecutive WiggleTransformer Block operations. The following explains the process of this Restructure operation in detail through S310 to S340.

In an embodiment of the present application, the S300 includes the following S310 to S330:

S310, input the second feature map to the Patch Merging module.

S320 , 2×2 patch blocks are selected at intervals in the height direction and the width direction of the second feature map through the Patch Merging module for splicing.

S330, perform a convolution with a convolution kernel of 1×1 on the spliced second feature map, and finally generate a second feature map after the Restructure operation.

Specifically, "Restructure operation" includes two steps, the first step is the splicing step in S320, and the second step is the convolution step in S330. "Restructure operation" is actually equivalent to a downsampling process in terms of processing principle.

The splicing process of S320 is shown in Fig. 8, which can be understood as dividing multiple patch blocks in the second feature map into multiple groups, as shown in Fig. 8(a), the upper left corner of Fig. 8(a) is a group of patch blocks , Figure 8(a) has a total of 4×4=16 groups of patch blocks. Each group consists of 4 adjacent patch blocks arranged in a 2×2 matrix. Then, the 4 patch blocks in each group are split and divided into the upper left corner patch block, the upper right corner patch block, the lower left corner patch block and the lower right corner patch block. Each group takes out the upper left corner patch block, and splices all the upper left corner patch blocks to generate the patch block of Figure (b). Each group takes out the upper right corner patch block, and splices all the upper right corner patch blocks to generate the patch block of Figure (c). Each group takes out the lower left corner patch block, and splices all the lower left corner patch blocks to generate the patch block of Figure (d). Each group takes out the patch block in the lower right corner, and splices all the patch blocks in the lower right corner to generate the patch block of Figure (e). Finally, the patch block of Figure (b), the patch block of Figure (c), the patch block of Figure (d), and the patch block of Figure (e) are stacked to generate a second feature map after splicing, and then the subsequent S330 is performed.

Following the above embodiment, after splicing, Z ^x+2 becomes a feature map of 28H, 28W, and 384 feature dimensions. During convolution, the feature dimension 384 is reduced to 192.

After performing S310 to S340, the second feature map after the Restructure operation is obtained. Subsequently, perform three consecutive WiggleTransformer Block operations on the second feature map after the Restructure operation (the principle is consistent with S210 to S230). Among them, the number of heads of Multi-Head Attention is 6, and the tensor dimension composed of Q, K, and V is (28, 28, 192). At this time, the number of windows is 28/7×28/7, that is, 4×4=16 windows in total. The final output third feature map is (28, 28, 192), that is, the resolution of the final third feature map is kept at H/8, W/8.

In an embodiment of the present application, the S400 includes the following S410 to S470:

S410, the third feature map is input to the Patch Merging module to perform the Restructure operation.

Specifically, in the same manner as the downsampling in S310 to S340, each group of 2×2 adjacent patch blocks in the third feature map is first spliced, and then a convolution with a convolution kernel of 1×1 is performed.

After splicing each group of 2×2 adjacent patch blocks in the third feature map, the feature map becomes (14, 14, 768). After convolution with a convolution kernel of 1×1, the feature dimension is reduced from 768 to 384 .

S420: Input the third feature map after the Restructure operation into the Wiggle Transformer Block, and perform the first Wiggle Transformer Block operation. In the process of performing the first Wiggle Transformer Block operation, the W-MSA module is used for window segmentation, multiple first windows are generated, and attention calculation is performed in each of the first windows formed after segmentation.

S530: Input the third feature map after the first Wiggle Transformer Block operation into the Wiggle Transformer Block, and perform the second Wiggle Transformer Block operation. In the process of performing the second WiggleTransformer Block operation, the WWL-MSA module is used to perform window segmentation, and multiple second windows are generated, and attention calculation is performed in each of the second windows formed after segmentation.

S440: Input the third feature map after the second Wiggle Transformer Block operation into the Wiggle Transformer Block, and perform the third Wiggle Transformer Block operation. In the process of performing the third WiggleTransformer Block operation, the WWR-MSA module is used to perform window segmentation, multiple third windows are generated, and attention calculation is performed in each of the third windows formed after segmentation.

S450, input the third feature map after the third Wiggle Transformer Block operation into the Wiggle Transformer Block, and perform the second first Wiggle Transformer Block operation. In the process of the second first Wiggle Transformer Block operation, the W-MSA module is used to perform window segmentation to generate multiple first windows, and the attention calculation is performed in each of the first windows formed after segmentation.

S460, input the third feature map after the second first Wiggle Transformer Block operation into the Wiggle Transformer Block, and perform the second second Wiggle Transformer Block operation. In the process of the second second Wiggle Transformer Block operation, the WWL-MSA module is used to perform window segmentation to generate multiple second windows, and the attention calculation is performed in each of the second windows formed after segmentation.

S470, input the fourth feature map after the second second Wiggle Transformer Block operation into the Wiggle Transformer Block, carry out the second third Wiggle Transformer Block operation, and use the finally obtained feature map as the fourth feature map. In the process of performing the second third Wiggle Transformer Block operation, the WWR-MSA module is used to perform window segmentation to generate multiple third windows, and the attention calculation is performed in each of the third windows formed after segmentation.

Specifically, in this embodiment, six consecutive WiggleTransformer Block operations are performed instead of three times, but the principle is consistent with the principle of three consecutive WiggleTransformer Block operations, that is, two consecutive WiggleTransformer Block operations, each of which has three consecutive WiggleTransformer Block operations. The WiggleTransformer Block operation.

The number of heads of Multi-Head Attention is 12, and the tensor dimension composed of Q, K, and V is (14, 14, 384). At this time, the number of windows is 14/7×14/7, that is, 2×2=16 windows in total. The resolution of the finally generated fourth feature map is kept at H/16, W/16.

In addition, the steps of S500 are consistent with the principles of the aforementioned S300. First, in the Restructure operation, after splicing each group of 2 × 2 adjacent patch blocks in the fourth feature map, the feature map becomes (7, 7, 1536). After the convolution kernel is 1 × 1, the features Dimension reduced from 1536 to 768.

When the window is split in S500, the number of heads of Multi-Head Attention is 24, and the tensor dimension composed of Q, K, and V is (7, 7, 768). At this time, the number of windows is 7/7×7/7, that is, there is only one window left. The resolution of the final generated fourth feature map is kept at H/32, W/32.

In an embodiment of the present application, the S600 includes the following S610 to S620:

S610: Send the fifth feature map to the RPN network.

S620, perform binary classification on the fifth feature map through the RPN network to obtain a classification loss value.

Specifically, the classification loss value is a value that represents the element type of the occluded part. For example, the classification loss value of birds is 0, the classification loss value of trees is 1, and the classification loss value of human beings is 2, then the element type of the occluded part can be qualitatively determined according to the obtained classification loss value.

In an embodiment of the present application, the S600 further includes the following steps:

S630, perform Bounding Box regression on the fifth feature map through the RPN network to obtain the regression loss value.

Specifically, the regression loss value is also a value.

The present invention designs and proposes a new network module for the complexity and uncertainty of the occlusion target, and adds the WWL-MSA module, WWL-MSA and a 3-layer convolution (convolution layer) structure to overcome the occlusion The occlusion of object-to-target detection is confusing, and the interaction between windows is increased to improve globality and robustness. Calculating the respective attentions in the window has the advantage of not only introducing the limitations of the CNN convolution operation, but also saving the amount of computation. And by merging patches, the resolution can be reduced, the number of channels can be adjusted to form a hierarchical design, and a certain amount of computation can also be saved. It can not only reduce the influence of occluders on the detection target, but also increase the detail features at multiple levels to improve the accuracy of occluded target detection.

The technical features of the above-described embodiments can be combined arbitrarily, and the execution order of each method step is not limited. For the sake of brevity, all possible combinations of the technical features in the above-described embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the description in this specification.

The above-mentioned embodiments only represent several embodiments of the present application, and the descriptions thereof are relatively specific and detailed, but should not be construed as a limitation on the scope of the patent of the present application. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of the present application, some modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims (10)

1. an industrial occlusion target detection method, is characterized in that, described method comprises: Input the to-be-processed image, perform patch segmentation on the to-be-processed image and apply a linear embedding layer for dimensionality reduction to obtain the first feature map; Perform three consecutive Wiggle Transformer Block operations on the first feature map to generate a second feature map; The second feature map is input to the Patch Merging module to perform the Restructure operation, and the second feature map after the Restructure operation is subjected to three consecutive WiggleTransformer Block operations to generate the third feature map; Input the third feature map to the Patch Merging module to perform the Restructure operation, and perform six consecutive WiggleTransformer Block operations on the third feature map after the Restructure operation to generate the fourth feature map; Input the fourth feature map to the Patch Merging module to perform the Restructure operation, and perform three consecutive WiggleTransformer Block operations on the fourth feature map after the Restructure operation to generate the fifth feature map; The fifth feature map is sent to the RPN network, and the occlusion target detection is performed on the fifth feature map through the RPN network, and the occlusion target detection result is output; In three consecutive WiggleTransformer Block operations, the W-MSA module, the WWL-MSA module and the WWR-MSA module are used to perform window segmentation, and attention is calculated in each window formed after segmentation. 2. The method for detecting an industrial occlusion target according to claim 1, wherein the input picture to be processed, the picture to be processed is subjected to patch segmentation and a linear embedding layer is applied for dimension reduction, and the first feature map is obtained, comprising: Input a picture to be processed with height H, width W, and channel number 3; Divide the image to be processed into

Patch blocks of the same size, each patch block is 4 pixels high and 4 pixels wide; Input multiple patch blocks of the same size as the original feature map into the linear embedding layer, and change the feature dimension of the original feature map to the preset dimension C, so as to convert the 2-dimensional original feature map into a 1-dimensional patch slice, and The transformed 1D patch slice is used as the first feature map. 3. The industrial occlusion target detection method according to claim 2, wherein the first feature map is subjected to three consecutive Wiggle Transformer Block operations, and the generation of the second feature map comprises: Input the first feature map into the Wiggle Transformer Block, and perform the first Wiggle Transformer Block operation; in the process of performing the first Wiggle Transformer Block operation, use the W-MSA module to perform window segmentation to generate multiple first windows, and in the process of performing the first Wiggle Transformer Block operation. The attention calculation is performed in the first window formed after each segmentation; Input the first feature map after the first Wiggle Transformer Block operation into the WiggleTransformer Block, and perform the second Wiggle Transformer Block operation; in the process of performing the second WiggleTransformer Block operation, use the WWL-MSA module to perform window segmentation to generate A plurality of second windows, and the attention calculation is performed in the second window formed after each segmentation; Input the first feature map after the second Wiggle Transformer Block operation into the Wiggle Transformer Block, perform the third Wiggle Transformer Block operation, and use the resulting feature map as the second feature map; in the process of performing the third Wiggle Transformer Block operation In , the WWR-MSA module is used for window segmentation, multiple third windows are generated, and attention calculation is performed in each of the third windows formed after segmentation. 4. The industrial occlusion target detection method according to claim 3, wherein the first feature map is input into the Wiggle Transformer Block, and the first Wiggle Transformer Block operation is performed, comprising: Input the first feature map into the Wiggle Transformer Block, and perform the Layer Normalization operation; The feature map after the Layer Normalization operation is divided into the W-MSA module window to generate multiple first windows, and the attention calculation is performed in the first window formed after each segmentation, and the W-MSA module window is generated. Feature map after segmentation and attention calculation; Residual connection is performed between the feature map and the first feature map after the W-MSA module window segmentation and attention calculation; Perform Layer Normalization operation on the feature map obtained after residual connection; Input the feature map after Layer Normalization operation to a 2-layer MLP neural network module for neural network processing; Perform three convolutions on the feature map processed by the neural network; The feature map after three convolutions and the feature map obtained after residual connection are connected again by residual connection, and finally the first feature map after the first Wiggle Transformer Block operation is obtained; the convolution kernels of three convolutions are 3× 3, 5×5 and 1×1. 5. The industrial occlusion target detection method according to claim 4, wherein the first feature map after the operation of the first Wiggle Transformer Block is input into the Wiggle Transformer Block, and the second Wiggle Transformer Block operation is performed, comprising: : Input the first feature map after the operation of the first Wiggle Transformer Block into the Wiggle Transformer Block, and perform the Layer Normalization operation; The feature map after the Layer Normalization operation is divided into the WWL-MSA module window to generate multiple second windows, and the attention calculation is performed in the second window formed after each segmentation, and the WWL-MSA module window is generated. Feature map after segmentation and attention calculation; Residual connection is performed between the feature map after WWL-MSA module window segmentation and attention calculation and the first feature map after the first WiggleTransformer Block operation; Perform Layer Normalization operation on the feature map obtained after residual connection; Input the feature map after Layer Normalization operation to a 2-layer MLP neural network module for neural network processing; Perform three convolutions on the feature map processed by the neural network; The feature map after three convolutions and the feature map obtained after residual connection are connected again by residual connection, and finally the first feature map after the second Wiggle Transformer Block operation is obtained; the convolution kernels of three convolutions are 3× 3, 5×5 and 1×1. 6. The industrial occlusion target detection method according to claim 5, wherein the first feature map after the second Wiggle Transformer Block operation is input into the Wiggle Transformer Block, the third Wiggle Transformer Block operation is performed, and the The final feature map is used as the second feature map, including: Input the first feature map after the second Wiggle Transformer Block operation into the Wiggle Transformer Block, and perform the Layer Normalization operation; The feature map after the Layer Normalization operation is divided into the WWR-MSA module window to generate multiple third windows, and the attention calculation is performed in each of the third windows formed after the segmentation, and the WWR-MSA module window is generated. Feature map after segmentation and attention calculation; Residual connection is performed between the feature map after the WWR-MSA module window segmentation and attention calculation and the first feature map after the second WiggleTransformer Block operation; Perform Layer Normalization operation on the feature map obtained after residual connection; Input the feature map after Layer Normalization operation to a 2-layer MLP neural network module for neural network processing; Perform three convolutions on the feature map processed by the neural network; The feature map after three convolutions and the feature map obtained after residual connection are connected again by residual connection, and finally the second feature map is obtained; the convolution kernels of three convolutions are 3×3, 5×5 and 1×1 respectively. . 7. The industrial occlusion target detection method according to claim 6, wherein the second feature map is input to the Patch Merging module to perform a Restructure operation, comprising: Input the second feature map to the Patch Merging module; In the height direction and width direction of the second feature map, the Patch Merging module selects 2×2 patch blocks at intervals for splicing to generate a spliced second feature map; Perform a convolution with a convolution kernel of 1×1 on the spliced second feature map, and finally generate the second feature map after the Restructure operation. 8. industrial occlusion target detection method according to claim 7, is characterized in that, described the 3rd feature map is input to Patch Merging module to carry out Restructure operation, the 3rd feature map after the Restructure operation is carried out six consecutive times. WiggleTransformer Block operation to generate the fourth feature map, including: Input the third feature map to the Patch Merging module to perform the Restructure operation; Input the third feature map after the Restructure operation into the Wiggle Transformer Block, and perform the first Wiggle Transformer Block operation; in the process of performing the first Wiggle Transformer Block operation, use the W-MSA module to perform window segmentation to generate multiple The first window, and the attention calculation is performed in the first window formed after each segmentation; Input the third feature map after the first Wiggle Transformer Block operation into the WiggleTransformer Block, and perform the second Wiggle Transformer Block operation; in the process of performing the second Wiggle Transformer Block operation, use the WWL-MSA module to perform window segmentation to generate Multiple second windows, and the attention calculation is performed in each second window formed after segmentation; Input the third feature map after the second Wiggle Transformer Block operation into the WiggleTransformer Block, and perform the third Wiggle Transformer Block operation; in the process of performing the third WiggleTransformer Block operation, use the WWR-MSA module to perform window segmentation to generate Multiple third windows, and the attention calculation is performed in the third window formed after each segmentation; Input the third feature map after the third Wiggle Transformer Block operation into the Wiggle Transformer Block, and perform the second first Wiggle Transformer Block operation; in the process of performing the second first Wiggle Transformer Block operation, use W-MSA The module performs window segmentation, generates multiple first windows, and performs attention calculation in the first windows formed after each segmentation; Input the third feature map after the second first Wiggle Transformer Block operation into the Wiggle Transformer Block, and perform the second second Wiggle Transformer Block operation; during the second second Wiggle Transformer Block operation, use The WWL-MSA module performs window segmentation, generates multiple second windows, and performs attention calculation in each of the second windows formed after segmentation; Input the third feature map after the second second Wiggle Transformer Block operation into the WiggleTransformer Block, perform the second third Wiggle Transformer Block operation, and use the resulting feature map as the fourth feature map; In the process of the third Wiggle Transformer Block operation, the WWR-MSA module is used for window segmentation to generate multiple third windows, and the attention calculation is performed in each of the third windows formed after segmentation. 9. industrial occlusion target detection method according to claim 8, is characterized in that, the 5th feature map is sent into RPN network, carries out occlusion target detection to the 5th feature map by RPN network, outputs occlusion target detection result, include: Send the fifth feature map to the RPN network; Binary classification is performed on the fifth feature map through the RPN network to obtain the classification loss value. 10. The industrial occlusion target detection method according to claim 9, wherein the fifth feature map is sent into the RPN network, the occlusion target detection is performed on the fifth feature map through the RPN network, and the occlusion target detection result is output. ,Also includes: Perform Bounding Box regression on the fifth feature map through the RPN network to obtain the regression loss value.

Images