CN114332919A - A pedestrian detection method, device and terminal device based on multi-spatial relationship perception - Google Patents

Abstract

The invention discloses a pedestrian detection method, device and terminal device based on multi-spatial relationship perception. The method includes step 1: collecting a pedestrian image data set and adjusting it to a fixed size for model training; step 2, using a YOLOX detection framework , input the image into the frame model, first perform data enhancement on the image; step 3, input the data-enhanced image into the Focus module, slice the image according to parity, obtain 4 images, and then proceed along the channel direction. Splicing; Step 4, input the spliced image into the backbone network of the YOLOX detection framework, and three branches are connected to the backbone network; Step 5, each branch contains 2 parts, a multi-spatial relationship perception module and a detection head, The multi-spatial relationship-aware module effectively fuses global information and local information together by combining the relationship between features in different spatial dimensions to obtain multi-spatial relationship-aware feature maps. This method not only focuses on global information, but also extracts local information, and effectively fuses the two to obtain more recognizable feature information and improve pedestrian detection performance.

Description

A pedestrian detection method, device and terminal device based on multi-spatial relationship perception

technical field

The invention relates to the field of image recognition research, in particular to a pedestrian detection method, and in particular to a pedestrian detection method, device and terminal device based on multi-spatial relationship perception.

Background technique

With the continuous development of smart city construction, many new artificial intelligence technologies are applied to smart transportation, smart government affairs, smart factories, etc., and each application is inseparable from the masses and serves people. Therefore, pedestrian detection is one of many application technologies. the premise. However, real scenes are often more complex, such as dense crowds causing bodies to overlap or overlap, or being blocked by objects, strong changes in illumination, blurred images caused by harsh weather factors (rain and snow, etc.), etc. These real situations increase the difficulty of pedestrian detection. difficulty. To this end, a pedestrian detection technology is urgently needed, which can dig deeper and discriminative features in the pedestrian area in the image, which is sufficient to characterize pedestrians in various environments.

In the process of realizing the present invention, the inventor found that there are at least the following problems in the prior art: the current popular pedestrian detection technologies are mostly based on convolutional neural networks (CNN), and most CNN pedestrian detection models use a limited receptive field, which is very difficult to achieve. It is difficult to combine global information to learn rich structural patterns, such as using CNN to detect and segment pedestrians to obtain final location information; such as using CNN combined with feature fusion for pedestrian detection; although some methods consider different receptive fields, but no It is a good combination of global and local information; in addition, there are some methods to enhance the learning ability of the model by stacking the network depth, which is very resource-intensive for both training and deployment.

SUMMARY OF THE INVENTION

In order to overcome the deficiencies of the prior art, the present invention provides a pedestrian detection method, device and terminal device based on multi-spatial relationship perception. Thereby, more recognizable feature information can be obtained, and the pedestrian detection performance can be improved. The technical solution is as follows:

The present invention provides a pedestrian detection method based on multi-spatial relationship perception, the method comprising the following steps:

Step 1: Collect a pedestrian image dataset and adjust it to a fixed size for model training.

Step 2, using YOLOX's detection framework, input the image into the framework model, and first perform data enhancement on the image.

Step 3: Input the data-enhanced image into the Focus module, slice the image according to parity, obtain 4 images, and then stitch them along the channel direction.

Step 4: Input the spliced image into the backbone network of the YOLOX detection framework. Three branches are connected to the backbone network. These three branches correspond to different receptive fields, and the three receptive fields can cover targets of different sizes.

Step 5, each branch contains 2 parts, a multi-spatial relationship perception module and a detection head. The multi-spatial relationship perception module effectively fuses global information and local information together by combining the relationship between features in different spatial dimensions, Obtain multi-spatial relation-aware feature maps.

The workflow of the multi-spatial relationship perception module is as follows:

The X dimension of the feature map input to the multi-spatial relationship perception module is H×W×C, where H is height, W is width, and C is the number of channels;

(1) Construct the relational feature map of H×W space;

In the H×W space range, the feature map X is decomposed into H×W feature vectors of length C, and the relationship information of the feature vector x _i mapped to the feature vector x _j is represented by ri _{, j} , and the calculation method is as follows:

和φ_H×W为2个嵌入函数，均由一个1×1的卷积层、一个BatchNormalization层和一个ReLU激活层组成。相应的，特征向量x_j映射到特征向量Jx_i的关系信息为r_j，i＝f_H×W(x_j，x_i)，则(r_i，j，r_j，i)描述了特征向量x_i和x_j之间的双向关系；对于单向关系，计算出所有特征向量之间的关系信息并进行堆叠即可得到一个亲和矩阵

矩阵通道数为H×W，因此双向关系可以得到两个不同的亲和矩阵M1和M2，对特征的局部信息进行深度挖掘。in,

and φ _H×W are 2 embedding functions, each consisting of a 1×1 convolutional layer, a BatchNormalization layer and a ReLU activation layer. Correspondingly, the relationship information of the feature vector x _j mapped to the feature vector Jx _i is r _{j, i} =f _H×W (x _j , x _i ), then (r _{i, j} , r _{j, i} ) describes the feature vector Two-way relationship between x _i and x _j ; for one-way relationship, calculate the relationship information between all eigenvectors and stack them to get an affinity matrix

The number of matrix channels is H×W, so two different affinity matrices M1 and M2 can be obtained from the bidirectional relationship, and the local information of the feature can be deeply mined.

将全局结构特征图F与两个亲和矩阵串联在一起，获得一个特征矩阵Y，公式如下：Retain the original global structure information. Specifically, after performing 1×1 convolution on the original feature map X, perform a global average pooling operation in the channel direction to obtain a global structure feature map F,

Concatenate the global structural feature map F with two affinity matrices to obtain a feature matrix Y, the formula is as follows:

均由一个一个1×1的卷积层、一个BatchNormalization层和一个ReLU激活层组成，相比于

和φ_H×W，它们的输出激活节点数量都是不一样的；将特征矩阵Y通过1×1的卷积，来融合特征矩阵中包含的所有全局和局部信息，从而得到属于片×W空间的关系特征图。pool denotes global average pooling, θ _H×W and

Both consist of a 1×1 convolutional layer, a BatchNormalization layer and a ReLU activation layer, compared to

and φ _H×W , the number of output activation nodes is different; the feature matrix Y is fused by 1×1 convolution to fuse all the global and local information contained in the feature matrix, so as to obtain the space belonging to the slice×W The relational feature map of .

(2) Construct the relational feature map of the channel space C;

Similarly, in the channel space range, the feature map X is decomposed into C feature vectors of length H×W, and the relationship information r _{a and b} of the feature vector x _a mapped to the feature vector x _b are:

阳φ_C函数与

和φ_H×W一致，只是输出维度不同；采用与步骤5(1)相同的计算方式获得亲和矩阵

即双向关系可以得到两个不同的亲和矩阵M′₁和M′₂。in

Yang φ _C function with

Consistent with φ _H×W , but the output dimension is different; use the same calculation method as step 5(1) to obtain the affinity matrix

That is, the bidirectional relationship can obtain two different affinity matrices M' ₁ and M' ₂ .

将结构特征图与两个亲和矩阵串联在一起，获得的特征矩阵Y′，计算方式如下：After performing 1×1 convolution on the original feature map X, perform global average pooling in the H×W dimension to obtain the structural feature map

Concatenate the structural feature map with the two affinity matrices to obtain the feature matrix Y′, which is calculated as follows:

Y'=[pool(θ _C (X)), θ _C (M' ₁ ), θ _C (M' ₂ )].

函数功能与θ_H×W和

一致，只是输出维度不同；将特征矩阵Y′通过1×1的卷积，来融合特征矩阵中包含的所有全局和局部信息，从而得到属于通道空间C的关系特征图。θ _C and

function function with θ _H×W and

Consistent, but the output dimensions are different; the feature matrix Y′ is convolutional by 1×1 to fuse all the global and local information contained in the feature matrix, so as to obtain the relational feature map belonging to the channel space C.

The multi-space relationship-aware feature map is obtained by dot-multiplying the relationship feature map of the H×W space and the channel space C.

Step 6: Put the multi-spatial relationship-aware feature map into the detection head. YOLOX decouples the classification and coordinate positioning. First, the channel is dimensionally reduced by a 1×1 convolution, followed by two lightweight branches, respectively. Perform classification and regression.

Preferably, the data enhancement in step 2 includes random horizontal flipping of the image, color dithering, multi-scale enhancement and mosaic coordinate enhancement methods.

Preferably, the receptive fields corresponding to the three branches in step 4 are down-sampling 8 times, 16 times, and 32 times respectively.

Preferably, in the training phase, the classification loss function adopts cross entropy, the regression loss function adopts GIOU loss, and the L1 norm is used to impose a penalty on the acquired position information.

Compared with the prior art, one of the above technical solutions has the following beneficial effects: through the multi-spatial relationship perception module, the relationship between features and features in different spatial dimensions can be deeply excavated, which not only focuses on the global information, but also Extract local information, and effectively fuse the two to establish a connection between the feature information in different spaces and the relationship information between the features, so that the features learned by the model are more recognizable and discriminative, so as to obtain more recognizable feature information. , to improve the accuracy of pedestrian detection.

Description of drawings

FIG. 1 is a flowchart of a multi-spatial relationship perception module provided by an embodiment of the present disclosure.

Detailed ways

In order to clarify the technical solution and working principle of the present invention, the embodiments of the present disclosure will be further described in detail below with reference to the accompanying drawings. All the above-mentioned optional technical solutions can be combined arbitrarily to form optional embodiments of the present disclosure, which will not be repeated here.

The terms "step 1", "step 2", "step 3" and similar descriptions in the description and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. order. It is to be understood that data so used may be interchanged under appropriate circumstances so that the embodiments of the application described herein can be practiced in sequences other than those described herein.

A first aspect: an embodiment of the present disclosure provides a pedestrian detection method based on multi-spatial relationship perception, and the method includes the following steps:

Step 1: Collect a pedestrian image dataset and adjust it to a fixed size for model training.

In step 2, the detection framework of YOLOX is adopted, which has a simple structure and does not need to manually set anchor frames, which is convenient for training and deployment. Input the image into the frame model, and first perform data enhancement on the image. Preferably, the data enhancement in step 2 includes random horizontal flipping of the image, color jittering, multi-scale enhancement and mosaic coordinate enhancement methods, etc., in order to expand the scale of the training set and improve the model. generalization ability.

Step 3: Input the data-enhanced image into the Focus module, slice the image according to parity, obtain 4 images, and then stitch them along the channel direction. The Focus module performs downsampling without increasing the amount of computation, and retains more complete image information.

Step 4: Input the spliced image into the backbone network of the YOLOX detection framework. Three branches are connected to the backbone network. These three branches correspond to different receptive fields, and the three receptive fields can cover targets of different sizes. Preferably, the receptive fields corresponding to the three branches in step 4 are down-sampling 8 times, 16 times, and 32 times respectively.

Step 5, each branch contains 2 parts, a multi-spatial relationship perception module and a detection head. The multi-spatial relationship perception module effectively fuses global information and local information together by combining the relationship between features in different spatial dimensions, Obtain multi-spatial relation-aware feature maps.

Accompanying drawing 1 is a kind of working flow chart of the multi-spatial relationship perception module, in conjunction with this figure, the work flow of the multi-spatial relationship perception module is as follows:

The X dimension of the feature map input to the multi-spatial relationship perception module is H×W×C, where H is height, W is width, and C is the number of channels;

(1) Construct the relational feature map of H×W space;

In the H×W space range, the feature map X is decomposed into H×W feature vectors of length C, and the relationship information of the feature vector x _i mapped to the feature vector x _j is represented by ri _{, j} , and the calculation method is as follows:

和φ_H×W为2个嵌入函数，均由一个1×1的卷积层、一个BatchNormalization层和一个ReLU激活层组成。相应的，特征向量x_j映射到特征向量x_i的关系信息为r_j，i＝f_H×W(x_j，x_i)，则(r_i，j，r_j，i)描述了特征向量x_i和x_j之间的双向关系；对于单向关系，计算出所有特征向量之间的关系信息并进行堆叠即可得到一个亲和矩阵

矩阵通道数为H×W，因此双向关系可以得到两个不同的亲和矩阵M₁和M₂，对特征的局部信息进行深度挖掘。in,

and φ _H×W are 2 embedding functions, each consisting of a 1×1 convolutional layer, a BatchNormalization layer and a ReLU activation layer. Correspondingly, the relationship information of the feature vector x _j mapped to the feature vector x _i is r _{j, i} = f _H×W (x _j , x _i ), then (r _{i, j} , r _{j, i} ) describes the feature vector Two-way relationship between x _i and x _j ; for one-way relationship, calculate the relationship information between all eigenvectors and stack them to get an affinity matrix

The number of matrix channels is H×W, so two different affinity matrices M ₁ and M ₂ can be obtained from the bidirectional relationship, and the local information of the feature can be deeply mined.

将全局结构特征图F与两个亲和矩阵串联在一起，获得一个特征矩阵Y，公式如下：In order to develop the global information of features at the same time, it is necessary to retain the original global structure information. Specifically, after performing 1×1 convolution on the original feature map X, perform a global average pooling operation in the channel direction to obtain a global structure feature map F,

Concatenate the global structural feature map F with two affinity matrices to obtain a feature matrix Y, the formula is as follows:

Y=[pool(θ _H×W (X)), θ _H×W (M ₁ ), θ _H×W (M ₂ )].

和φ_H×W，它们的输出激活节点数量都是不一样的；将特征矩阵Y通过1×1的卷积，来融合特征矩阵中包含的所有全局和局部信息，从而得到属于H×W空间的关系特征图。pool represents global average pooling. Both θ _H×W and θ _H×W consist of a 1×1 convolutional layer, a BatchNormalization layer and a ReLU activation layer, compared to

and φ _H×W , the number of output activation nodes is different; the feature matrix Y is fused by 1×1 convolution to fuse all the global and local information contained in the feature matrix, so as to obtain the space belonging to H×W The relational feature map of .

(2) Construct the relational feature map of the channel space C;

Similarly, in the channel space range, the feature map X is decomposed into C feature vectors of length H×W, and the relationship information r _{a and b} of the feature vector x _a mapped to the feature vector x _b are:

和φ_C函数与

和φ_H×W一致，只是输出维度不同；采用与步骤5(1)相同的计算方式获得亲和矩阵

即双向关系可以得到两个不同的亲和矩阵M′₁和M′₂；in

and φ _C function and

Consistent with φ _H×W , but the output dimension is different; use the same calculation method as step 5(1) to obtain the affinity matrix

That is, the bidirectional relationship can obtain two different affinity matrices M′ ₁ and M′ ₂ ;

将结构特征图与两个亲和矩阵串联在一起，获得的特征矩阵Y′，计算方式如下：Different from the structural feature map F obtained in step 5, in this step, after 1×1 convolution is performed on the original feature map X, global average pooling is performed in the H×W dimension to obtain the structural feature map

Concatenate the structural feature map with the two affinity matrices to obtain the feature matrix Y′, which is calculated as follows:

函数功能与θ_H×W和

一致，只是输出维度不同；将特征矩阵Y′通过1×1的卷积，来融合特征矩阵中包含的所有全局和局部信息，从而得到属于通道空间C的关系特征图。θ _C and

function function with θ _H×W and

Consistent, but the output dimensions are different; the feature matrix Y′ is convolutional by 1×1 to fuse all the global and local information contained in the feature matrix, so as to obtain the relational feature map belonging to the channel space C.

Dot multiplication of the relational feature maps of H×W space and channel space C to obtain a multi-space relation-aware feature map, which contains the global and local information of features in different spatial dimensions, and fully fuses them to improve the performance. Validity and discriminative power of features.

Step 6: Put the multi-spatial relationship-aware feature map into the detection head. Unlike the traditional YOLO series detection head, which couples classification and coordinate positioning for training, YOLOX decouples classification and coordinate positioning, first through a 1 × 1 The convolution of the channel reduces the dimension of the channel, followed by two lightweight branches for classification and regression respectively, which can effectively improve the convergence speed of the model.

Preferably, in the training phase, the classification loss function adopts cross entropy, the regression loss function adopts GIOU loss, and the L1 norm is used to impose a penalty on the acquired position information.

In a second aspect, embodiments of the present disclosure provide a pedestrian detection device based on multi-spatial relationship perception. Based on the same technical concept, the device can implement or execute the multi-space relationship-based detection device in any of the possible implementation manners. A spatial relationship aware pedestrian detection method.

Preferably, the device includes a data acquisition unit, a first data processing unit, a second data processing unit, and a result acquisition unit;

The data acquisition unit is used to perform the steps of step 1 of a pedestrian detection method based on multi-spatial relationship perception according to any one of all possible implementations;

The first data processing unit is used to execute the steps of step 2 and step 3 of a pedestrian detection method based on multi-spatial relationship perception according to any one of all possible implementations;

The second data processing unit is used to perform the steps of step 4 and step 5 of a pedestrian detection method based on multi-spatial relationship perception described in any one of all possible implementations;

The result obtaining unit is configured to execute the steps of step 6 of the pedestrian detection method based on multi-spatial relationship perception described in any one of the possible implementation manners.

It should be noted that, when the pedestrian detection device based on multi-spatial relationship perception provided in the above embodiment executes a pedestrian detection method based on multi-spatial relationship perception, only the division of the above functional modules is used as an example to illustrate, and the practical application In the device, the above-mentioned function distribution can be completed by different function modules according to the needs, that is, the internal structure of the device is divided into different function modules, so as to complete all or part of the functions described above. In addition, a pedestrian detection device based on multi-spatial relationship perception provided by the above embodiments and a pedestrian detection method based on multi-spatial relationship perception belong to the same concept.

In a third aspect, an embodiment of the present disclosure provides a terminal device, where the terminal device includes the apparatus for detecting pedestrians based on multi-spatial relationship perception according to any one of all possible implementation manners.

The present invention has been exemplarily described above with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited by the above-mentioned methods, and all kinds of insubstantial improvements made by the method concept and technical solution of the present invention are adopted; Improvements, equivalent replacements, and direct application of the above concepts and technical solutions of the present invention to other occasions are all within the protection scope of the present invention.

Claims (7)

1. a pedestrian detection method based on multi-spatial relationship perception, is characterized in that, this method comprises the steps: Step 1, collect the pedestrian image data set, adjust it to a fixed size for model training; Step 2, using the detection framework of YOLOX, input the image into the frame model, and first perform data enhancement on the image; Step 3: Input the image after data enhancement into the Focus module, perform slicing operation on the image according to parity, obtain 4 images, and then splicing along the channel direction; Step 4: Input the spliced image into the backbone network of the YOLOX detection framework. Three branches are connected to the backbone network. These three branches correspond to different receptive fields, and the three receptive fields can cover targets of different sizes; Step 5, each branch contains 2 parts, a multi-spatial relationship perception module and a detection head. The multi-spatial relationship perception module effectively fuses global information and local information together by combining the relationship between features in different spatial dimensions, Obtain multi-spatial relation-aware feature maps; The workflow of the multi-spatial relationship perception module is as follows: The X dimension of the feature map input to the multi-spatial relationship perception module is H×W×C, where H is height, W is width, and C is the number of channels; (1) Construct the relational feature map of H×W space; In the H×W space range, the feature map X is decomposed into H×W feature vectors of length C, and the relationship information of the feature vector x _i mapped to the feature vector x _j is represented by ri _{, j} , and the calculation method is as follows:

in,

and φ _H×W are two embedding functions, which are composed of a 1×1 convolution layer, a BatchNormalization layer and a ReLU activation layer; correspondingly, the relationship information of the feature vector x _j mapped to the feature vector x _i is r _{j, i} = f _H×W (x _j , x _i ), then (r _{i, j} , r _{j, i} ) describes the bidirectional relationship between the feature vectors _xi and x _j ; for the unidirectional relationship, calculate The relationship information between all eigenvectors and stacking can get an affinity matrix

The number of matrix channels is H×W, so two different affinity matrices M ₁ and M ₂ can be obtained from the bidirectional relationship, and the local information of the feature can be deeply mined; Retain the original global structure information. Specifically, after performing 1×1 convolution on the original feature map X, perform a global average pooling operation in the channel direction to obtain a global structure feature map F,

Concatenate the global structural feature map F with two affinity matrices to obtain a feature matrix Y, the formula is as follows:

pool denotes global average pooling, θ _H×W and

Both consist of a 1×1 convolutional layer, a BatchNormalization layer and a ReLU activation layer, compared to

and φ _H×W , the number of output activation nodes is different; the feature matrix Y is convolved by 1×1 to fuse all the global and local information contained in the feature matrix, so as to obtain the space belonging to H×W The relational feature map of ; (2) Construct the relational feature map of the channel space C; Similarly, in the channel space range, the feature map X is decomposed into C feature vectors of length H×W, and the relationship information r _{a and b} of the feature vector x _a mapped to the feature vector x _b are:

in

and φ _C function and

Consistent with φ _H×W , but the output dimension is different; use the same calculation method as step 5(1) to obtain the affinity matrix

That is, the bidirectional relationship can obtain two different affinity matrices M′ ₁ and M′ ₂ ; After performing 1×1 convolution on the original feature map X, perform global average pooling in the H×W dimension to obtain the structural feature map

Concatenate the structural feature map with the two affinity matrices to obtain the feature matrix Y′, which is calculated as follows:

θ _C and

function function with θ _H×W and

Consistent, but the output dimension is different; the feature matrix Y′ is convolutional by 1×1 to fuse all the global and local information contained in the feature matrix, so as to obtain the relational feature map belonging to the channel space C; Multiply the relationship feature map of H×W space and channel space C to obtain the multi-space relationship-aware feature map; Step 6: Put the multi-spatial relationship-aware feature map into the detection head. YOLOX decouples the classification and coordinate positioning. First, the channel is dimensionally reduced by a 1×1 convolution, followed by two lightweight branches, respectively. Perform classification and regression. 2 . The pedestrian detection method based on multi-spatial relationship perception according to claim 1 , wherein the data enhancement in step 2 includes random horizontal flipping of images, color dithering, multi-scale enhancement and mosaic coordinate enhancement methods. 3 . 3 . The pedestrian detection method based on multi-spatial relationship perception according to claim 1 , wherein the receptive fields corresponding to the three branches in step 4 are down-sampling 8 times, 16 times, and 32 times respectively. 4 . 4. A pedestrian detection method based on multi-spatial relationship perception according to any one of claims 1-3, characterized in that, in the training phase, the classification loss function adopts cross entropy, the regression loss function adopts GIOU loss, and uses L1 The norm imposes a penalty on the acquired location information. 5 . A pedestrian detection device based on multi-spatial relationship perception, characterized in that, the device can implement a pedestrian detection method based on multi-spatial relationship perception according to any one of claims 1 to 4 . 6. A pedestrian detection device based on multi-spatial relationship perception according to claim 5, wherein the device comprises a data acquisition unit, a first data processing unit, a second data processing unit, and a result acquisition unit; The data acquisition unit is used to perform the steps of step 1 of the pedestrian detection method based on multi-spatial relationship perception according to any one of claims 1-4; The first data processing unit is used to perform the steps of step 2 and step 3 of a pedestrian detection method based on multi-spatial relationship perception according to any one of claims 1-4; The second data processing unit is used to perform the steps of step 4 and step 5 of the pedestrian detection method based on multi-spatial relationship perception according to any one of claims 1-4; The result obtaining unit is configured to execute the steps of step 6 of the pedestrian detection method based on multi-spatial relationship perception according to any one of claims 1-4. 7. A terminal device, characterized in that the terminal device comprises the pedestrian detection device based on multi-spatial relationship perception according to any one of claims 5 or 6.

Images