CN117036890A - Training of pedestrian detection model, pedestrian detection method, device, equipment and medium - Google Patents

Abstract

The invention discloses a method, a device, equipment and a medium for training a pedestrian detection model and detecting pedestrians. Obtaining a visible light sample image and a thermal infrared sample image aiming at a target pedestrian scene; and performing element-by-element addition operation on the characteristics of the visible light sample image and the characteristics of the thermal infrared sample image to obtain the multi-mode fusion image characteristics. And carrying out feature reconstruction based on the multi-mode fusion image features to obtain visible light reconstruction features and thermal infrared reconstruction features. Determining a first single-mode similarity loss between the visible light extraction feature and the visible light reconstruction feature, a second single-mode similarity loss between the thermal infrared extraction feature and the thermal infrared reconstruction feature, a first multi-mode interaction loss between the thermal infrared reconstruction feature and the visible light extraction feature, and a second multi-mode interaction loss between the visible light reconstruction feature and the thermal infrared extraction feature. And training the initial pedestrian detection model according to the similarity loss and the interaction loss until the model training stopping condition is met, so as to obtain the target pedestrian detection model.

Description

Training of pedestrian detection models, pedestrian detection methods, devices, equipment and media

Technical field

The present invention relates to the field of artificial intelligence technology, and in particular to the training of a pedestrian detection model, pedestrian detection methods, devices, equipment and media.

Background technique

Pedestrian detection models can identify and track pedestrians appearing in images or videos, thereby improving efficiency and accuracy in traffic safety, surveillance, and search and rescue.

In related technologies, multi-modal feature fusion methods (such as splicing different modal features) can be used to detect pedestrians. However, the mutual interference between multi-modal features will affect the accuracy of pedestrian detection.

Contents of the invention

The embodiments of this specification are intended to solve one of the technical problems in the related art, at least to a certain extent. To this end, the embodiments of this specification propose a pedestrian detection model training, pedestrian detection method, device, equipment and medium.

The embodiment of this specification provides a training method for a pedestrian detection model. The method includes:

Obtain visible light sample images and thermal infrared sample images for the target pedestrian scene;

Feature reconstruction is performed based on the multi-modal fusion image features between the visible light sample image and the thermal infrared sample image to obtain visible light reconstruction features and thermal infrared reconstruction features; wherein the multi-modal fusion image features are based on the visible light The characteristics of the sample image and the characteristics of the thermal infrared sample image are obtained by performing an element-by-element addition operation;

Determining a first single-mode similarity loss between the visible light extraction feature of the visible light sample image and the visible light reconstruction feature, and a second single-mode similarity loss between the thermal infrared extraction feature of the thermal infrared sample image and the thermal infrared reconstruction feature. State similarity loss, the first multi-modal interaction loss between the thermal infrared reconstruction feature and the visible light extraction feature, the second multi-modal interaction loss between the visible light reconstruction feature and the thermal infrared extraction feature;

According to the first single-modal similarity loss, the second single-modal similarity loss, the first multi-modal interaction loss and the second multi-modal interaction loss, the initial pedestrian detection model is simultaneously single-modal. Modal supervised training and cross-modal supervised training are performed until the model stop training conditions are met and the target pedestrian detection model is obtained.

In one embodiment, the initial pedestrian detection model includes a parallel first encoding network and a second encoding network, the first encoding network and the second encoding network are jointly connected to a fusion component, and the fusion component is connected to a parallel a first decoding network and a second decoding network; performing feature reconstruction based on multi-modal fusion image features between the visible light sample image and the thermal infrared sample image to obtain visible light reconstruction features and thermal infrared reconstruction features, including:

Input the visible light sample image into the first encoding network for feature extraction to obtain visible light modal features;

Input the thermal infrared sample image into the second encoding network for feature extraction to obtain thermal infrared modal features;

The fusion component performs an element-by-element addition operation on the visible light modal features and the thermal infrared modal features to obtain the multi-modal fused image features;

The multi-modal fusion image features are input into the first decoding network and the second decoding network for feature reconstruction respectively, and the visible light reconstruction features and the thermal infrared reconstruction features are correspondingly obtained.

In one embodiment, the method for determining multi-modal fusion image features includes:

Obtaining the visible light modal characteristics of the visible light sample image and the thermal infrared modal characteristics of the thermal infrared sample image;

An element-by-element addition operation is performed on the thermal infrared modal features and the visible light modal features to obtain the multi-modal fusion image features.

In one embodiment, the acquisition of visible light sample images and thermal infrared sample images for the target pedestrian scene includes:

Obtaining an initial visible light image and an initial thermal infrared image taken for the target pedestrian scene;

The initial visible light image and the initial thermal infrared image are preprocessed according to the input data size of the initial pedestrian detection model to obtain the visible light sample image and the thermal infrared sample image; wherein, after normalization, The visible light sample image and the normalized thermal infrared sample image are used as input data of the initial pedestrian detection model.

In one embodiment, the visible light sample image and the thermal infrared sample image are normalized in the following manner:

According to the first channel mean and the first channel standard deviation corresponding to the visible light sample image, normalize the visible light sample image to obtain input data corresponding to the visible light sample image;

According to the second channel mean value and the second channel standard deviation corresponding to the thermal infrared sample image, the thermal infrared sample image is normalized to obtain input data corresponding to the thermal infrared sample image.

In one implementation, the following loss function is used to calculate model loss data:

L＝L1+L2+L3+L4

L1=MSE( _RGBi , _RGB′i )

L2=MSE (TH _i , TH′ _i )

L3=MSE( _RGBi , _TH′i )

L4=MSE (TH _i , RGB′ _i )

Among them, L is the model loss data, L1 is the first single-modal similarity loss, L2 is the second single-modal similarity loss, L3 is the first multi-modal interaction loss, L4 is the second multi-modal interaction loss, RGB _i is the visible light extraction feature, RGB′ _i is the visible light reconstruction feature, TH _i is the thermal infrared extraction feature, TH′ _i is the thermal infrared reconstruction feature, i represents the feature count number, and MSE represents the mean square error loss.

The embodiment of this specification provides a pedestrian detection method, which method includes:

Obtain visible light inspection images and thermal infrared inspection images taken for any pedestrian scene;

The visible light image to be detected and the visible light image to be detected are input into the target pedestrian detection model obtained by the method in any of the above embodiments to perform pedestrian detection and obtain a pedestrian detection result.

The embodiment of this specification provides a pedestrian detection method, which method includes:

Obtain visible light inspection images and thermal infrared inspection images taken for any pedestrian scene;

The visible light image to be detected and the visible light image to be detected are input into the target pedestrian detection model for pedestrian detection, and pedestrian detection results are obtained; wherein the loss data of the target pedestrian detection model during the training process includes single-modal loss and Cross-modal loss; the single-modal loss and the cross-modal loss are used together to supervise the training of the target pedestrian detection model to extract the multi-modal fusion between the visible light image to be detected and the thermal infrared image to be detected. Image feature capabilities;

The single-modal loss includes a first single-modal similarity loss between the visible light extraction features and the visible light reconstruction features of the visible light sample image, and a second single-modal similarity loss between the thermal infrared extraction features and the thermal infrared reconstruction features of the thermal infrared sample image. similar losses;

The cross-modal loss includes a first multi-modal interaction loss between the thermal infrared reconstruction feature and the visible light extraction feature, and a second multi-modal interaction loss between the visible light reconstruction feature and the thermal infrared extraction feature. .

In one embodiment, the target pedestrian detection model includes a parallel first encoding network and a second encoding network, the first encoding network and the second encoding network are jointly connected to a fusion component; the visible light to be detected is The image and the visible light image to be detected are input into the target pedestrian detection model for pedestrian detection, and pedestrian detection results are obtained, including:

Input the visible light image to be detected into the first encoding network for feature extraction to obtain visible light features to be detected;

Input the thermal infrared image to be inspected into the second encoding network for feature extraction to obtain the thermal infrared image to be inspected;

The fusion component performs an element-by-element addition operation on the visible light features to be inspected and the thermal infrared features to be inspected to obtain the fused image features to be inspected;

Target detection is performed based on the fused image features to be detected, and the pedestrian detection result is obtained.

In one embodiment, performing pedestrian detection based on the fused image features to be detected to obtain the pedestrian detection result includes:

Convolution, pooling, and activation processing operations are performed according to the fused image features to be detected to perform target detection and obtain the pedestrian detection result.

The embodiment of this specification provides a training device for a pedestrian detection model. The device includes:

The sample image acquisition module is used to acquire visible light sample images and thermal infrared sample images of the target pedestrian scene;

An image feature reconstruction module, configured to perform feature reconstruction based on the multi-modal fusion image features between the visible light sample image and the thermal infrared sample image, to obtain visible light reconstruction features and thermal infrared reconstruction features; wherein, the multi-modal fusion The image features are obtained by element-by-element addition based on the features of the visible light sample image and the features of the thermal infrared sample image;

Loss data determination module, used to determine the first single-modal similarity loss between the visible light extraction feature of the visible light sample image and the visible light reconstruction feature, the thermal infrared extraction feature of the thermal infrared sample image and the thermal infrared reconstruction feature The second single-modal similarity loss between features, the first multi-modal interaction loss between the thermal infrared reconstruction feature and the visible light extraction feature, the second multi-modal interaction loss between the visible light reconstruction feature and the thermal infrared extraction feature Modal interaction loss;

a detection model determination module, configured to determine the initial The pedestrian detection model performs single-modal supervised training and cross-modal supervised training at the same time until the model stop training conditions are met, and the target pedestrian detection model is obtained.

The embodiment of this specification provides a pedestrian detection device, which includes:

The image acquisition module to be inspected is used to acquire the visible light image to be inspected and the thermal infrared image to be inspected that are captured for any pedestrian scene;

The detection result determination module is used to input the visible light image to be detected and the visible light image to be detected into the target pedestrian detection model obtained in any of the above embodiments to perform pedestrian detection and obtain pedestrian detection results.

The embodiment of this specification provides a pedestrian detection device, which includes:

The image acquisition module to be inspected is used to acquire the visible light image to be inspected and the thermal infrared image to be inspected that are captured for any pedestrian scene;

The detection result determination module is used to input the visible light image to be detected and the visible light image to be detected into a target pedestrian detection model for pedestrian detection, and obtain pedestrian detection results; wherein, the loss of the target pedestrian detection model during the training process The data includes single-modal loss and cross-modal loss; the single-modal loss and the cross-modal loss are used together to supervise the training of the target pedestrian detection model to extract the visible light image to be detected and the thermal infrared image to be detected. The ability of multi-modal fusion of image features between images;

The single-modal loss includes a first single-modal similarity loss between the visible light extraction features and the visible light reconstruction features of the visible light sample image, and a second single-modal similarity loss between the thermal infrared extraction features and the thermal infrared reconstruction features of the thermal infrared sample image. similar losses;

The cross-modal loss includes a first multi-modal interaction loss between the thermal infrared reconstruction feature and the visible light extraction feature, and a second multi-modal interaction loss between the visible light reconstruction feature and the thermal infrared extraction feature. .

Embodiments of this specification provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented.

An embodiment of this specification provides a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device can perform the method described in any of the above embodiments. step.

An embodiment of this specification provides a chip that includes a storage unit and a processing unit. The storage unit stores a computer program. When the processing unit executes the computer program, it implements the steps of the method described in any of the above embodiments.

In the implementation of the above description, first, a visible light sample image and a thermal infrared sample image for the target pedestrian scene are obtained; then, based on the characteristics of the visible light sample image and the characteristics of the thermal infrared sample image, an element-by-element addition operation is performed to obtain a multi-modal fusion image Features, and perform feature reconstruction based on multi-modal fusion image features to obtain visible light reconstruction features and thermal infrared reconstruction features; finally, based on the first single-modal similarity loss between the visible light extraction features of the visible light sample image and the visible light reconstruction features, thermal infrared The second single-modal similarity loss between the thermal infrared extraction features and the thermal infrared reconstruction features of the sample image, the first multi-modal interaction loss between the thermal infrared reconstruction features and the visible light extraction features, and the first multi-modal interaction loss between the visible light reconstruction features and the thermal infrared extraction features. The second multi-modal interaction loss is to perform single-modal supervised training and cross-modal supervised training on the initial pedestrian detection model at the same time to obtain the target pedestrian detection model. On the one hand, preliminary feature fusion is achieved through element-by-element addition operations to obtain multi-modal fusion image features, reducing the mutual interference between the two modal features of visible light images and thermal infrared images; on the other hand, through the first single modal The four losses of similarity loss, second single-modal similarity loss, first multi-modal interaction loss and second multi-modal interaction loss optimize and update the model parameters, making the model capable of obtaining more accurate multi-modal fusion image features. ability, which can improve the quality of multi-modal fusion image features, thereby improving the accuracy of pedestrian detection results.

Description of the drawings

Figure 1a is a schematic diagram of the training initial pedestrian detection model based on the YOLOv5 framework provided by the implementation of this specification;

Figure 1b is a schematic diagram of the target pedestrian detection model based on the YOLOv5 framework provided by the implementation of this specification;

Figure 1c is a schematic flowchart of the training method of the pedestrian detection model provided by the embodiment of this specification;

Figure 2a is a schematic flowchart of obtaining visible light reconstruction features and thermal infrared reconstruction features provided by the embodiment of this specification;

Figure 2b is a schematic diagram of the structure of the first coding network provided by the embodiment of this specification;

Figure 2c is a schematic diagram of the CBS module provided by the embodiment of this specification;

Figure 2d is a schematic diagram of the CSP1_X module provided by the implementation of this specification;

Figure 2e is a schematic diagram of the Res unit module provided by the embodiment of this specification;

Figure 2f is a schematic diagram of the CSP2_X module provided by the implementation of this specification;

Figure 2g is a schematic diagram of the SPPF module provided in the embodiment of this specification;

Figure 2h is a schematic diagram of determining multi-modal fusion image features provided by the embodiment of this specification;

Figure 2i is a schematic diagram of a fusion component provided by an embodiment of this specification;

Figure 2j is a schematic diagram of the first decoding network provided by the embodiment of this specification;

Figure 3 is a schematic flowchart of obtaining multi-modal fusion image features provided by the embodiment of this specification;

Figure 4 is a schematic flowchart of obtaining a visible light sample image and a thermal infrared sample image provided by the embodiment of this specification;

Figure 5 is a schematic flowchart of the pedestrian detection method provided by the embodiment of this specification;

Figure 6 is a schematic flow chart of the pedestrian detection method provided by the embodiment of this specification;

Figure 7 is a schematic flowchart of obtaining pedestrian detection results provided by the embodiment of this specification;

Figure 8 is a schematic flow chart of the training method of the pedestrian detection model provided by the embodiment of this specification;

Figure 9 is a schematic diagram of the training device of the pedestrian detection model provided by the embodiment of this specification;

Figure 10 is a schematic diagram of a pedestrian detection device provided in an embodiment of this specification;

Figure 11 is a schematic diagram of a pedestrian detection device provided in an embodiment of this specification;

FIG. 12 is an internal structure diagram of a computer device provided in an embodiment of this specification.

Detailed ways

Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the drawings are exemplary and are intended to explain the present invention and are not to be construed as limiting the present invention.

In the task of pedestrian detection using convolutional neural networks, the depth of the convolutional neural network has an important impact on the performance of the model. In the process of using multi-modal data for pedestrian detection, when the number of convolutional neural network layers is increased, the convolutional neural network can extract more complex feature patterns. Therefore, when the model extracts multi-modal features for pedestrian detection, Better results can be achieved.

However, with the extraction of multi-modal features by convolutional neural networks, multi-modal features will interfere with each other, and the detection accuracy of convolutional neural networks will become saturated or even decline. The residual network uses short-circuit connections to solve the problem of saturation or even decline in accuracy. However, the residual network does congruent mapping, which transfers redundant features to subsequent convolutional layers. The redundant information will affect Pedestrian detection causes interference and affects the detection of pedestrians.

In related technologies, feature fusion methods can directly use fused features for pedestrian detection. However, the quality of fused features in related technologies may be low, and because the feature fusion methods in related technologies do not take into account the selectivity of features, they cannot fully utilize the complementary features between multi-modal data, thus affecting the performance of pedestrian detection. Accuracy. Therefore, it is necessary to improve the quality of fused features when using multi-modal features for pedestrian detection.

Based on this, embodiments of this specification provide a training method for a pedestrian detection model. First, obtain the visible light sample image and thermal infrared sample image for the target pedestrian scene; then, perform an element-by-element addition operation based on the characteristics of the visible light sample image and the characteristics of the thermal infrared sample image to obtain multi-modal fusion image features; and based on the multi-modal State fusion image features are used for feature reconstruction to obtain visible light reconstruction features and thermal infrared reconstruction features. Finally, according to the first single-modal similarity loss between the visible light extraction features and the visible light reconstruction features of the visible light sample image, the second single-modal similarity loss between the thermal infrared extraction features and the thermal infrared reconstruction features of the thermal infrared sample image, the thermal infrared The first multi-modal interaction loss between the reconstructed features and the visible light extraction features, the second multi-modal interaction loss between the visible light reconstruction features and the thermal infrared extraction features; according to the first single-modal similarity loss, the second single-modal similarity loss , the first multi-modal interaction loss and the second multi-modal interaction loss, perform single-modal supervised training and cross-modal supervised training on the initial pedestrian detection model at the same time until the model stop training conditions are met, and the target pedestrian detection model is obtained.

In the implementation of this specification, on the one hand, preliminary feature fusion is achieved through element-by-element addition operations to obtain multi-modal fusion image features, reducing mutual interference between the two modal features of visible light images and thermal infrared images; on the other hand, The model parameters are optimized and updated through four losses: the first single-modal similarity loss, the second single-modal similarity loss, the first multi-modal interaction loss, and the second multi-modal interaction loss, so that the model has the ability to obtain more accurate multi-modal parameters. The ability of modal fusion image features can improve the quality of multi-modal fusion image features, thereby improving the accuracy of pedestrian detection results.

The embodiment of this specification provides a scenario example of a pedestrian detection model training method. The camera collects the initial visible light image (RGB) and the initial thermal infrared image (Thermal) of the actual application scene, and the camera uploads the initial visible light image and the initial thermal infrared image to the server. The server performs image preprocessing on the initial visible light image and the initial thermal infrared image to obtain an RGB image and a Thermal image. The server side can build a training sample set, and each training sample in the training sample set includes visible light images and thermal infrared images.

In the model training stage, RGB images and Thermal images are input into the initial pedestrian detection model. Please refer to Figure 1a. The target pedestrian detection model includes an encoder 106, an encoder 108, a Fusion module 110, a decoder 112, and a decoder 114. By inputting the RGB image 102 to the encoder 106 for feature extraction, the visible light modal features of the RGB image 102 can be obtained. When passing through the corresponding convolution module of the encoder 106, the visible light extraction features of the RGB image 102 can be obtained. By inputting the Thermal image 104 to the encoder 108 for feature extraction, the thermal infrared modal features of the Thermal image 104 can be obtained. When passing through the corresponding convolution module of the encoder 108, the thermal infrared extraction features of the Thermal image 104 can be obtained. Through the Fusion module, visible light modal features and thermal infrared modal features can be fused in a pixel-by-pixel addition manner to obtain multi-modal fusion image features. The multi-modal fusion image features are input to the decoder 112 and the decoder 114 respectively. The decoder 112 can realize the reconstruction of the RGB image data and obtain the visible light reconstruction features. The decoder 114 can realize the reconstruction of the Thermal image data. Thermal infrared reconstruction features can be obtained. By comparing the similarity between visible light extraction features and visible light reconstruction features, the first single-mode similarity loss Loss1 can be obtained. Through similarity comparison between thermal infrared extracted features and thermal infrared reconstructed features, the second single-modal similarity loss Loss2 can be obtained. Through interactive supervision of visible light extraction features and thermal infrared reconstruction features, the first multi-modal interactive loss Loss3 can be obtained. Through interactive supervision of thermal infrared extraction features and visible light reconstruction features, the second multi-modal interaction loss Loss4 can be obtained. The loss data can be obtained by adding the single-modal similarity loss and the multi-modal interaction loss. Through the loss data, the parameters of the initial pedestrian detection model can be updated until the model stop training conditions are met, and the target pedestrian detection model is obtained. Exemplarily, the encoder 106, the encoder 108, and the decoding are based on the first single-modal similarity loss Loss1, the second single-modal similarity loss Loss2, the first multi-modal interaction loss Loss3, and the second multi-modal interaction loss Loss4. The parameters of the decoder 112 and the decoder 114 are updated. It should be noted that the Fusion module 110 performs preliminary fusion (ie, pixel-by-pixel addition) of visible light modal features and thermal infrared modal features, and the Fusion module 110 does not need to update parameters.

In the model inference stage, the visible light image to be detected and the thermal infrared image to be detected are input into the target pedestrian detection model. Referring to Figure 1b, the target pedestrian detection model includes a parameter-updated encoder 128, a parameter-updated encoder 130, a Fusion module 110, and a CONV module (Standard convolution mudule, standard convolution module) 132. The visible light image 124 to be detected can be used as the input of the encoder 128, and the visible light characteristics to be detected of the visible light image 124 to be detected can be obtained through the encoder 128, and the thermal infrared image 126 to be detected can be used as the input of the encoder 130, and the encoder 130 can The thermal infrared features to be detected of the thermal infrared image to be detected are obtained. Through the Fusion module (fusion module) 110, the visible light features to be inspected and the thermal infrared features to be inspected are feature fused to obtain the fused image features to be inspected. The fused image features to be detected can be subjected to a convolution processing operation through the CONV module 132 to perform target detection and obtain pedestrian detection results 134 .

The embodiment of this specification provides a training method for a pedestrian detection model. Please refer to Figure 1c. The method may include the following steps:

S110. Obtain visible light sample images and thermal infrared sample images of the target pedestrian scene.

Among them, visible light sample images can provide texture details with high spatial resolution and clarity in a manner consistent with the human visual system, capturing reflected light. For example, the visible light sample image can be an RGB image. The RGB image has three channels and contains red, green, and blue visible light color information. Thermal infrared sample images can distinguish targets from their background based on differences in thermal radiation, which works well in all-weather and all-day/night conditions. The thermal infrared sample image has only one channel, which contains the intensity information of near-infrared light. The thermal infrared sample image can be a grayscale image. From the perspective of imaging principles, the wavelength ranges of visible light sample images and thermal infrared sample images are also different, and different sharpness and lighting conditions may produce very different effects on the two types of images. Infrared rays can detect the heat energy emitted by the human body, and the heat of pedestrians will show obvious features in the thermal infrared sample image. This allows thermal infrared sample images to more accurately determine whether the target is a pedestrian in some situations, especially at night or in low-light environments. Therefore, thermal infrared sample images play a very important role in pedestrian detection and can improve the accuracy and reliability of pedestrian detection. The visible light sample image and the thermal infrared sample image can be sample images of the target pedestrian scene at the same time and in the same scene. The visible light sample image and the thermal infrared sample image can be used as a pair of samples to train the initial pedestrian detection model to obtain the target pedestrian detection. Model.

Specifically, the server locally stores a training sample set, and the visible light sample image and thermal infrared sample image of the target pedestrian scene are directly obtained from the training sample set. In other embodiments, the target pedestrian scene can be continuously photographed through an image acquisition device to obtain a visible light image and a thermal infrared image, and the obtained visible light image and thermal infrared image are sent to the server, and the server performs cropping based on the visible light image and the thermal infrared image. Processing or data enhancement processing to obtain visible light sample images and thermal infrared sample images of the target pedestrian scene.

For example, the visible light sample image may be a 640*640*3 image, and the thermal infrared sample image may be a 640*640*1 image.

It should be noted that the image acquisition device may be at least one of a video camera, a still camera, an infrared camera, a fisheye camera, etc.

S120. Perform feature reconstruction based on the multi-modal fusion image features between the visible light sample image and the thermal infrared sample image, and obtain visible light reconstruction features and thermal infrared reconstruction features.

Among them, the multi-modal fusion image features are obtained by element-by-element addition based on the features of the visible light sample image and the features of the thermal infrared sample image. Feature reconstruction usually refers to using some algorithms or models to extract important information representing different features from the original data, and then using this information to reconstruct the data.

In some cases, in pedestrian detection tasks, simply using visible light sample images for detection will be affected by various factors, such as cloudy days, rainy days, haze, etc. At night and in low-light environments, traditional visible light image acquisition equipment cannot obtain clear images, so it is difficult to detect pedestrians through visible light sample images, while infrared image acquisition equipment can use infrared rays to illuminate targets, thereby detecting pedestrians at night and in low light. Obtaining bright images in the environment can improve the accuracy of pedestrian detection. Infrared sample images can penetrate haze, smog, rain, snow and other weather conditions and will not be affected by ambient light. Therefore, in extreme weather, pedestrians can be accurately detected through infrared sample images. Infrared images and visible light images are complementary. Feature fusion based on the characteristics of the visible light sample image and the characteristics of the thermal infrared sample image can obtain multi-modal fusion image features with strong robustness and large amount of information. This paper adopts a multi-modal method that fuses visible light images and thermal infrared images to achieve the purpose of improving pedestrian detection.

Specifically, the features of the visible light sample image and the features of the thermal infrared sample image are added element by element to obtain multi-modal fusion image features. The decoder corresponding to the visible light mode uses the multi-modal fusion image features to perform feature reconstruction, and obtains the visible light reconstruction features. The decoder corresponding to the thermal infrared modality uses the multi-modal fusion image features to perform feature reconstruction, and obtains the thermal infrared reconstruction features.

For example, the visible light reconstruction feature may be a 320*320*32 image, and the thermal infrared sample image may be a 320*320*32 image.

S130. Determine the first single-modal similarity loss between the visible light extraction features and the visible light reconstruction features of the visible light sample image, the second single-modal similarity loss between the thermal infrared extraction features and the thermal infrared reconstruction features of the thermal infrared sample image, and the thermal infrared The first multi-modal interaction loss between the reconstructed features and the visible light extraction features, and the second multi-modal interaction loss between the visible light reconstruction features and the thermal infrared extraction features.

The visible light extraction features may be features of visible light sample images extracted through convolution operations. Thermal infrared extraction features may be features of thermal infrared sample images extracted through convolution operations. Unimodal similarity loss is a loss function used to train models, which measures the similarity between different samples within the same category and can promote the clustering of similar samples in the feature space. Multimodal interaction loss is a loss function used to train multimodal deep learning models. It achieves cross-modal interaction and joint modeling by combining information from different modal data (such as different types of images). The different modes in this embodiment may be visible light image mode and thermal infrared image mode.

In some cases, in complex and changeable environments, the two modalities of data, visible light sample images and thermal infrared sample images, have their own advantages and disadvantages. For example, in an environment with sufficient lighting, visible light sample images have good practicality, while thermal infrared sample images may not greatly improve the quality of fused features. Considering the importance of mutual supervision between the two extracted features and the existence of complementary features in multi-modal data, the model is used to more effectively extract multi-modal fusion image features for pedestrian detection by optimizing the interactive supervision loss function. Single-modal similarity loss can be used to optimize the model's extraction of visible light features and thermal infrared features.

Specifically, by selecting an appropriate single-modal similarity loss function, the first single-modal similarity loss is determined based on the visible light extraction features and the visible light reconstruction features of the visible light sample image. The second single-modal similarity loss is determined based on the thermal infrared extraction features and thermal infrared reconstruction features of the thermal infrared sample image. By selecting an appropriate multi-modal interaction loss function, the first multi-modal interaction loss is determined based on thermal infrared reconstruction features and visible light extraction features, and the second multi-modal interaction loss is determined based on visible light reconstruction features and thermal infrared extraction features.

In some implementations, the visible light sample image is subjected to feature extraction through the convolution module corresponding to the first encoding network, and the visible light extraction features of the visible light sample image can be obtained. The thermal infrared sample image is feature extracted through the convolution module corresponding to the second encoding network, and the thermal infrared extraction features of the thermal infrared sample image can be obtained. The size of visible light extraction features can be 320*320*32, and the size of thermal infrared extraction features can be 320*320*32.

S140. According to the first single-modal similarity loss, the second single-modal similarity loss, the first multi-modal interaction loss, and the second multi-modal interaction loss, simultaneously perform single-modal supervised training and cross-modal training on the initial pedestrian detection model. State-supervised training is performed until the model stop training conditions are met, and the target pedestrian detection model is obtained.

Among them, single-modal supervised training is a machine learning method that uses labeled data from a single data source for model training. This method is typically applied to input data of only one type (e.g., images) and can be used to identify and classify various patterns and features through supervised learning algorithms. Cross-modal supervised training refers to using label information in one modality (such as one type of image) to assist the learning of another modality (such as another type of image). In this way, model performance can be improved in the absence of sufficient labeled data, and the model can learn more comprehensive features from multiple modalities.

In some cases, by training the initial pedestrian detection model through multi-modal interaction loss, higher-quality multi-modal fusion image features can be obtained, and the improved-quality multi-modal fusion image features can be used for pedestrian detection work. Improve the accuracy of pedestrian detection. The quality of visible light extraction features and thermal infrared extraction features can be improved through single-modal similarity loss. Furthermore, by improving the quality of visible light extraction features and thermal infrared extraction features, the quality of multi-modal fusion image features can be improved.

Specifically, the parameters of the initial pedestrian detection model are updated based on the first single-modal similarity loss and the second single-modal similarity loss to achieve single-modal supervised training. The parameters of the initial pedestrian detection model are updated based on the first multi-modal interaction loss and the second multi-modal interaction loss to achieve cross-modal supervised training. By analogy, continue to train the updated initial pedestrian detection model. When the model training stop condition is reached, the target pedestrian detection model can be obtained. Among them, the stopping condition of model training can be that the model loss data tends to converge, or it can be that the training rounds reach a preset number of rounds.

In the above embodiment, first, a visible light sample image and a thermal infrared sample image for the target pedestrian scene are obtained; then, based on the characteristics of the visible light sample image and the characteristics of the thermal infrared sample image, an element-by-element addition operation is performed to obtain the multi-modal fusion image features , and perform feature reconstruction based on multi-modal fusion image features to obtain visible light reconstruction features and thermal infrared reconstruction features; finally, based on the first single-modal similarity loss between the visible light extraction features and visible light reconstruction features of the visible light sample image, the thermal infrared sample The second single-modal similarity loss between the thermal infrared extraction features and the thermal infrared reconstruction features of the image, the first multi-modal interaction loss between the thermal infrared reconstruction features and the visible light extraction features, and the third multi-modal interaction loss between the visible light reconstruction features and the thermal infrared extraction features. Second, multi-modal interaction loss is used to perform single-modal supervised training and cross-modal supervised training on the initial pedestrian detection model at the same time to obtain the target pedestrian detection model. On the one hand, preliminary feature fusion is achieved through element-by-element addition operations to obtain multi-modal fusion image features, reducing the mutual interference between the two modal features of visible light images and thermal infrared images; on the other hand, through the first single modal The four losses of similarity loss, second single-modal similarity loss, first multi-modal interaction loss and second multi-modal interaction loss optimize and update the model parameters, making the model capable of obtaining more accurate multi-modal fusion image features. ability, which can improve the quality of multi-modal fusion image features, thereby improving the accuracy of pedestrian detection results.

In some embodiments, please refer to Figure 2a. The initial pedestrian detection model includes a parallel first encoding network and a second encoding network. The first encoding network and the second encoding network are jointly connected to a fusion component. The fusion component is connected to a parallel third encoding network. a decoding network and a second decoding network. Feature reconstruction is performed based on the multi-modal fusion image features between visible light sample images and thermal infrared sample images to obtain visible light reconstruction features and thermal infrared reconstruction features, which may include the following steps:

S210. Input the visible light sample image into the first encoding network for feature extraction to obtain visible light modal features.

S220. Input the thermal infrared sample image into the second encoding network for feature extraction to obtain thermal infrared modal features.

Among them, the encoding network is a deep learning neural network used to convert high-dimensional input data (such as images, audio, or text) into low-dimensional representations for more effective analysis and processing.

In some cases, the parallel first encoding network and the second encoding network can be a dual-path feature extraction network. The dual-path feature extraction network is a neural network structure in which there are two input paths, each path contains the same or different feature extraction layers, and finally the outputs of the two paths are added pixel by pixel to form the final network output. This structure can effectively initially fuse different types of features, reduce the interference of features between different modalities, and improve the performance and robustness of the model.

Specifically, the visible light sample image is input into the first coding network for feature extraction. Through the convolution operation in the first coding network, the features of the visible light sample image modality can be extracted to obtain the visible light modality features. The thermal infrared sample image is input into the second encoding network for feature extraction. Through the convolution operation in the second encoding network, the modal features of the thermal infrared sample image can be extracted to obtain the thermal infrared modal features.

For example, the initial pedestrian detection model may be a cross-modal supervision model (Cross-Modal Supervision, CMS) of visible light sample images and thermal infrared sample images based on the YOLOv5 framework. Refer to Figure 2b. Figure 2b shows the structure of the first encoding network. The structure of the first encoding network 210 consists of a CBS module (Content-Based Switching, content-based switching network) and a CSP1_X module (Cross Stage Partial Network). , CSP2_X module, SPPF module (Fast Spatial Pyramid Pooling Mudule, fast spatial pyramid pooling module). Referring to Figure 2c, the CBS module 220 is composed of a Conv module (Standard convolution mudule, standard convolution module), a BN module (Batch Normalization, batch normalization module), and a SILU module (Sigmoid-WeightedLinear Unit, adaptive linear rectification unit). , the SILU activation function is a variant of the swish activation function, and the formula of the SILU activation function is SILU(X)=X.sigmoid(X). Referring to Figure 2d, the CSP1_X module 230 is composed of a CBS module, a Res unit module (Residual Unit, residual unit), a Concat module (Concatenation, a connection module), a BN module, and a Silu module. Referring to Figure 2e, the Res unit module 240 is composed of a CBS module and an ADD module (Address Decoder). Referring to Figure 2f, the CSP2_X module 250 is composed of a CBS module, a Concat module (Concatenation, connection module), a BN module and a Silu module. Referring to Figure 2g, the SPPF module 260 is composed of a CBS module, a Maxpool module (MaximumPooling, a maximum pooling module), and a Concat module (Concatenation, a connection module). It should be noted that the structural composition of the second encoding network is consistent with the structural composition of the first encoding network.

For example, a visible light sample image with a size of 640*640*3 pixels can be input into the first encoding network for feature extraction, and a visible light modality feature with a size of 16*16*512 pixels can be obtained. The thermal infrared sample image with a size of 640*640*1 pixels can be input into the second encoding network for feature extraction, and a thermal infrared modal feature with a size of 16*16*512 elements can be obtained.

S230: Perform an element-by-element addition operation on the visible light modal features and the thermal infrared modal features through the fusion component to obtain multi-modal fusion image features.

In some cases, the prediction performance of the pedestrian detection model can be improved by initially fusing visible light modal features and thermal infrared modal features, and fusing features from different sources can reduce the possibility of overfitting and increase the generalizability of the pedestrian detection model. ization ability. It should be noted that the multi-modal fusion image features in this embodiment are simple and preliminary multi-modal additive fusion features, which are obtained by element-by-element addition of visible light modal features and thermal infrared modal features.

Specifically, by fusing components, elements at the characteristic positions of the visible light modality Elements in the same position as the thermal infrared modal signature /> An addition operation is performed to fuse the visible light modal features and the thermal infrared modal features to obtain the multi-modal fusion image feature U.

For example, referring to FIG. 2h, the visible light modality feature 270 may include four elements, namely A ₁ , A ₂ , A ₃ , and A ₄ . The element values of A ₁ , A ₂ , A ₃ and A ₄ are 35, 67, 24 and 48 respectively. The thermal infrared modal signature 280 may include four elements, namely B ₁ , B ₂ , B ₃ , and B ₄ . The element values of B ₁ , B ₂ , B ₃ and B ₄ are 15, 29, 7 and 36 respectively. Adding the element value 35 of element A ₁ of the visible light modality feature and the element value 15 of element B ₁ of the thermal infrared modality feature can obtain the element value 50 of element C ₁ of the multi-modal fusion image feature 290 . Adding the element value 67 of element A ₂ of the visible light modality feature and the element value 29 of element B ₂ of the thermal infrared modality feature can obtain the element value 96 of element C ₂ of the multi-modal fusion image feature 290 . Adding the element value 24 of element A ₃ of the visible light modality feature and the element value 7 of element B ₃ of the thermal infrared modality feature can obtain the element value 31 of element C ₃ of the multi-modal fusion image feature 290 . The element value 48 of element A ₄ of the visible light modality feature and the element value 36 of element B ₄ of the thermal infrared modality feature are added together to obtain the element value 84 of element C ₄ of the multi-modal fusion image feature 230 .

In some implementations, referring to Figure 2i, the fusion component may be a Fusion module. The visible light modal features 202 and the thermal infrared modal features 204 are input to the Fusion module 206. Through the Fusion module 206, the visible light modal features and the thermal infrared modal features can be added pixel by pixel to obtain the multi-modal fusion image feature 208. .

S240. Input the multi-modal fusion image features into the first decoding network and the second decoding network for feature reconstruction respectively, and correspondingly obtain visible light reconstruction features and thermal infrared reconstruction features.

In some cases, feature reconstruction of multi-modal fused image features is achieved through the first decoding network and the second decoding network, and the difference between visible light modal features and thermal infrared modal features can be accurately identified, improving the efficiency of feature extraction. quality.

Specifically, the multi-modal fusion image features obtained by the pixel-by-pixel addition operation are input into the first decoding network, and up-sampling, convolution and other operations are performed through the first decoding network to obtain visible light reconstruction features. The multi-modal fusion image features obtained by the pixel-by-pixel addition operation are input into the second decoding network, and through the second decoding network, upsampling, convolution and other operations are performed to obtain thermal infrared reconstruction features.

For example, referring to Figure 2j, the first decoding network 212 may be composed of a CBS module and an upsampling module. The composition of the second decoding network structure is the same as the composition of the first decoding network structure. The multi-modal fusion image features obtained by the pixel-by-pixel addition operation are input to the first decoding network to obtain visible light reconstruction features. The multi-modal fusion image features obtained by the pixel-by-pixel addition operation are input to the second decoding network to obtain thermal infrared reconstruction features.

In the above embodiment, by determining the visible light modal features, thermal infrared modal features, visible light reconstruction features, and thermal infrared reconstruction features, the single-modal similarity loss and multi-modal interaction loss of the initial pedestrian detection model can be determined, thereby optimizing the initial pedestrian detection. Parameters for model feature extraction to improve the quality of feature extraction.

In some implementations, please refer to Figure 3. The method of determining multi-modal fusion image features may include the following steps:

S310. Obtain the visible light modal characteristics of the visible light sample image and the thermal infrared modal characteristics of the thermal infrared sample image.

S320. Perform an element-by-element addition operation on the thermal infrared modal features and the visible light modal features to obtain multi-modal fusion image features.

Specifically, visible light modal features are obtained by performing feature extraction on visible light sample images. By performing feature extraction on the thermal infrared sample image, the thermal infrared modal features are obtained. Then, the element at the position of the visible light modal feature and the element at the same position of the thermal infrared modal feature can be added, and the element-by-element addition operation between the visible light modal feature and the thermal infrared modal feature can be realized to obtain Multimodal fusion image features.

In the above embodiment, the visible light modal features of the visible light sample image and the thermal infrared modal features of the thermal infrared sample image are obtained, and the thermal infrared modal features and the visible light modal features are added element by element to obtain a multi-modal fusion image. feature. By fusing visible light modal features and thermal infrared modal features, the prediction performance of the pedestrian detection model can be improved, and fusing features from different sources can reduce the possibility of overfitting and increase the generalization ability of the pedestrian detection model.

In some implementations, please refer to Figure 4, the training method of the pedestrian detection model may also include the following steps:

S410. Obtain the initial visible light image and the initial thermal infrared image captured for the target pedestrian scene.

S420. Preprocess the initial visible light image and the initial thermal infrared image according to the input data size of the initial pedestrian detection model to obtain a visible light sample image and a thermal infrared sample image.

Among them, the normalized visible light sample image and the normalized thermal infrared sample image are used as input data of the initial pedestrian detection model.

In some cases, the pixel data of the initial visible light image and the initial thermal infrared image are normalized so that they fall within a specific range, usually between 0 and 1. Through normalization processing, the dimensional differences between different variables can be eliminated for better comparison and analysis, and the consumption of computing resources can also be saved.

Specifically, the initial visible light image and the initial thermal infrared image can be obtained by photographing the target pedestrian scene in a real application scenario through an image acquisition device. The initial visible light image and the initial thermal infrared image appear in pairs, and each target pedestrian scene can have at least one pair of initial visible light image and initial thermal infrared image. According to the input data size of the initial pedestrian detection model, image preprocessing (such as cropping or scaling) is performed on the initial visible light sample image and the initial thermal infrared sample image to obtain the visible light sample image and thermal image of the target pedestrian scene. Infrared sample image.

For example, the input data size of the initial pedestrian detection model may be 640*640. The initial visible light image and the initial thermal infrared image can be preprocessed by proportional scaling or cropping. The image size of the initial visible light image and the initial thermal infrared image is converted to 640*640 through preprocessing.

In the above embodiment, the initial visible light image and the initial thermal infrared image captured for the target pedestrian scene are obtained, and the initial visible light image and the initial thermal infrared image are preprocessed according to the input data size of the initial pedestrian detection model to obtain the visible light sample image and Thermal infrared sample image. By preprocessing the initial visible light image and the initial thermal infrared image, input data can be provided for subsequent feature extraction.

In some embodiments, the visible light sample image and the thermal infrared sample image are normalized in the following manner:

According to the first channel mean and the first channel standard deviation corresponding to the visible light sample image, the visible light sample image is normalized to obtain input data corresponding to the visible light sample image.

According to the second channel mean value and the second channel standard deviation corresponding to the thermal infrared sample image, the thermal infrared sample image is normalized to obtain input data corresponding to the thermal infrared sample image.

The channel mean can be the sum of the pixel values of all pixels on any channel divided by the number of pixels. Channel standard deviation can be a measure of the dispersion of pixels in any one channel. Channel standard deviation measures the difference between the pixel value of each pixel and the mean, and is the square root of the variance. The larger the standard deviation, the more spread out the pixel values are, and vice versa.

Specifically, the visible light sample image has an R channel, a G channel, and a B channel. For any one of the three RGB channels, use the pixel data on any channel to first subtract the mean value of the first channel on any channel, and then divide it by the standard deviation of the first channel on any channel, we can get The input data corresponding to the visible light sample image on any channel. For the thermal infrared sample image, the input data corresponding to the thermal infrared sample image can be obtained by first subtracting the second channel mean value from the pixel data of the thermal infrared sample image, and then dividing it by the second channel standard deviation.

In the above embodiment, through normalization processing, dimensional differences between different variables can be eliminated to facilitate better comparison and analysis, and can also save the consumption of computing resources.

In some implementations, the following loss function is used to calculate model loss data:

L＝L1+L2+L3+L4

L1=MSE( _RGBi , _RGB′i )

L2=MSE (TH _i , TH′ _i )

L3=MSE( _RGBi , _TH′i )

L4=MSE (TH _i , RGB′ _i )

Among them, L is the model loss data, L1 is the first single-modal similarity loss, L2 is the second single-modal similarity loss, L3 is the first multi-modal interaction loss, L4 is the second multi-modal interaction loss, RGB _i is the visible light extraction feature, RGB′ _i is the visible light reconstruction feature, TH _i is the thermal infrared extraction feature, TH′ _i is the thermal infrared reconstruction feature, i represents the feature count number, and MSE represents the mean square error loss.

Among them, the mean square error loss is the average sum of the squared errors of the corresponding points in the predicted data and the original data.

Specifically, the visible light reconstruction features and the visible light extraction features are compared for similarity, and the mean square error loss function MSE is used to iterate the parameters of the initial pedestrian detection model to minimize the loss between the visible light reconstruction features and the visible light extraction features, and a more effective algorithm can be obtained. Fusion features. The first single-modal similarity loss formula between visible light reconstruction features and visible light extraction features is as follows:

L1=MSE( _RGBi , _RGB′i )

Compare the similarity between thermal infrared reconstruction features and thermal infrared extraction features, use the mean square error loss function MSE, iterate the parameters of the initial pedestrian detection model, and minimize the loss between the thermal infrared reconstruction features and thermal infrared extraction features, which can be more effective. fusion characteristics. The second single-modal similarity loss formula between thermal infrared reconstructed features and thermal infrared extracted features is as follows:

L2=MSE (TH _i , TH′ _i )

Visible light extraction features and thermal infrared reconstruction features are interactively supervised, and the two modal data features are used for loss calculation. Using the mean square error loss function MSE to iterate the parameters of the initial pedestrian detection model and minimize the loss between visible light extraction features and thermal infrared reconstruction features, more effective fusion features can be obtained. The first multimodal interaction loss formula between visible light extraction features and thermal infrared reconstruction features is as follows:

L3=MSE( _RGBi , _TH′i )

Thermal infrared extraction features and visible light reconstruction features are interactively supervised, and the two modal data features are used for loss calculation. Using the mean square error loss function MSE to iterate the parameters of the initial pedestrian detection model to minimize the loss between thermal infrared extraction features and visible light reconstruction features, more effective fusion features can be obtained. The second multimodal interaction loss formula between thermal infrared extraction features and visible light reconstruction features is as follows:

L4=MSE (TH _i , RGB′ _i )

By adding the single-modal similarity loss and the multi-modal interaction loss, the model loss data can be obtained, and the loss can be iteratively optimized to obtain the target pedestrian detection model. The formula for model loss data is as follows:

L＝L1+L2+L3+L4

In the above embodiment, by determining the first single-modal similarity loss between the visible light extraction features and the visible light reconstruction features of the visible light sample image, and the second single-modal similarity between the thermal infrared extraction features and the thermal infrared reconstruction features of the thermal infrared sample image. Loss, visible light extraction features and thermal infrared extraction features can be optimized. By determining the first multi-modal interaction loss between thermal infrared reconstruction features and visible light extraction features, and the second multi-modal interaction loss between visible light reconstruction features and thermal infrared extraction features, the quality of the fused features can be improved, thereby improving pedestrian detection results. accuracy.

The embodiment of this specification provides a pedestrian detection method. Please refer to Figure 5. The method may include the following steps:

S510: Obtain the visible light image to be inspected and the thermal infrared image to be inspected that are captured for any pedestrian scene.

S520: Input the visible light image to be detected and the visible light image to be detected into the target pedestrian detection model obtained in any of the above embodiments to perform pedestrian detection and obtain a pedestrian detection result.

Specifically, by photographing any pedestrian scene through an image acquisition device, an initial visible light image to be inspected and an initial thermal infrared image to be inspected can be obtained. The initial visible light image to be detected and the initial thermal infrared image to be detected are input into the target pedestrian detection model, and pedestrian detection is implemented through the target pedestrian detection model to obtain the pedestrian detection results.

In the above embodiments, the visible light image to be inspected and the thermal infrared image to be inspected that are captured for any pedestrian scene are acquired, and the visible light image to be inspected and the visible light image to be inspected are input into the target pedestrian detection model obtained in any of the above embodiments. Perform pedestrian detection and obtain pedestrian detection results. Pedestrian detection can be applied to vehicle assisted driving systems, intelligent video surveillance, robots, aerial images, human-computer interaction systems, motion analysis, etc.

The embodiment of this specification provides a pedestrian detection method. Please refer to Figure 6. The method may include the following steps:

S610: Obtain the visible light image to be inspected and the thermal infrared image to be inspected that are captured for any pedestrian scene.

S620: Input the visible light image to be detected and the visible light image to be detected into the target pedestrian detection model for pedestrian detection, and obtain pedestrian detection results.

Among them, the loss data of the target pedestrian detection model during the training process includes single-modal loss and cross-modal loss. Single-modal loss and cross-modal loss are used together to supervise the ability of the training target pedestrian detection model to extract multi-modal fusion image features between the visible light image to be detected and the thermal infrared image to be detected. The single-modal loss includes the first single-modal similarity loss between the visible light extraction features and the visible light reconstruction features of the visible light sample image, and the second single-modal similarity loss between the thermal infrared extraction features and the thermal infrared reconstruction features of the thermal infrared sample image. . The cross-modal loss includes the first multi-modal interaction loss between thermal infrared reconstruction features and visible light extraction features, and the second multi-modal interaction loss between visible light reconstruction features and thermal infrared extraction features.

Specifically, by photographing any pedestrian scene through an image acquisition device, an initial visible light image to be inspected and an initial thermal infrared image to be inspected can be obtained. By performing image preprocessing on the initial visible light image to be detected and the initial thermal infrared image to be detected, the visible light image to be detected and the thermal infrared image to be detected can be obtained with the same size as the input data of the target pedestrian detection model. The visible light image to be inspected and the thermal infrared image to be inspected after image preprocessing are input to the target pedestrian detection model, and pedestrian detection is implemented through the target pedestrian detection model to obtain the pedestrian detection results.

In the above embodiment, the visible light image to be inspected and the thermal infrared image to be inspected that are captured for any pedestrian scene are acquired, and the visible light image to be inspected and the visible light image to be inspected are input into the target pedestrian detection model for pedestrian detection, and the pedestrian detection results are obtained. . Pedestrian detection can be applied to vehicle assisted driving systems, intelligent video surveillance, robots, aerial images, human-computer interaction systems, motion analysis, etc.

In some implementations, please refer to FIG. 7 , the target pedestrian detection model includes a first encoding network and a second encoding network connected in parallel, and the first encoding network and the second encoding network are jointly connected to the fusion component. Input the visible light image to be detected and the visible light image to be detected into the target pedestrian detection model for pedestrian detection, and obtain the pedestrian detection results, which may include the following steps:

S710. Input the visible light image to be inspected into the first encoding network for feature extraction to obtain the visible light image to be inspected.

S720: Input the thermal infrared image to be inspected into the second encoding network for feature extraction to obtain the thermal infrared image to be inspected.

S730: Use the fusion component to perform an element-by-element addition operation on the visible light features to be inspected and the thermal infrared features to be inspected, to obtain the fused image features to be inspected.

S740: Perform target detection based on the fusion image features to be detected, and obtain pedestrian detection results.

Specifically, the visible light image to be detected is input into the first coding network included in the target pedestrian detection model for feature extraction to obtain the visible light features to be detected, and the thermal infrared image to be detected is input into the second coding network included in the target pedestrian detection model. Perform feature extraction to obtain thermal infrared features to be inspected. Through the fusion component included in the target pedestrian detection model, the visible light features to be detected and the thermal infrared features to be detected are added element by element to obtain the fused image features to be detected. Target detection is performed on the fused image features to be examined, and pedestrian detection results are obtained.

In the above embodiment, the visible light image to be inspected is input into the first encoding network for feature extraction to obtain the visible light inspection features, and the thermal infrared image to be inspected is input into the second encoding network for feature extraction to obtain the thermal infrared features to be inspected. , through the fusion component, the visible light features to be detected and the thermal infrared features to be detected are added element by element to obtain the fusion image features to be detected, and target detection is performed based on the fusion image features to be detected to obtain pedestrian detection results. Pedestrian detection can be applied to vehicle assisted driving systems, intelligent video surveillance, robots, aerial images, human-computer interaction systems, motion analysis, etc.

In some embodiments, performing pedestrian detection based on the fused image features to be detected and obtaining the pedestrian detection results may include: performing convolution, pooling, and activation processing operations based on the fused image features to be detected to perform target detection and obtain the pedestrian detection results. .

Specifically, the fusion image features to be detected are input into the convolution layer for convolution processing, and the convolution processing result is obtained. The convolution processing results are input into the pooling layer for pooling processing, and the pooling processing results are obtained. The pooling processing results are activated through the activation layer for target detection to obtain pedestrian target results. For example, activation processing can be implemented through a sigmoid function.

In the above embodiment, convolution, pooling, and activation processing operations are performed according to the fused image features to be detected to perform target detection and obtain pedestrian detection results. Pedestrian detection can be applied to vehicle assisted driving systems, intelligent video surveillance, robots, aerial images, human-computer interaction systems, motion analysis, etc.

The embodiment of this specification also provides a training method for a pedestrian detection model. The initial pedestrian detection model includes a parallel first encoding network and a second encoding network. The first encoding network and the second encoding network are jointly connected to the fusion component, and the fusion component is connected There is a first decoding network and a second decoding network connected in parallel. For example, please refer to Figure 8. The training method of the pedestrian detection model may include the following steps:

S802. Obtain the initial visible light image and the initial thermal infrared image captured for the target pedestrian scene.

S804. Preprocess the initial visible light image and the initial thermal infrared image according to the input data size of the initial pedestrian detection model to obtain a visible light sample image and a thermal infrared sample image.

S806. Input the visible light sample image into the first encoding network for feature extraction to obtain visible light modal features.

S808. Input the thermal infrared sample image into the second encoding network for feature extraction to obtain thermal infrared modal features.

S810: Perform an element-by-element addition operation on the visible light modal features and the thermal infrared modal features through the fusion component to obtain multi-modal fusion image features.

S812. Input the multi-modal fusion image features into the first decoding network and the second decoding network for feature reconstruction respectively, and correspondingly obtain visible light reconstruction features and thermal infrared reconstruction features.

S814. Determine the first single-modal similarity loss between the visible light extraction features and the visible light reconstruction features of the visible light sample image, the second single-modal similarity loss between the thermal infrared extraction features and the thermal infrared reconstruction features of the thermal infrared sample image, and the thermal infrared The first multi-modal interaction loss between the reconstructed features and the visible light extraction features, and the second multi-modal interaction loss between the visible light reconstruction features and the thermal infrared extraction features.

Specifically, the following loss function is used to calculate the model loss data:

L＝L1+L2+L3+L4

L1=MSE( _RGBi , _RGB′i )

L2=MSE (TH _i , TH′ _i )

L3=MSE( _RGBi , _TH′i )

L4=MSE (TH _i , RGB′ _i )

Among them, L is the model loss data, L1 is the first single-modal similarity loss, L2 is the second single-modal similarity loss, L3 is the first multi-modal interaction loss, L4 is the second multi-modal interaction loss, RGB _i is the visible light modal feature, RGB′ _i is the visible light reconstruction feature, TH _i is the thermal infrared modal feature, TH′ _i is the thermal infrared reconstruction feature, i represents the feature count number, and MSE represents the mean square error loss.

S816. According to the first single-modal similarity loss, the second single-modal similarity loss, the first multi-modal interaction loss, and the second multi-modal interaction loss, simultaneously perform single-modal supervised training and cross-modal training on the initial pedestrian detection model. State-supervised training is performed until the model stop training conditions are met, and the target pedestrian detection model is obtained.

The embodiment of this specification provides a training device 900 for a pedestrian detection model. Please refer to Figure 9. The training device 900 for a pedestrian detection model includes: a sample image acquisition module 910, an image feature reconstruction module 920, a loss data determination module 930, and a detection model determination module. Module 940.

The sample image acquisition module 910 is used to acquire visible light sample images and thermal infrared sample images for the target pedestrian scene;

The image feature reconstruction module 920 is configured to perform feature reconstruction based on the multi-modal fusion image features between the visible light sample image and the thermal infrared sample image to obtain visible light reconstruction features and thermal infrared reconstruction features; wherein, the multi-modal The fused image features are obtained by performing an element-by-element addition operation based on the features of the visible light sample image and the features of the thermal infrared sample image;

Loss data determination module 930 is used to determine the first single-modal similarity loss between the visible light extraction feature of the visible light sample image and the visible light reconstruction feature, the thermal infrared extraction feature of the thermal infrared sample image and the thermal infrared The second single-modal similarity loss between the reconstructed features, the first multi-modal interaction loss between the thermal infrared reconstructed features and the visible light extraction features, the second between the visible light reconstructed features and the thermal infrared extracted features Multimodal interaction loss;

The detection model determination module 940 is configured to determine the first single-modal similarity loss, the second single-modal similarity loss, the first multi-modal interaction loss, and the second multi-modal interaction loss. The initial pedestrian detection model undergoes single-modal supervised training and cross-modal supervised training at the same time until the model stop training conditions are met, and the target pedestrian detection model is obtained.

The embodiment of this specification provides a pedestrian detection device 1000. Please refer to FIG. 10. The pedestrian detection device 1000 includes: an image acquisition module 1010 and a detection result determination module 1020.

The image to be inspected acquisition module 1010 is used to acquire the visible light image to be inspected and the thermal infrared image to be inspected that are captured for any pedestrian scene;

The detection result determination module 1020 is used to input the visible light image to be detected and the visible light image to be detected into the target pedestrian detection model obtained in any of the above embodiments to perform pedestrian detection and obtain pedestrian detection results.

The embodiment of this specification provides a pedestrian detection device 1100. Please refer to Figure 11. The pedestrian detection device 1100 includes: an image acquisition module 1110 to be detected, a detection result determination module 1120, a similarity loss determination module 1130, and an interaction loss determination module 1140.

The image to be inspected acquisition module 1110 is used to acquire the visible light image to be inspected and the thermal infrared image to be inspected that are captured for any pedestrian scene;

The detection result determination module 1120 is used to input the visible light image to be detected and the visible light image to be detected into a target pedestrian detection model for pedestrian detection, and obtain pedestrian detection results; wherein, the target pedestrian detection model is used during the training process. The loss data includes single-modal loss and cross-modal loss; the single-modal loss and the cross-modal loss are used together to supervise the training of the target pedestrian detection model to extract the visible light image to be detected and the thermal infrared image to be detected. The ability to detect multi-modal fusion image features between images;

The single-modal loss includes a first single-modal similarity loss between the visible light extraction features and the visible light reconstruction features of the visible light sample image, and a second single-modal similarity loss between the thermal infrared extraction features and the thermal infrared reconstruction features of the thermal infrared sample image. similar losses;

The cross-modal loss includes a first multi-modal interaction loss between the thermal infrared reconstruction feature and the visible light extraction feature, and a second multi-modal interaction loss between the visible light reconstruction feature and the thermal infrared extraction feature. .

For a detailed description of the training device of the pedestrian detection model and the pedestrian detection device, please refer to the description of the training method of the pedestrian detection model and the pedestrian detection method above, and will not be described again here.

In some embodiments, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. 12 . The computer device includes a processor, memory, communication interface, display screen and input device connected through a system bus. Wherein, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage medium stores operating systems and computer programs. This internal memory provides an environment for the execution of operating systems and computer programs in non-volatile storage media. The communication interface of the computer device is used for wired or wireless communication with external terminals. The wireless mode can be implemented through WIFI, operator network, NFC (Near Field Communication) or other technologies. The computer program, when executed by a processor, implements a method for training a pedestrian detection model. The display screen of the computer device may be a liquid crystal display or an electronic ink display. The input device of the computer device may be a touch layer covered on the display screen, or may be a button, trackball or touch pad provided on the computer device shell. , it can also be an external keyboard, trackpad or mouse, etc.

Those skilled in the art can understand that the structure shown in Figure 12 is only a block diagram of a partial structure related to the solution disclosed in this specification, and does not constitute a limitation on the computer equipment to which the solution disclosed in this specification is applied. Specifically, A computer device may include more or fewer components than shown in the figures, or some combinations of components, or have a different arrangement of components.

An embodiment of this specification provides a chip, which includes a storage unit and a processing unit. The storage unit stores a computer program. When the processing unit executes the computer program, it implements the steps of the method in any of the above embodiments.

In some embodiments, a computer device is provided, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, it implements the method steps in the above embodiments.

The embodiments of this specification provide a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in any of the above embodiments are implemented.

One embodiment of the present specification provides a computer program product. The computer program product includes instructions. When the instructions are executed by a processor of a computer device, the computer device can perform the steps of the method of any of the above embodiments.

It should be noted that the logic and/or steps represented in the flowcharts or otherwise described herein, for example, may be considered to be a sequenced list of executable instructions for implementing logical functions, which may be embodied in any computer. in a readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can retrieve and execute instructions from the instruction execution system, apparatus, or device) Used by instruction execution systems, devices or equipment. For the purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections with one or more wires (electronic device), portable computer disk cartridges (magnetic device), random access memory (RAM), Read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium may even be paper or other suitable medium on which the program may be printed, as the program may be printed, for example, by optical scanning of the paper or other medium, followed by editing, interpretation, or in other suitable manner if necessary Processing to obtain a program electronically and then store it in computer memory.

Claims (20)

1. A training method for a pedestrian detection model, characterized in that the method includes: Obtain visible light sample images and thermal infrared sample images for the target pedestrian scene; Feature reconstruction is performed based on the multi-modal fusion image features between the visible light sample image and the thermal infrared sample image to obtain visible light reconstruction features and thermal infrared reconstruction features; wherein the multi-modal fusion image features are based on the visible light The characteristics of the sample image and the characteristics of the thermal infrared sample image are obtained by performing an element-by-element addition operation; Determining a first single-mode similarity loss between the visible light extraction feature of the visible light sample image and the visible light reconstruction feature, and a second single-mode similarity loss between the thermal infrared extraction feature of the thermal infrared sample image and the thermal infrared reconstruction feature. State similarity loss, the first multi-modal interaction loss between the thermal infrared reconstruction feature and the visible light extraction feature, the second multi-modal interaction loss between the visible light reconstruction feature and the thermal infrared extraction feature; According to the first single-modal similarity loss, the second single-modal similarity loss, the first multi-modal interaction loss and the second multi-modal interaction loss, the initial pedestrian detection model is simultaneously single-modal. Modal supervised training and cross-modal supervised training are performed until the model stop training conditions are met and the target pedestrian detection model is obtained. 2. The method of claim 1, wherein the initial pedestrian detection model includes a parallel first encoding network and a second encoding network, the first encoding network and the second encoding network are jointly connected to a fusion component. , the fusion component is connected with a parallel first decoding network and a second decoding network; the feature reconstruction is performed based on the multi-modal fusion image features between the visible light sample image and the thermal infrared sample image to obtain visible light reconstruction features and thermal infrared reconstruction features, including: Input the visible light sample image into the first encoding network for feature extraction to obtain visible light modal features; Input the thermal infrared sample image into the second encoding network for feature extraction to obtain thermal infrared modal features; The fusion component performs an element-by-element addition operation on the visible light modal features and the thermal infrared modal features to obtain the multi-modal fused image features; The multi-modal fusion image features are input into the first decoding network and the second decoding network for feature reconstruction respectively, and the visible light reconstruction features and the thermal infrared reconstruction features are correspondingly obtained. 3. The method according to claim 1, characterized in that the method for determining the multi-modal fusion image features includes: Obtaining the visible light modal characteristics of the visible light sample image and the thermal infrared modal characteristics of the thermal infrared sample image; An element-by-element addition operation is performed on the thermal infrared modal features and the visible light modal features to obtain the multi-modal fusion image features. 4. The method according to claim 1, characterized in that, the method further comprises: Obtaining an initial visible light image and an initial thermal infrared image taken for the target pedestrian scene; The initial visible light image and the initial thermal infrared image are preprocessed according to the input data size of the initial pedestrian detection model to obtain the visible light sample image and the thermal infrared sample image; wherein, after normalization, The visible light sample image and the normalized thermal infrared sample image are used as input data of the initial pedestrian detection model. 5. The method according to claim 4, characterized in that the visible light sample image and the thermal infrared sample image are normalized in the following manner: According to the first channel mean and the first channel standard deviation corresponding to the visible light sample image, normalize the visible light sample image to obtain input data corresponding to the visible light sample image; According to the second channel mean value and the second channel standard deviation corresponding to the thermal infrared sample image, the thermal infrared sample image is normalized to obtain input data corresponding to the thermal infrared sample image. 6. The method according to any one of claims 1 to 5, characterized in that the following loss function is used to calculate the model loss data: L＝L1+L2+L3+L4 L1＝MSE(RGB _i ,RGB′ _i ) L2＝MSE(TH _i ,TH′ _i ) L3＝MSE(RGB _i ,TH′ _i ) L4＝MSE(TH _i ,RGB′ _i ) Among them, L is the model loss data, L1 is the first single-modal similarity loss, L2 is the second single-modal similarity loss, L3 is the first multi-modal interaction loss, L4 is the second multi-modal interaction loss, RGB _i is the visible light extraction feature, RGB′ _i is the visible light reconstruction feature, TH _i is the thermal infrared extraction feature, TH′ _i is the thermal infrared reconstruction feature, i represents the feature count number, and MSE represents the mean square error loss. 7. A pedestrian detection method, characterized in that the method includes: Obtain visible light inspection images and thermal infrared inspection images taken for any pedestrian scene; The visible light image to be detected and the visible light image to be detected are input into the target pedestrian detection model obtained by the method described in any one of claims 1 to 6 for pedestrian detection, and a pedestrian detection result is obtained. 8. A pedestrian detection method, characterized in that the method includes: Obtain visible light inspection images and thermal infrared inspection images taken for any pedestrian scene; The visible light image to be detected and the visible light image to be detected are input into the target pedestrian detection model for pedestrian detection, and pedestrian detection results are obtained; wherein the loss data of the target pedestrian detection model during the training process includes single-modal loss and cross-modal loss; the single-modal loss and the cross-modal loss are used together to supervise the training of the target pedestrian detection model to extract the multi-modality between the visible light image to be detected and the thermal infrared image to be detected. The ability to fuse image features; The single-modal loss includes a first single-modal similarity loss between the visible light extraction features and the visible light reconstruction features of the visible light sample image, and a second single-modal similarity loss between the thermal infrared extraction features and the thermal infrared reconstruction features of the thermal infrared sample image. similar losses; The cross-modal loss includes a first multi-modal interaction loss between the thermal infrared reconstruction feature and the visible light extraction feature, and a second multi-modal interaction loss between the visible light reconstruction feature and the thermal infrared extraction feature. . 9. The method of claim 8, wherein the target pedestrian detection model includes a parallel first encoding network and a second encoding network, the first encoding network and the second encoding network are jointly connected to a fusion component. ; Inputting the visible light image to be detected and the visible light image to be detected into a target pedestrian detection model for pedestrian detection, and obtaining pedestrian detection results, including: Input the visible light image to be detected into the first encoding network for feature extraction to obtain visible light features to be detected; Input the thermal infrared image to be inspected into the second encoding network for feature extraction to obtain the thermal infrared image to be inspected; The fusion component performs an element-by-element addition operation on the visible light features to be inspected and the thermal infrared features to be inspected to obtain the fused image features to be inspected; Target detection is performed based on the fused image features to be detected, and the pedestrian detection result is obtained. 10. The method according to claim 9, characterized in that, performing pedestrian detection based on the fused image features to be detected to obtain the pedestrian detection result includes: Convolution, pooling, and activation processing operations are performed according to the fused image features to be detected to perform target detection and obtain the pedestrian detection result. 11. A training device for pedestrian detection model, characterized in that the device includes: The sample image acquisition module is used to acquire visible light sample images and thermal infrared sample images of the target pedestrian scene; An image feature reconstruction module, configured to perform feature reconstruction based on the multi-modal fusion image features between the visible light sample image and the thermal infrared sample image, to obtain visible light reconstruction features and thermal infrared reconstruction features; wherein, the multi-modal fusion The image features are obtained by element-by-element addition based on the features of the visible light sample image and the features of the thermal infrared sample image; Loss data determination module, used to determine the first single-modal similarity loss between the visible light extraction feature of the visible light sample image and the visible light reconstruction feature, the thermal infrared extraction feature of the thermal infrared sample image and the thermal infrared reconstruction feature The second single-modal similarity loss between features, the first multi-modal interaction loss between the thermal infrared reconstruction feature and the visible light extraction feature, the second multi-modal interaction loss between the visible light reconstruction feature and the thermal infrared extraction feature Modal interaction loss; a detection model determination module, configured to determine the initial The pedestrian detection model performs single-modal supervised training and cross-modal supervised training at the same time until the model stop training conditions are met, and the target pedestrian detection model is obtained. 12. The device according to claim 11, wherein the initial pedestrian detection model includes a first encoding network and a second encoding network in parallel, the first encoding network and the second encoding network are jointly connected to the fusion component , the fusion component is connected to a parallel first decoding network and a second decoding network; the image feature reconstruction module is also used to input the visible light sample image into the first encoding network for feature extraction to obtain visible light Modal features; input the thermal infrared sample image into the second encoding network for feature extraction to obtain thermal infrared modal features; use the fusion component to combine the visible light modal features and the thermal infrared modal features. The features are added element by element to obtain the multi-modal fused image features; the multi-modal fused image features are input into the first decoding network and the second decoding network for feature reconstruction respectively, corresponding to The visible light reconstruction signature and the thermal infrared reconstruction signature. 13. The device according to claim 11, wherein the device further comprises a fusion feature determination module for obtaining the visible light modality feature of the visible light sample image and the thermal infrared modality of the thermal infrared sample image. Features: Perform an element-by-element addition operation on the thermal infrared modal features and the visible light modal features to obtain the multi-modal fusion image features. 14. The device according to any one of claims 11 to 13, characterized in that the following loss function is used to calculate the model loss data: L＝L1+L2+L3+L4 L1＝MSE(RGB _i ,RGB′ _i ) L2＝MSE(TH _i ,TH′ _i ) L3＝MSE(RGB _i ,TH′ _i ) L4＝MSE(TH _i ,RGB′ _i ) Among them, L is the model loss data, L1 is the first single-modal similarity loss, L2 is the second single-modal similarity loss, L3 is the first multi-modal interaction loss, L4 is the second multi-modal interaction loss, RGB _i is the visible light extraction feature, RGB′ _i is the visible light reconstruction feature, TH _i is the thermal infrared extraction feature, TH′ _i is the thermal infrared reconstruction feature, i represents the feature count number, and MSE represents the mean square error loss. 15. A pedestrian detection device, characterized in that the device includes: The image acquisition module to be inspected is used to acquire the visible light image to be inspected and the thermal infrared image to be inspected that are captured for any pedestrian scene; Detection result determination module, used to input the visible light image to be detected and the visible light image to be detected into the target pedestrian detection model obtained by the method according to any one of claims 1 to 6 to perform pedestrian detection and obtain pedestrian detection results. . 16. A pedestrian detection device, characterized in that the device includes: The image acquisition module to be inspected is used to acquire the visible light image to be inspected and the thermal infrared image to be inspected that are captured for any pedestrian scene; The detection result determination module is used to input the visible light image to be detected and the visible light image to be detected into a target pedestrian detection model for pedestrian detection, and obtain pedestrian detection results; wherein, the target pedestrian detection model during the training process The loss data includes single-modal loss and cross-modal loss; the single-modal loss and the cross-modal loss are used together to supervise the training of the target pedestrian detection model to extract the visible light image to be detected and the thermal infrared image to be detected. The ability to detect multi-modal fusion image features between images; The single-modal loss includes a first single-modal similarity loss between the visible light extraction features and the visible light reconstruction features of the visible light sample image, and a second single-modal similarity loss between the thermal infrared extraction features and the thermal infrared reconstruction features of the thermal infrared sample image. similar losses; The cross-modal loss includes a first multi-modal interaction loss between the thermal infrared reconstruction feature and the visible light extraction feature, and a second multi-modal interaction loss between the visible light reconstruction feature and the thermal infrared extraction feature. . 17. The device according to claim 16, wherein the target pedestrian detection model includes a parallel first encoding network and a second encoding network, the first encoding network and the second encoding network are jointly connected to the fusion component ; The detection result determination module is also used to input the visible light image to be detected into the first encoding network for feature extraction to obtain visible light features to be detected; and to input the thermal infrared image to be detected into the second coding network. Feature extraction is performed in the coding network to obtain the thermal infrared features to be detected; the visible light features to be detected and the thermal infrared features to be detected are added element by element through the fusion component to obtain the fusion image features to be detected; based on the The image features to be detected are fused to perform target detection, and the pedestrian detection results are obtained. 18. A computer device, comprising a memory and a processor, the memory stores a computer program, characterized in that when the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented. . 19. A computer-readable storage medium with a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented. 20. A chip, including a storage unit and a processing unit, the storage unit stores a computer program, characterized in that when the processing unit executes the computer program, the method according to any one of claims 1 to 10 is implemented. A step of.