CN108681707A - Wide-angle model recognizing method and system based on global and local Fusion Features - Google Patents

Abstract

The invention discloses a large-angle vehicle identification method and system based on fusion of global and local features. When a vehicle passes through a vehicle identification collection area, an image containing the vehicle is intercepted. First, the intercepted image containing the vehicle is cut to obtain complex background removal. The vehicle picture is divided into vehicle face image block, rear image block and wheel image block. Then import the vehicle picture, car face image block, car rear image block and wheel image block into the deep multi-branch convolutional neural network to perform feature training on the global and local features of the vehicle, and combine the vehicle picture features with each segment block features for feature fusion. Finally, classifiers are used to classify and identify the fused features. The invention fuses the global and local features of the large-angle vehicle, and can obviously improve the accuracy of large-angle vehicle identification.

Description

Large-angle car recognition method and system based on global and local feature fusion

technical field

The invention relates to the technical field of intelligent identification, in particular to a large-angle vehicle identification method and system based on fusion of global and local features.

Background technique

Vehicle type recognition technology has become the need of social development and is an important research field in intelligent transportation systems. It is also a hot topic in the research of artificial intelligence image recognition, image processing and pattern recognition. It has important application value and research significance.

Traditional vehicle model recognition always relies on manual recognition and license plate recognition, and all use bayonet pictures. Among them, the efficiency of manual recognition is extremely low, and the recognition accuracy is not high. Effective identification of illegal and criminal vehicles such as license plates. In addition, the scene where the large-angle vehicle is located is changeable, and the appearance of the vehicle body is changeable, such as color, shape, size, etc. In addition, the shooting distance and shooting angle of each camera are different, which will affect the final vehicle model recognition result.

Contents of the invention

Aiming at the problem of low recognition accuracy in existing vehicle model recognition methods, the present invention provides a large-angle vehicle model recognition method and system based on fusion of global and local features.

In order to solve the above problems, the present invention is achieved through the following technical solutions:

A large-angle car recognition method based on global and local feature fusion, including the following steps:

Step 1, when the vehicle passes through the vehicle type identification collection area, intercept the image containing the vehicle;

Step 2, preprocessing the intercepted image containing the vehicle:

Step 2.1, first cut the acquired image using a label frame to obtain a vehicle image with complex background removed;

Step 2.2, using the target detection SSD algorithm on the vehicle picture to detect the car face part, car rear part and wheel part of the vehicle respectively, and obtain the car face image segmentation, vehicle rear image segmentation and wheel image segmentation;

Step 2.3. Rotate and mirror the vehicle image, vehicle face image block, vehicle rear image block, and wheel image block for data enhancement, and the enhanced vehicle image, vehicle face image block, and vehicle rear image Cut the block and wheel image blocks into a uniform size to build a vehicle database;

Step 3, importing the built vehicle database into the deep multi-branch convolutional neural network to carry out feature training on the global and local features of the vehicle;

Step 4, performing weighted feature fusion on the features of the vehicle picture, vehicle face image block, car rear image block and wheel image block obtained through feature training;

Step 5. Classify and identify the fused features through a classifier.

In the above step 2.2, the specific steps of the target detection SSD algorithm are as follows:

Step 2.2.1, input the vehicle picture with the complex background removed, and perform forward propagation to obtain the basic characteristics of the vehicle;

Step 2.2.2, setting candidate regions of different sizes and different aspect ratios at each position of the feature;

Step 2.2.3, matching the candidate area with the real frame;

Step 2.2.4, the position offset of each candidate area is predicted and the category confidence is output by the predictor;

Step 2.2.5, calculate and adjust the weight of each layer through the reverse broadcast of the multi-task loss function.

In the above step 2.2.5, the multi-task loss function is the sum of the position loss function and the confidence loss function.

In the above step 4, when performing weighted feature fusion on features,

The vehicle image block weight coefficient w1 is:

w1=k1/(k1+k2+k3+k4);

The block weight coefficient w2 of the car face image is:

w2=k2/(k1+k2+k3+k4);

The block weight coefficient w3 of the rear image is:

w3=k3/(k1+k2+k3+k4);

The wheel image block weight coefficient w4 is:

w4=k4/(k1+k2+k3+k4);

Among them, k1 is the weight of the vehicle image block, k2 is the weight of the car face image block, k3 is the weight of the rear image block, and k4 is the weight of the wheel image block.

A large-angle car recognition system based on the fusion of global and local features designed based on the above method, consists of video image acquisition module, vehicle picture detection module, vehicle picture segmentation module, vehicle global and local image feature extraction module, vehicle Composed of global and local image fusion modules, and large-angle vehicle model recognition modules;

Video image acquisition module: used to acquire vehicle monitoring video;

Vehicle picture detection module: used to detect the vehicles in the acquired road traffic intersection surveillance video, and intercept the vehicles in the video surveillance;

Vehicle Image Segmentation Module: It is used to detect the face, rear, and wheel parts of the vehicle, and segment the face, rear, and wheel parts of the vehicle.

Vehicle global and local image feature extraction module; through the deep multi-branch convolutional neural network, feature extraction is performed on the entire vehicle image, the vehicle face image block, the rear image block, and the wheel image block;

The vehicle global and local image fusion module; then the features extracted by each branch are subjected to feature fusion, so that more abundant features of the vehicle can be obtained.

Large-angle vehicle model identification module: it is used to identify the large-angle vehicle model based on the fused features, so as to identify the vehicle model in the vehicle test library.

In the above solution, the video image acquisition modules are 4 high-definition cameras arranged on 4 diagonal lines.

Compared with the traditional vehicle type recognition method, the present invention performs local segmentation on the vehicle, and extracts, fuses and classifies the global and local features through the deep multi-branch convolutional neural network, so that the recognition accuracy is greatly improved.

Description of drawings

Figure 1 is a flow chart of a large-angle vehicle recognition method based on global and local feature fusion.

Figure 2 is a structural diagram of a large-angle vehicle recognition system based on global and local feature fusion.

Figure 3 is a structural diagram of a deep multi-branch convolutional neural network.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in combination with specific examples and with reference to the accompanying drawings.

As shown in Figure 1, a large-angle vehicle identification method and system based on global and local feature fusion, the specific steps are as follows:

Step S1: When the vehicle passes through the vehicle type identification collection area (road traffic intersection), locate the position of the vehicle, and intercept the image containing the vehicle.

Step S2: Perform preprocessing on the intercepted image containing the vehicle.

Step S21: Acquiring information on road traffic intersections containing vehicle pictures.

Step S22: Crop the acquired image using a label frame to obtain a vehicle image with complex background removed.

Step S23: After obtaining the vehicle picture, use the target detection SSD algorithm to detect the face part, the rear part and the wheel part of the vehicle respectively, and obtain the face image block, the rear image block, and the wheel image block.

The target detection SSD algorithm specifically includes:

(1) Input the vehicle picture with the complex background removed, and perform forward propagation to obtain the basic characteristics of the vehicle;

(2) Set candidate regions of different sizes and different aspect ratios at each position of the feature;

(3) Match the candidate area with the real frame;

(4) Output the position offset prediction and category confidence of each candidate area through the predictor;

(5) Through the multi-task loss function, the weight of each layer is calculated and adjusted through reverse broadcast. The multi-task loss function is the sum of the position loss function (l _loc ) and the confidence loss function (l _conf ).

The multi-task loss function formula is as follows:

Among them, N represents the number of matching candidate frame regions, Represents the weight factor.

The confidence loss function formula is as follows:

Among them, c represents confidence.

The position loss function formula is as follows:

Among them, l represents the prediction frame, g represents the real value, d represents the candidate area frame, (cx, cy) represents the center point, the width is w, and the height is h.

Step S24: Rotating and mirroring the vehicle image, the vehicle face image block, the rear image block, and the wheel image block respectively for data enhancement, and cutting the enhanced vehicle image and each block image into a uniform size, Build a vehicle database.

Step S3: Construct a deep multi-branch convolutional neural network, and import vehicle pictures, car face image blocks, car rear image blocks, and wheel image blocks into the deep multi-branch convolutional neural network for the global sum of the vehicle. The local features are used for feature training, and then the vehicle image features and each block feature are used for feature fusion.

The deep multi-branch convolutional neural network as shown in Figure 3 specifically includes: a convolutional layer Conv1, a pooling layer Pool1, a convolutional layer Conv2, a pooling layer Pool2, and a convolutional layer Conv3 connected successively by vehicle picture branches , pooling layer Pool3, convolutional layer Conv4, pooling layer Pool4, the convolutional layer Conv5, pooling layer Pool5, convolutional layer Conv6, pooling layer Pool6, convolutional layer Conv7, Pooling layer Pool7, convolutional layer Conv8, pooling layer Pool8, convolutional layer Conv9, pooling layer Pool9, convolutional layer Conv10, pooling layer Pool10, convolutional layer Conv11, pooling layer Layer Pool11, convolutional layer Conv12, pooling layer Pool12, convolutional layer Conv13, pooling layer Pool13, convolutional layer Conv14, pooling layer Pool14, convolutional layer Conv15, pooling layer Pool15, the convolutional layer Conv16, and the pooling layer Pool16 also include that the pooling layer Pool4 is connected to the connection layer Concat through the fully connected layer Fc1, and the pooling layer Pool8 is connected to the connection layer Concat through the fully connected layer Fc2. The pooling layer Pool12 is connected to the connection layer Concat through the fully connected layer Fc3, the pooling layer Pool16 is connected to the connection layer Concat through the fully connected layer Fc4, and the classification layer Softmax+coco_loss is connected to the connection layer Concat through the fully connected layer Fc5 connect.

The convolution layer Conv performs a convolution operation on the input image through a convolution kernel, and then uses a neuron activation function to calculate an output value of the convolution.

The pooling layer Pool compresses the features output by the convolutional layer, simplifies the computational complexity of the deep network, and extracts main features.

The fully connected layer Fc connects each node of the upper layer with all nodes of the adjacent layer.

The connection layer Concat is to fuse the features output by each fully connected layer above.

The classification layer Softmax+coco_loss classifies the features input by the connection layer.

The feature fusion specifically includes:

The vehicle image block weight coefficient is: w1=k1/(k1+k2+k3+k4);

The block weight coefficient of the car face image is: w2=k2/(k1+k2+k3+k4);

The block weight coefficient of the rear image is: w3=k2/(k1+k2+k3+k4);

The wheel image block weight coefficient is: w4=k2/(k1+k2+k3+k4);

Among them, k1 is the weight of the vehicle image block, k2 is the weight of the car face image block, k3 is the weight of the rear image block, and k4 is the weight of the wheel image block. Define the output of the vehicle image block as x1, and the car face image The block output is x2, the rear image block output is x3, the wheel image output is x4, and the final feature fusion output is: y=w1*x1+w2*x2+w3*x3+w4*x4.

Step S4: Perform feature learning on the fused features through the Softmax loss classification loss function and the coco_loss loss function.

The Softmax loss function is calculated using the following formula to obtain the probability output of each class

Among them, x _i is the i-th node value of the Softmax layer, y _i is the i-th output value, and n is the number of nodes in the Softmax layer.

The coco_loss loss function mainly shortens the characteristics of samples of the same type and distances the characteristics of samples of different classifications.

(1) Normalization of input features and center features

Among them, c _k is the feature center of the kth class object, f ⁽ⁱ⁾ represents the input feature, i=1,...,N, that is, the batch_size is N.

(2) Calculate the cosine distance between the input feature and the center of each feature

The cosine distance is defined as The value range is [-1,+1], and the larger the value, the higher the similarity.

(3) Calculate the coco_loss loss function

Among them, B represents the entire batch. The numerator term represents the cosine distance between the input feature f ⁽ⁱ⁾ and its corresponding central feature; the denominator term represents the sum of the distances from the input feature to all central features.

The present invention can extract multi-part and rich features of large-angle vehicles through a deep multi-branch convolutional neural network, and then fuse these rich features, thereby greatly improving the accuracy of identifying large-angle vehicles.

A large-angle vehicle identification system based on the fusion of global and local features designed based on the above method, as shown in Figure 2, includes video image acquisition module, vehicle image detection module, vehicle image segmentation module, vehicle global and local Image feature extraction module, vehicle global and local image feature fusion module, large-angle vehicle model recognition module.

Video image collection module: used to obtain the vehicle monitoring video in the vehicle identification collection area (road traffic intersection), which can be four high-definition cameras.

Vehicle picture detection module: used to detect the vehicles in the acquired road traffic intersection monitoring video, and intercept the vehicles in the video monitoring.

Vehicle Image Segmentation Module: It is used to detect the face, rear, and wheel parts of the vehicle, and segment the face, rear, and wheel parts of the vehicle.

Vehicle global and local image feature extraction module: through the deep multi-branch convolutional neural network, feature extraction is performed on the entire vehicle image, the vehicle face image block, the rear image block, and the wheel image block;

Vehicle global and local image feature fusion module: Then feature fusion is performed on the features extracted by each branch, so that more abundant features of the vehicle can be obtained;

Large-angle vehicle model identification module: it is used to identify the large-angle vehicle model based on the fused features, so as to identify the vehicle model in the vehicle test library.

Locate the position of the vehicle when the vehicle passes through the traffic intersection, and intercept the image containing the vehicle, preprocess the intercepted image containing the vehicle, first use the label frame to crop the intercepted image containing the vehicle, and obtain the vehicle with the complex background removed Image, divide the vehicle image into car face image blocks, car rear image blocks, and wheel image blocks. Build a deep multi-branch convolutional neural network, import vehicle pictures, car face image blocks, car rear image blocks, and wheel image blocks into the deep multi-branch convolutional neural network to perform global and local features of the vehicle Feature training, and then feature fusion of vehicle image features and each block feature, and then classify and identify the fused features through a classifier. The beneficial effect of the present invention is that the global and local features of large-angle vehicles can be fused, and compared with the existing traditional vehicle identification methods, the accuracy of large-angle vehicle identification can be significantly improved.

It should be noted that although the above-mentioned embodiments of the present invention are illustrative, they are not intended to limit the present invention, so the present invention is not limited to the above specific implementation manners. Without departing from the principles of the present invention, all other implementations obtained by those skilled in the art under the inspiration of the present invention are deemed to be within the protection of the present invention.

Claims (6)

1. The large-angle vehicle identification method based on global and local feature fusion is characterized in that the steps are as follows: Step 1, when the vehicle passes through the vehicle type identification collection area, intercept the image containing the vehicle; Step 2, preprocessing the intercepted image containing the vehicle: Step 2.1, first cut the acquired image using a label frame to obtain a vehicle image with complex background removed; Step 2.2, using the target detection SSD algorithm on the vehicle picture to detect the car face part, car rear part and wheel part of the vehicle respectively, and obtain the car face image segmentation, vehicle rear image segmentation and wheel image segmentation; Step 2.3, rotate and mirror the vehicle picture, the vehicle face image block, the car rear image block and the wheel image block, and the enhanced vehicle picture, the car face image block, the car rear image block and the wheel image Cut the blocks into a uniform size to build a vehicle database; Step 3, importing the built vehicle database into the deep multi-branch convolutional neural network to carry out feature training on the global and local features of the vehicle; Step 4, performing weighted feature fusion on the features of the vehicle picture, vehicle face image block, car rear image block and wheel image block obtained through feature training; Step 5. Classify and identify the fused features through a classifier. 2. the large-angle vehicle identification method based on global and local feature fusion according to claim 1, is characterized in that, in step 2.2, the concrete steps of target detection SSD algorithm are as follows: Step 2.2.1, input the vehicle picture with the complex background removed, and perform forward propagation to obtain the basic characteristics of the vehicle; Step 2.2.2, setting candidate regions of different sizes and different aspect ratios at each position of the feature; Step 2.2.3, matching the candidate area with the real frame; Step 2.2.4, the position offset of each candidate area is predicted and the category confidence is output by the predictor; Step 2.2.5, calculate and adjust the weight of each layer through the reverse broadcast of the multi-task loss function. 3. The large-angle vehicle identification method based on global and local feature fusion according to claim 2, characterized in that, in step 2.2.5, the multi-task loss function is the sum of the position loss function and the confidence loss function. 4. The large-angle vehicle identification method based on global and local feature fusion according to claim 1, wherein in step 4, when carrying out weighted feature fusion to features, The vehicle image block weight coefficient w1 is: w1=k1/(k1+k2+k3+k4); The block weight coefficient w2 of the car face image is: w2=k2/(k1+k2+k3+k4); The block weight coefficient w3 of the rear image is: w3=k3/(k1+k2+k3+k4); The wheel image block weight coefficient w4 is: w4=k4/(k1+k2+k3+k4); Among them, k1 is the weight of the vehicle image block, k2 is the weight of the car face image block, k3 is the weight of the rear image block, and k4 is the weight of the wheel image block. 5. A large-angle vehicle identification system based on global and local feature fusion, which is characterized in that it consists of sequentially connected video image acquisition module, vehicle picture detection module, vehicle picture segmentation module, vehicle global and local image feature extraction module, vehicle global and It consists of a local image feature fusion module and a large-angle vehicle model recognition module; Video image acquisition module: used to acquire vehicle monitoring video; Vehicle picture detection module: used to detect the vehicles in the acquired road traffic intersection surveillance video, and intercept the vehicles in the video surveillance; Vehicle image segmentation module: used to detect the face, rear, and wheel parts of the vehicle, and segment the face, rear, and wheel parts of the vehicle; Vehicle global and local image feature extraction module: through the deep multi-branch convolutional neural network, feature extraction is performed on the entire vehicle image, the vehicle face image block, the rear image block, and the wheel image block; Vehicle global and local image feature fusion module: Then feature fusion is performed on the features extracted by each branch, so that more abundant features of the vehicle can be obtained; Large-angle vehicle model identification module: used to identify the large-angle vehicle model based on the fused features, so as to identify the vehicle model in the vehicle test library. 6. The large-angle vehicle identification system based on global and local feature fusion according to claim 5, wherein the video image acquisition module is 4 high-definition cameras arranged on 4 diagonal lines.