CN116052150A - Vehicle face recognition method for shielding license plate - Google Patents

Abstract

The invention discloses a vehicle face recognition method aiming at a shielded license plate, which solves the technical problem of the existing recognition of the shielded license plate. According to the invention, color prediction information and picture vector information are obtained according to the color prediction module, the global perception module and the detail perception module, then a corresponding folder is found according to the color prediction information, cosine distance calculation is sequentially carried out on the corresponding folder and the picture vector information in the folder, and the first 5 are selected as recognition results according to the sequence from big to small. Compared with the traditional method, the method has the advantages that the auxiliary flow of vehicle color classification is increased, the identification is carried out through the global perception module and the detail perception module, multiple normalization calculation is carried out in the global perception module from batches and channels, namely, vehicles which shield license plates are searched and identified in a vehicle picture library from the multi-dimension of color appearance, global information and detail information, the content expression of searching capacity and models on the vehicles which shield the license plates is improved, and the vehicles which do not shield the license plates are successfully found out.

Description

A vehicle face recognition method for obscured license plates

Technical Field

The present invention relates to the technical field of computer vision and pattern recognition, and in particular to a vehicle face recognition method for obscured license plates.

Background Art

Every household chooses cars as a means of transportation. However, in a complex traffic environment, some drivers may violate the law such as speeding and overloading on a certain road, cover their license plates to avoid being tracked by the traffic police, and restore the license plates to normal after a period of time and drive normally. This phenomenon makes traffic control and monitoring difficult and difficult to solve. Although there are a large number of cameras on the road that can monitor the tracking of illegal cars, most of the features such as car models and colors are the same. It is difficult to find the corresponding target images in the massive image library taken by these non-overlapping cameras by manpower alone, and it takes a lot of time to search. Therefore, a method is needed to use deep learning technology to find the car face images that appear in the image library as unobstructed vehicles in the car face image library according to the features of the car face, so as to find the illegal vehicles. The existing deep learning-based methods all extract an effective global feature representation for each car face image, and lack the consideration of detailed feature information. In addition, it is difficult to obtain information about appearance changes through the features extracted by neural networks, and the car face recognition results obtained after sorting are actually very different from the actual vehicles in the images to be tested.

Summary of the invention

The purpose of the present invention is to solve the above-mentioned defects in the prior art and provide a vehicle face recognition method for obscured license plates.

The purpose of the present invention can be achieved by adopting the following technical solutions:

A vehicle face recognition method for a blocked license plate, the vehicle face recognition method comprising the following steps:

S1. Input a car face image P in the dataset into the backbone network to obtain a feature map _Fp∈R ^C×W×H , where C represents the number of feature map channels, W and H are the width and height of the feature map, and R represents the real number domain;

S2, input the feature map _Fp into the color prediction module to predict the vehicle color, and obtain the predicted vehicle color result i, and the color prediction result i is used in step S6;

其中D为列向量的维度，向量Q₁包含的是车脸图片的全局信息；S3, input the feature map _Fp into the global perception module for prediction, and obtain the vector

Where D is the dimension of the column vector, and the vector Q ₁ contains the global information of the car face image;

向量Q₂包含的是车脸图片的细节信息；S4, input the feature map _Fp into the detail perception module for prediction, and obtain the vector

Vector Q ₂ contains the detailed information of the car face image;

S5. Concatenate vector _Q1 and vector _Q2 to obtain vector _Qr∈R ^D×1 . Vector _Qr contains the global information and detailed information of the car face image.

S6. According to the predicted vehicle color result i in step S2, the vector Q _r is stored in a folder named i;

S7, repeating steps S1 to S6 for all the car face images in the image library in the data set, to obtain a vector set _Qi = {Qi _,k | _1≤k≤In }, where _In is the number of vectors in the folder i, and Qi _,k refers to the vector corresponding to the kth image in the folder i;

S8, input the vehicle face image to be tested and execute step S1 in sequence to obtain a feature map F _q ;

S9, input feature map F _q and execute step S2 to obtain a color prediction result i _q . The color prediction result i is used in step S11;

S10, input feature map F _q and perform steps S3, S4 and S5 in sequence to obtain a concatenated vector Q _q ∈R ^D×1 , where the vector Q _q contains the global information and detail information of the vehicle face image to be tested;

I_nq为文件夹i_q中向量的数目，

指的是文件夹i_q中第g张图片对应的向量，颜色预测分类的处理的使得车脸图片的识别更有针对性；S11. Calculate the cosine distance between vector Q _q and all vectors in folder i _q to obtain the set

I _nq is the number of vectors in folder i _q ,

Refers to the vector corresponding to the g-th picture in the folder i _q. The color prediction and classification processing makes the recognition of car face pictures more targeted;

按距离的取值从大到小进行排序，选择最高的前5个向量对应的图片作为预测结果。S12, will gather

Sort the distance values from large to small, and select the images corresponding to the top 5 highest vectors as the prediction results.

Furthermore, the backbone network structure is connected from the input layer to the output layer in sequence: convolution layer conv1_1, Relu layer conv1_1_relu, convolution layer conv1_2, Relu layer conv1_2_relu, pooling layer max_pooling1, convolution layer conv2_1, Relu layer conv2_1_relu, convolution layer conv2_2, BN layer conv2_2_bn, Relu layer conv2_2_relu, pooling layer max_pooling2, convolution layer conv3_1, Relu layer con v3_1_relu, convolution layer conv3_2, Relu layer conv3_2_relu, convolution layer conv3_3, Relu layer conv3_3_relu, pooling layer max_pooling3, convolution layer conv4_1, Relu layer conv4_1_relu, convolution layer conv4_2, Relu layer conv4_2_relu, convolution layer conv4_3, Relu layer conv4_3_relu, pooling layer max_pooling4, convolution layer conv5_1, Relu layer conv5_ 1_relu, convolution layer conv5_2, Relu layer conv5_2_relu, convolution layer conv5_3, Relu layer conv5_3_relu, pooling layer max_pooling5, convolution layer fc6, Relu layer fc6_relu, convolution layer fc7, Relu layer fc7_relu, convolution layer conv6_1, Relu layer conv6_1_relu, convolution layer conv6_2, Relu layer conv6_2_relu, convolution layer conv7_1, Relu layer conv7_1_rel u, convolutional layer conv7_2, Relu layer conv7_2_relu, convolutional layer conv8_1, Relu layer conv8_1_relu, convolutional layer conv8_2, Relu layer conv8_2_relu, convolutional layer conv9_1, Relu layer conv9_1_relu, convolutional layer conv9_2, Relu layer conv9_2_relu, convolutional layer conv10_1, Relu layer conv10_1_relu, convolutional layer conv10_2, Relu layer conv10_2_relu.

Furthermore, the color prediction module structure is connected in sequence from the input layer to the output layer: pooling layer global_pooling_1, fc layer fc_1, and fc layer fc_2.

Furthermore, the global perception module structure is connected in sequence from the input layer to the output layer: pooling layer global_pooling_2, multi-dimensional normalization layer conv1_1_in_bn, convolution layer conv11_1, Relu layer conv11_1_relu, convolution layer conv11_2, Relu layer conv11_2_relu; among which, the multi-dimensional normalization layer conv1_1_in_bn is composed of the BN layer conv11_1_bn and the IN layer conv11_1_in.

Furthermore, the detail perception module structure is connected in sequence from the input layer to the output layer: feature detail compression layer horizontal_pooling_1, BN layer conv12_1_bn, convolution layer conv12_1, Relu layer conv12_1_relu, convolution layer conv12_2, and Relu layer conv12_2_relu.

Furthermore, the process of step S2 is as follows:

S21, input the feature map _Fp to the global average pooling layer global_pooling_1 to obtain the feature map _Ea∈RC ^×1 , which is used to compress the information of the feature map _Fp ;

S22, input the feature map E _a into the fc layer fc_1 and fc layer fc_2, and obtain the vector E _f ∈ R ^{N ×1} through the softmax function, where N is the number of vehicle colors in the data set, and predict the vehicle color through the fully connected structure of the two fc layers;

S23, get the position subscript i corresponding to the highest value in the vector E _f , and output the vehicle color result i;

Furthermore, the process of step S3 is as follows:

S31, input the feature map F _p into the global average pooling layer global_pooling_2 to obtain a vector V∈R ^D×1 , which is used to compress the information of the feature map F _p ;

S32. Input the vector V to the BN layer conv11_1_bn to obtain a vector V _b ∈ ^RD×1 ; the vector V _b is the result of normalizing the vector V from the batch direction;

S33, input the vector V to the IN layer conv11_1_in to obtain the vector V _i ∈R ^D×1 ; the vector V _i is the result of normalizing the vector V from the channel direction;

S34, concatenate the vector V _b and the vector V _i to obtain a vector V _c ∈R ^2D×1 ;

S35. Pass the vector V _c through the convolution layer conv11_1, the Relu layer conv11_1_relu, the convolution layer conv11_2, and the Relu layer conv11_2_relu to obtain the vector Q _1. Therefore, the vector Q ₁ obtains the data of the vector V after batch normalization and channel normalization, avoiding overfitting and loss of style features.

Furthermore, the process of step S4 is as follows:

S41, input the feature map _Fp to the feature detail compression layer horizontal_pooling_1 for compression, and then input it to the BN layer conv12_1_bn for normalization to obtain the feature map _Js∈R ^C×H×1 ;

S42. Input the feature map J _s to the convolution layer conv12_1, the Relu layer conv12_1_relu, the convolution layer conv12_2, and the Relu layer conv12_2_relu to obtain a vector Q ₂ .

Furthermore, the feature detail compression layer is used to compress detail features for subsequent recognition, and its working process is as follows:

Assume that the input of the feature detail compression layer is the feature map M∈RC ^×W×H , and the output is the feature map Y∈RC ^×H×1 .

The feature map M is divided into a feature map set A ^* = {A _m |1≤m≤H} with a number of H in the height direction of the feature map, where A _m ∈ ^RC×W×1 refers to the mth feature map in the set A ^* ;

Compress a certain feature graph A _z of the set A ^* to obtain the result k _z . The compression formula is as follows:

Where _Az,x,y is the x-th row vector with y channels in the feature map _Az at the z-th position of the set A ^* ;

Formula (1) is executed for each feature map in the set A ^* and the results are aggregated into a column vector T∈R ^H×1 , where the value at the t-th position in the column vector T represents the result of the feature map at the t-th position of the set A ^* calculated using formula (1).

Furthermore, the cosine distance is used to calculate the similarity of two vectors, and the calculation process is as follows:

Assume that the cosine distance between vector J and vector B is calculated to obtain the value l. The calculation expression is as follows:

l = 1 - J·B ^T .

In the formula, the symbol “·” represents the bit-by-bit multiplication of the elements in the vector, and the superscript “T” represents the vector transpose operation.

Compared with the prior art, the present invention has the following advantages and effects:

(1) The present invention can find 5 vehicles similar to the obstructed license plate image in the image library, and can find the image of the unobstructed vehicle that matches the vehicle in the image library among the 5 images.

(2) Compared with the traditional method, the present invention makes improvements in the normalization of neural network and feature vector, thereby improving the ability of feature vector to express vehicle face images.

(3) Compared with the traditional method, the present invention adds a color prediction module as an auxiliary for subsequent recognition. The vehicle faces are classified according to their colors and stored in folders of different colors, which makes the recognition of vehicle faces more targeted.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

FIG1 is a flow chart of a vehicle face recognition method for an obscured license plate disclosed in the present invention.

FIG. 2 is a schematic diagram of a vehicle face recognition method for an obscured license plate disclosed in the present invention.

FIG3 is a bar graph comparing the accuracy of the present method under the premise of Example 1 with other methods when using the VehicleID dataset;

FIG4 is a bar graph comparing the accuracy of the present method with other methods under the premise of Example 2 when using the VeRi dataset;

FIG5 is a structural diagram of the backbone network in the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Example 1

In this embodiment 1, the VehicleId dataset is used. Each image in the dataset has an ID tag corresponding to an identity in the real world. The training set contains 13,164 vehicles and 113,346 images. The test set contains 2,400 vehicles and 19,777 images.

The method comprises the following steps:

Step S1, obtaining the number of vehicle color types N in the data set, in which N=9, and the number of color types is used for dimension setting in the fc layer fc_2 in the color prediction module; and randomly selecting 1000 pictures from the test set according to the vehicle ID label as the picture library to be tested;

Step S2, experimental environment. The experimental environment of this embodiment only requires general hardware configuration and a graphics processing unit (GPU) that can increase the computing speed for accelerated computing. The model building, training, and testing of training results are all completed under the Pytorch deep learning framework and FastReid tools, using the Compute Unified Device Architecture (CUDA) to enable the GPU to solve complex computing problems. The specific operating environment configuration required for the experiment of this embodiment is shown in Table 1.

Table 1. Configuration table of the experimental operation environment of this embodiment

Step S3: construct a vehicle face recognition method for obscured license plates. The structure diagram is shown in FIG2 . The specific steps of constructing the structure are as follows:

Step S31, build a backbone network, as shown in Figure 5, the backbone network structure is connected from the input layer to the output layer in sequence: convolution layer conv1_1, Relu layer conv1_1_relu, convolution layer conv1_2, Relu layer conv1_2_relu, pooling layer max_pooling1, convolution layer conv2_1, Relu layer conv2_1_relu, convolution layer conv2_2, BN layer conv2_2_bn, Relu layer conv2_2_relu, pooling layer max_pooling2, convolution layer conv3_1 , Relu layer conv3_1_relu, convolution layer conv3_2, Relu layer conv3_2_relu, convolution layer conv3_3, Relu layer conv3_3_relu, pooling layer max_pooling3, convolution layer conv4_1, Relu layer conv4_1_relu, convolution layer conv4_2, Relu layer conv4_2_relu, convolution layer conv4_3, Relu layer conv4_3_relu, pooling layer max_pooling4, convolution layer conv5_1, Relu layer conv5_1_relu, convolution layer conv5_2, Relu layer conv5_2_relu, convolution layer conv5_3, Relu layer conv5_3_relu, pooling layer max_pooling5, convolution layer fc6, Relu layer fc6_relu, convolution layer fc7, Relu layer fc7_relu, convolution layer conv6_1, Relu layer conv6_1_relu, convolution layer conv6_2, Relu layer conv6_2_relu, convolution layer conv7_1, Relu layer conv7_1_ relu, convolutional layer conv7_2, Relu layer conv7_2_relu, convolutional layer conv8_1, Relu layer conv8_1_relu, convolutional layer conv8_2, Relu layer conv8_2_relu, convolutional layer conv9_1, Relu layer conv9_1_relu, convolutional layer conv9_2, Relu layer conv9_2_relu, convolutional layer conv10_1, Relu layer conv10_1_relu, convolutional layer conv10_2, Relu layer conv10_2_relu;

Step S32, constructing a color prediction module, the color prediction module structure is sequentially connected from the input layer to the output layer: pooling layer global_pooling_1, fc layer fc_1, fc layer fc_2;

Step S33, construct a global perception module. The structure of the global perception module is connected in sequence from the input layer to the output layer: pooling layer global_pooling_2, multi-dimensional normalization layer conv1_1_in_bn, convolution layer conv11_1, Relu layer conv11_1_relu, convolution layer conv11_2, Relu layer conv11_2_relu; wherein, the multi-dimensional normalization layer conv1_1_in_bn is composed of the BN layer conv11_1_bn and the IN layer conv11_1_in;

Step S34, construct a detail perception module, the structure of the detail perception module is connected in sequence from the input layer to the output layer: feature detail compression layer horizontal_pooling_1, BN layer conv12_1_bn, convolution layer conv12_1, Relu layer conv12_1_relu, convolution layer conv12_2, Relu layer conv12_2_relu;

Step S4, divide the training set images into a picture library and a to-be-queried library according to the vehicle ID, wherein the vehicle ID in the to-be-queried library is unique, and the picture library can contain multiple pictures of the same vehicle ID. First, the processed training set images are trained using this method. During the training process, the optimizer used is Adam, the learning rate is set to 0.00035, the learning rate momentum is set to 0.0005, the training batch size is set to 64, the epoch is set to 60, and the learning rate is reduced by a multiple of 0.1 when the epoch is 30 and 50.

Step S5: Input a certain car face image P in the data set into the backbone network to obtain a feature map F _p ∈ ^RC×W×H , where C represents the number of feature map channels, W and H represent the width and height of the feature map, and R represents the real number domain;

Step S6, inputting the feature map _Fp into the color prediction module to predict the vehicle color, and obtaining the predicted vehicle color result as i, where 1≤i≤N;

The specific process is as follows:

S61, input the feature map _Fp to the global average pooling layer global_pooling_1 to obtain the feature map _Ea∈RC ^×1 , which is used to compress the information of the feature map _Fp ;

S62, input the feature map E _a into the fc layer fc_1 and the fc layer fc_2, and obtain the vector E _f ∈ R ^{N ×1} through the softmax function, and predict the vehicle color through the fully connected structure of the two fc layers;

S63. Take the position index i corresponding to the highest value in vector E _f and output the vehicle color result i

其中D为列向量的维度，向量Q₁包含的是车脸图片的全局信息；在实施例1中，D＝2048；Step S7: Input the feature map _Fp into the global perception module for prediction to obtain the vector

Where D is the dimension of the column vector, and the vector _Q1 contains the global information of the vehicle face image; in Example 1, D=2048;

The specific process is as follows:

S71, input the feature map _Fp to the global average pooling layer global_pooling_2 to obtain a vector V∈R ^D×1 ;

S72, input the vector V to the BN layer conv11_1_bn to obtain a vector V _b ∈ ^RD×1 ;

S73, input the vector V to the IN layer conv11_1_in to obtain the vector V _i ∈R ^D×1 ;

S74, concatenate the vector V _b and the vector V _i to obtain a vector V _c ∈R ^2D×1 ;

S75. Pass the vector V _c through the convolution layer conv11_1, the Relu layer conv11_1_relu, the convolution layer conv11_2, and the Relu layer conv11_2_relu to obtain the vector Q ₁ .

向量Q₂包含的是车脸图片的细节信息；Step S8: Input the feature map _Fp into the detail perception module for prediction to obtain the vector

Vector Q ₂ contains the detailed information of the car face image;

The specific process is as follows:

S41, input the feature map _Fp to the feature detail compression layer horizontal_pooling_1 for compression, and then input it to the BN layer conv12_1_bn for normalization to obtain the feature map _Js∈R ^C×H×1 ;

S42. Input the feature map J _s to the convolution layer conv12_1, the Relu layer conv12_1_relu, the convolution layer conv12_2, and the Relu layer conv12_2_relu to obtain a vector Q ₂ .

Among them, the working process of the feature detail compression layer in the detail perception module is as follows:

Assume that the input of the feature detail compression layer is the feature map M∈RC ^×W×H and the output is the feature map Y∈RC ^×H×1 :

The feature map M is divided into a feature map set A ^* = {A _m |1≤m≤H} with a number of H in the height direction of the feature map, where A _m ∈ ^RC×W×1 refers to the mth feature map in the set A ^* ;

Compress a certain feature graph A _z of the set A ^* to obtain the result k _z . The compression formula is as follows:

Where _Az,x,y is the x-th row vector with y channels in the feature map _Az at the z-th position of the set A ^* ;

Formula (1) is executed for each feature map in the set A ^* and the results are aggregated into a column vector T∈R ^H×1 , where the value at the t-th position in the column vector T represents the result of the feature map at the t-th position of the set A ^* calculated using formula (1).

Step S9, concatenate vector _Q1 and vector _Q2 to obtain vector _Qr∈R ^D×1 , where vector _Qr contains global information and detailed information of the vehicle face image;

Step S10, according to the predicted vehicle color result i in step S6, the vector Q _r is stored in a folder named i;

Step S11, perform the operations of steps S5 to S10 on all the car face pictures in the picture library of the test set in sequence, and obtain a vector set _Qi = {Qi _,k | _1≤k≤In }, where _In is the number of vectors in the folder i, and Qi _,k refers to the vector corresponding to the kth picture in the folder i;

Step S12: Input the vehicle face image to be tested and execute step S5 in sequence to obtain a feature map F _q ;

Step S13, input the feature map F _q and execute step S6 to obtain a color prediction result i _q . The color prediction result i is used in step S11;

Step S14: Input the feature map _{Fq and} perform the operations of steps S7, ^S8 and S9 in sequence to obtain a concatenated vector _Qq∈RD ^×1 , where the vector _Qq contains the global information and detail information of the vehicle face image to be tested;

I_nq为文件夹i_q中向量的数目，

指的是文件夹i_q中第g张图片对应的向量，颜色预测分类的处理的使得车脸图片的识别更有针对性；Step S15: Calculate the cosine distance between vector Q _q and all vectors in folder i _q to obtain the set

I _nq is the number of vectors in folder i _q ,

Refers to the vector corresponding to the g-th picture in the folder i _q. The color prediction and classification processing makes the recognition of car face pictures more targeted;

The calculation process of the cosine distance is as follows:

Assume that the cosine distance between vector J and vector B is calculated to obtain the value l. The calculation expression is as follows:

l = 1 - J·B ^T .

In the formula, the symbol “·” represents the bit-by-bit multiplication of the elements in the vector, and the superscript “T” represents the vector transpose operation.

按距离的取值从大到小进行排序，选择最高的前5个向量对应的图片作为预测结果。Step S16: Set

Sort the distance values from large to small, and select the images corresponding to the top 5 highest vectors as the prediction results.

Step S17, execute steps S12, S13, S14 and S15 in sequence on the image library to be tested in the test set. The specific process is shown in FIG1 , and all prediction results of the image library to be tested in the test set are obtained, and the accuracy is calculated.

The experimental comparison results of Example 1 are shown in Figure 3. Example 1 is compared with other methods on a test image library of 1,000 images. Five images contain corresponding images marked as accurate. The method obtains an accuracy rate of 88.1%, which is 70.21% higher than the BOW-CN method, 62.47% higher than the LOMO method, 27.61% higher than the FACT method, 27.38% higher than the NuFACT method, 17.81% higher than VAML, and 4.73% higher than QD-DLF, which proves the effectiveness of the method and can successfully identify the corresponding unobstructed license plate.

Example 2

In this embodiment 2, the public VeRi dataset is selected, which contains more than 50,000 images of 776 vehicles, including 37,778 training sets, 11,579 library images, and 1,678 test images. Each image in the dataset has an ID tag corresponding to an identity in the real world.

The method comprises the following steps:

Step S1, obtaining the number N of vehicle color types in the data set, in which N=9, and the number of color types is used for dimension setting in the fc layer fc_2 in the color prediction module;

Step S2, experimental environment. The experimental environment of this embodiment only requires general hardware configuration and a graphics processing unit (GPU) that can improve the computing speed for accelerated computing. The model building, training, and testing of training results are all completed under the Pytorch deep learning framework and Fastreid tools, using the Compute Unified Device Architecture (CUDA) to enable the GPU to solve complex computing problems. Specifically, the hardware and software configurations used in this embodiment are: GPU version is GeForce RTX 2080Ti; CPU version is Intel (R) Xeon (R) Silver 4216CPU@2.10GHz; operating system version is CentOS 8.3.2011; Python version is 3.6.13; CUDA version is 11.2.

Step S3: construct a vehicle face recognition method for obscured license plates. The structure diagram is shown in FIG2 . The specific steps of constructing the structure are as follows:

Step S31, build a backbone network, as shown in Figure 5, the backbone network structure is connected from the input layer to the output layer in sequence: convolution layer conv1_1, Relu layer conv1_1_relu, convolution layer conv1_2, Relu layer conv1_2_relu, pooling layer max_pooling1, convolution layer conv2_1, Relu layer conv2_1_relu, convolution layer conv2_2, BN layer conv2_2_bn, Relu layer conv2_2_relu, pooling layer max_pooling2, convolution layer conv3_1 , Relu layer conv3_1_relu, convolution layer conv3_2, Relu layer conv3_2_relu, convolution layer conv3_3, Relu layer conv3_3_relu, pooling layer max_pooling3, convolution layer conv4_1, Relu layer conv4_1_relu, convolution layer conv4_2, Relu layer conv4_2_relu, convolution layer conv4_3, Relu layer conv4_3_relu, pooling layer max_pooling4, convolution layer conv5_1, Relu layer conv5_1_relu, convolution layer conv5_2, Relu layer conv5_2_relu, convolution layer conv5_3, Relu layer conv5_3_relu, pooling layer max_pooling5, convolution layer fc6, Relu layer fc6_relu, convolution layer fc7, Relu layer fc7_relu, convolution layer conv6_1, Relu layer conv6_1_relu, convolution layer conv6_2, Relu layer conv6_2_relu, convolution layer conv7_1, Relu layer conv7_1_ relu, convolutional layer conv7_2, Relu layer conv7_2_relu, convolutional layer conv8_1, Relu layer conv8_1_relu, convolutional layer conv8_2, Relu layer conv8_2_relu, convolutional layer conv9_1, Relu layer conv9_1_relu, convolutional layer conv9_2, Relu layer conv9_2_relu, convolutional layer conv10_1, Relu layer conv10_1_relu, convolutional layer conv10_2, Relu layer conv10_2_relu;

Step S32, constructing a color prediction module, the color prediction module structure is sequentially connected from the input layer to the output layer: pooling layer global_pooling_1, fc layer fc_1, fc layer fc_2;

Step S33, construct a global perception module. The structure of the global perception module is connected in sequence from the input layer to the output layer: pooling layer global_pooling_2, multi-dimensional normalization layer conv1_1_in_bn, convolution layer conv11_1, Relu layer conv11_1_relu, convolution layer conv11_2, Relu layer conv11_2_relu; wherein, the multi-dimensional normalization layer conv1_1_in_bn is composed of the BN layer conv11_1_bn and the IN layer conv11_1_in;

Step S34, construct a detail perception module, the structure of the detail perception module is connected in sequence from the input layer to the output layer: feature detail compression layer horizontal_pooling_1, BN layer conv12_1_bn, convolution layer conv12_1, Relu layer conv12_1_relu, convolution layer conv12_2, Relu layer conv12_2_relu;

Step S4, divide the training set images into a picture library and a to-be-queried library according to the vehicle ID, wherein the vehicle ID in the to-be-queried library is unique, and the picture library can contain multiple pictures of the same vehicle ID. First, the processed training set images are trained using this method. During the training process, the optimizer used is SGD, the learning rate is set to 0.001, the learning rate momentum is set to 0.0005, the training batch size is set to 64, the epoch is set to 60, and the learning rate is reduced by a multiple of 0.1 when the epoch is 30 and 50.

Step S5: Input a certain car face image P in the data set into the backbone network to obtain a feature map F _p ∈ ^RC×W×H , where C represents the number of feature map channels, W and H represent the width and height of the feature map, and R represents the real number domain;

Step S6, inputting the feature map _Fp into the color prediction module to predict the vehicle color, and obtaining the predicted vehicle color result as i, where 1≤i≤N;

The specific process is as follows:

S21, input the feature map _Fp to the global average pooling layer global_pooling_1 to obtain the feature map _Ea∈RC ^×1 , which is used to compress the information of the feature map _Fp ;

S22, input the feature map E _a into the fc layer fc_1 and fc layer fc_2, and obtain the vector E _f ∈ R ^{N ×1} through the softmax function, and predict the vehicle color through the fully connected structure of the two fc layers;

S23. Get the position subscript i corresponding to the highest value in vector E _f , and output the vehicle color result i.

其中D为列向量的维度，向量Q₁包含的是车脸图片的全局信息；在实施例2中，D＝2048；Step S7: Input the feature map _Fp into the global perception module for prediction to obtain the vector

Where D is the dimension of the column vector, and the vector _Q1 contains the global information of the vehicle face image; in Example 2, D=2048;

The specific process is as follows:

S71, input the feature map _Fp to the global average pooling layer global_pooling_2 to obtain a vector V∈R ^D×1 ;

S72, input the vector V to the BN layer conv11_1_bn to obtain a vector V _b ∈ ^RD×1 ;

S73, input the vector V to the IN layer conv11_1_in to obtain the vector V _i ∈R ^D×1 ;

S74, concatenate the vector V _b and the vector V _i to obtain a vector V _c ∈R ^2D×1 ;

S75. Pass the vector V _c through the convolution layer conv11_1, the Relu layer conv11_1_relu, the convolution layer conv11_2, and the Relu layer conv11_2_relu to obtain the vector Q ₁ .

向量Q₂包含的是车脸图片的细节信息；Step S8: Input the feature map _Fp into the detail perception module for prediction to obtain the vector

Vector Q ₂ contains the detailed information of the car face image;

The specific process is as follows:

S41, input the feature map _Fp to the feature detail compression layer horizontal_pooling_1 for compression, and then input it to the BN layer conv12_1_bn for normalization to obtain the feature map _Js∈R ^C×H×1 ;

S42, input the feature map J _s to the convolution layer conv12_1, the Relu layer conv12_1_relu, the convolution layer conv12_2, and the Relu layer conv12_2_relu to obtain the vector Q ₂ ;

Among them, the working process of the feature detail compression layer in the detail perception module is as follows:

Assume that the input of the feature detail compression layer is the feature map M∈RC ^×W×H and the output is the feature map Y∈RC ^×H×1 :

The feature map M is divided into a feature map set A ^* = {A _m |1≤m≤H} with a number of H in the height direction of the feature map, where A _m ∈ ^RC×W×1 refers to the mth feature map in the set A ^* ;

Compress a certain feature graph A _z of the set A ^* to obtain the result k _z . The compression formula is as follows:

Where _Az,x,y is the x-th row vector with y channels in the feature map _Az at the z-th position of the set A ^* ;

Formula (1) is executed for each feature map in the set A ^* and the results are aggregated into a column vector T∈R ^H×1 , where the value at the t-th position in the column vector T represents the result of the feature map at the t-th position of the set A ^* calculated using formula (1).

Step S9, concatenate vector _Q1 and vector _Q2 to obtain vector _Qr∈R ^D×1 , where vector _Qr contains global information and detailed information of the vehicle face image;

Step S10, according to the predicted vehicle color result i in step S6, the vector Q _r is stored in a folder named i;

Step S11, perform the operations of steps S5 to S10 on all the car face pictures in the picture library in sequence, and obtain a vector set _Qi = {Qi _,k | _1≤k≤In }, where _In is the number of vectors in the folder i, and Qi _,k refers to the vector corresponding to the kth picture in the folder i;

Step S12: Input the vehicle face image to be tested and execute step S5 in sequence to obtain a feature map F _q ;

Step S13, input the feature map F _q and execute step S6 to obtain a color prediction result i _q . The color prediction result i is used in step S11;

Step S14: Input the feature map _{Fq and} perform the operations of steps S7, ^S8 and S9 in sequence to obtain a concatenated vector _Qq∈RD ^×1 , where the vector _Qq contains the global information and detail information of the vehicle face image to be tested;

I_nq为文件夹i_q中向量的数目，

指的是文件夹i_q中第g张图片对应的向量，颜色预测分类的处理的使得车脸图片的识别更有针对性；Step S15: Calculate the cosine distance between vector Q _q and all vectors in folder i _q to obtain the set

I _nq is the number of vectors in folder i _q ,

Refers to the vector corresponding to the g-th picture in the folder i _q. The color prediction and classification processing makes the recognition of car face pictures more targeted;

The calculation process of the cosine distance is as follows:

Assume that the cosine distance between vector J and vector B is calculated to obtain the value l. The calculation expression is as follows:

l = 1 - J·B ^T .

In the formula, the symbol “·” represents the bit-by-bit multiplication of the elements in the vector, and the superscript “T” represents the vector transpose operation.

按距离的取值从大到小进行排序，选择最高的前5个向量对应的图片作为预测结果。Step S16: Set

Sort the distance values from large to small, and select the images corresponding to the top 5 highest vectors as the prediction results.

Step S17, execute steps S12, S13, S14 and S15 in sequence on the image library to be tested in the test set. The specific process is shown in FIG1 , and all prediction results of the image library to be tested in the test set are obtained, and the accuracy is calculated.

The experimental comparison results of Example 2 are shown in Figure 4. Example 2 is compared with other methods on a test image library of 1,678 pictures. Five pictures contain corresponding pictures marked as accurate. The method obtains an accuracy rate of 97.3%, which is 43.61% higher than the BOW-CN method, 50.82% higher than the LOMO method, 24.42% higher than the FACT method, 5.88% higher than the NuFACT method, 6.48% higher than VAML, and 2.84% higher than QD-DLF, which proves the effectiveness of the method and can successfully identify the corresponding unobstructed license plate.

In summary, the above embodiment discloses a vehicle face recognition method for obstructed license plates. The present invention obtains color prediction information and image vector information according to the color prediction module, the global perception module, and the detail perception module, and then finds the corresponding folder according to the color prediction information, and performs cosine distance calculation with the image vector information in the folder in turn, sorts them from large to small, and selects the first 5 as the recognition results. It is used to solve the technical problem of existing identification of obstructed license plates. Compared with the traditional method, the present invention adds an auxiliary process of vehicle color classification, identifies through the global perception module and the detail perception module, and performs multiple normalization calculations from batches and channels in the global perception module, which can improve the search capability and the model's content expression of obstructed vehicles, and successfully find the corresponding vehicles in the unobstructed license plate state. The high recognition rate of this method can be reflected by the experimental results of Example 1 and Example 2.

The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims (10)

1. The vehicle face recognition method for the license plate shielding is characterized by comprising the following steps of:

s1, inputting a certain face picture P in a data set into a backbone network to obtain a feature map F _p ∈R ^C×E×H Wherein C represents the number of channels of the feature map, W and H are the width and height of the feature map, and R represents the real number domain;

s2, mapping the characteristic diagram F _p Input to colorThe method comprises the steps that a prediction module predicts the color of a vehicle to obtain a predicted vehicle color result i;

s3, mapping the characteristic diagram F _p Inputting to global perception module for prediction to obtain vector

Wherein D is the dimension of the column vector;

s4, mapping the characteristic diagram F _p Inputting to a detail perception module for prediction to obtain a vector

S5, vector Q ₁ Sum vector Q ₂ Splicing to obtain a vector Q _r ∈R ^D×1 ；

S6, according to the predicted vehicle color result i of the step S2, the vector Q is calculated _r A folder named i;

s7, sequentially and repeatedly executing the operations from the step S1 to the step S6 on all the face pictures in the picture library in the data set to obtain a vector set Q _i ＝{Q _i,k |1≤k≤I _n }, wherein I _n For the number of vectors in folder i, Q _i,k Refers to a vector corresponding to a kth picture in the folder i;

s8, inputting face pictures to be detected, and sequentially executing the step S1 to obtain a feature map F _q ；

S9, inputting a feature map F _q Executing step S2 to obtain a color prediction result of i _q ；

S10, inputting a feature map F _q Sequentially executing the operations of the steps S3, S4 and S5 to obtain a spliced vector Q _q ∈R ^D×1 ；

S11, vector Q _q And file i _q Cosine distance calculation is carried out on all vectors in the set to obtain a set

I _nq For folder i _q Number of medium vectors, ++>

Refers to a folder i _q Vector corresponding to the g-th picture;

s12, collecting

And sequencing the images according to the distance from large to small, and selecting the images corresponding to the top 5 vectors as prediction results.

2. The method for recognizing the face of a vehicle for shielding a license plate according to claim 1, wherein the backbone network structure is sequentially connected from an input layer to an output layer as follows: convolutional layers conv1_1, relu layer conv1_relu, convolutional layer conv1_2, relu layer conv1_2_relu, pooling layer max_pooling1, convolutional layer conv2_1, relu layer conv2_1_relu, convolutional layer conv2_2, BN layer conv2_2_bn, relu layer conv2_2_relu, pooling layer max_pooling2, convolutional layer conv3_1, relu layer conv3_1_relu, convolutional layer conv3_2 Relu layer conv3_2_relu, convolution layer conv3_3, relu layer conv3_3_relu, pooling layer max_pooling3, convolution layer conv4_1, relu layer conv4_1_relu, convolution layer conv4_2, relu layer conv4_2_relu, convolution layer conv4_3, relu layer conv4_3_relu, pooling layer max_pooling4, convolution layer conv5_1, relu layer conv5_1_relu, convolution layer conv5_2 Relu layer conv5_2_Relu, convolution layer conv5_3, relu layer conv5_3_Relu, pooling layer max_pooling5, convolution layer fc6, relu layer fc6_Relu, convolution layer fc7, relu layer fc7_Relu, convolution layer conv6_1, relu layer conv6_1_Relu, convolution layer conv6_2, relu layer conv6_2_Relu, convolution layer conv7_1, relu layer conv7_1_Relu, convolution layer conv7_2_Relu, convolution layer conv8_1, relu layer conv8_1_Relu, convolution layer conv8_2, relu layer conv8_2_Relu, convolution layer conv9_1, convolution layer conv9_2_conv2_conv10_con2_conv10_con2, convolution layer conv10_2_conv10_con2_con2.

3. The method for recognizing a vehicle face aiming at a license plate shielding according to claim 1, wherein the color prediction module structure is sequentially connected from an input layer to an output layer as follows: pooling layers global_pooling_1, fc layer fc_1, fc layer fc_2.

4. The method for recognizing the face of a vehicle aiming at shielding a license plate according to claim 1, wherein the global perception module structure is sequentially connected from an input layer to an output layer as follows: pooling layer global_pooling_2, multidimensional normalization layer conv1_1_in_bn, convolution layer conv11_1, relu layer conv11_1_relu, convolution layer conv11_2, relu layer conv11_2_relu; the multidimensional normalization layer conv1_1_in_bn is composed of a BN layer conv11_1_bn and an IN layer conv11_1_in.

5. The method for recognizing the face of the vehicle for shielding the license plate according to claim 1, wherein the detail perception module structure is sequentially connected from the input layer to the output layer as follows: feature detail compression layer horizontal_working_1, BN layer conv12_1_bn, convolution layer conv12_1, relu layer conv12_1_relu, convolution layer conv12_2, relu layer conv12_2_relu.

6. A method for recognizing a vehicle face for shielding a license plate according to claim 3, wherein the step S2 is as follows:

s21, feature map F _p Inputting to global average pooling layer global_pooling_1 to obtain feature map E _a ∈R ^C×1 ；

S22, combining the characteristic diagram E _a Input to fc layer fc_1, fc layer fc_2, and obtain vector E by softmax function _f ∈R ^N×1 Where N is the number of vehicle colors in the dataset;

s23, at vector E _f And (5) taking the position index i corresponding to the highest value, and outputting a vehicle color result i.

7. The method for recognizing a vehicle face for shielding a license plate according to claim 4, wherein the step S3 is as follows:

s31, feature map F _p Input to global_pooling_2 layer to obtain vector V e R ^D×1 ；

S32, inputting the vector V into the BN layer conv11_1_bn to obtain the vector V _b ∈R ^D×1 ；

S33, inputting the vector V into the IN layer conv11_1_in to obtain the vector V _i ∈R ^D×1 ；

S34, vector V _b Vector V of AND _i Splicing to obtain a vector V _c ∈R ^2D×1 ；

S35, vector V _c Vector Q is obtained by convolving layers conv11_1, relu layer conv11_1_relu, convolving layers conv11_2, relu layer conv11_2_relu ₁ 。

8. The method for recognizing the face of the vehicle for shielding the license plate according to claim 5, wherein the step S4 is as follows:

s41, feature map F _p Inputting to feature detail compression layer horizontal_compressing_1 and BN layer conv12_1_bn to obtain feature diagram J _s ∈R ^C×H×1 ；

S42, feature map J _s Input to convolution layers conv12_1, relu layer conv12_1_relu, convolution layers conv12_2, relu layer conv12_2_relu to obtain vector Q ₂ 。

9. The method for recognizing the face of a vehicle for shielding a license plate according to claim 5, wherein the working process of the feature detail compression layer is as follows:

assuming that the input of the feature detail compression layer is a feature map M E R ^C×W×H Output is a feature map Y ε R ^C×H×1 ，

For the feature map M, the feature map M is segmented into a feature map set A with the number of H in the height direction of the feature map ^* ＝{A _m 1.ltoreq.m.ltoreq.H, where A _m ∈R ^C×W×1 Refers to the set A ^* A feature map of the m-th bit in the list;

for set A ^* A certain characteristic diagram A _z Compressing to obtain result k _z The compression formula is as follows:

wherein A is _z,x,y For set A ^* Characteristic map A of the z-th position in (b) _z An xth row vector with the middle channel number of y;

for set A ^* Each feature map in the list is calculated by a formula (1) and is collected into a column vector T E R ^H×1 Wherein the value of the T-th position in the column vector T represents the set a ^* The feature map of the t-th position of (c) is calculated by the formula (1).

10. The method for recognizing a car face aiming at a license plate shielding according to claim 1, wherein the cosine distance is calculated as follows:

assume that vector J and vector B perform cosine distance calculation to obtain a value l, and the calculation expression is as follows:

l＝1-J·B ^T

in the formula, the symbol "·" represents the bitwise multiplication of elements in the vector, and the right-hand superscript "T" represents the vector transpose operation.

Images