CN116052150A - Vehicle face recognition method for shielding license plate - Google Patents

Abstract

The invention discloses a vehicle face recognition method aiming at a shielded license plate, which solves the technical problem of the existing recognition of the shielded license plate. According to the invention, color prediction information and picture vector information are obtained according to the color prediction module, the global perception module and the detail perception module, then a corresponding folder is found according to the color prediction information, cosine distance calculation is sequentially carried out on the corresponding folder and the picture vector information in the folder, and the first 5 are selected as recognition results according to the sequence from big to small. Compared with the traditional method, the method has the advantages that the auxiliary flow of vehicle color classification is increased, the identification is carried out through the global perception module and the detail perception module, multiple normalization calculation is carried out in the global perception module from batches and channels, namely, vehicles which shield license plates are searched and identified in a vehicle picture library from the multi-dimension of color appearance, global information and detail information, the content expression of searching capacity and models on the vehicles which shield the license plates is improved, and the vehicles which do not shield the license plates are successfully found out.

Description

Vehicle face recognition method for shielding license plate

Technical Field

The invention relates to the technical field of computer vision and pattern recognition, in particular to a vehicle face recognition method aiming at a license plate shielding.

Background

The home households all choose the automobile as the riding tool. However, under a complex traffic environment, some drivers may perform illegal actions such as overspeed and overload on a certain road, shelter from the license plate of the driver to avoid tracking of traffic police, and resume normal and normal running of the license plate after a period of time, so that the phenomenon becomes difficult to detect and monitor traffic control and control, and is difficult to solve. Although a large number of cameras are arranged on a road to monitor the tracking of illegal vehicles, the characteristics of the colors of the types of the vehicles are the same, and the detection of corresponding target pictures in a massive picture library shot by the non-overlapping cameras is difficult to realize by manpower, and a large amount of time is wasted. Therefore, a method is needed to find out the illegal vehicle by using a deep learning technology to find out the face picture of the blocked vehicle in the picture library in a non-blocking form according to the characteristics of the face in the face picture library. The existing deep learning-based method is to extract an effective global feature representation for each face picture, and lacks consideration of detail feature information. In addition, the feature extracted through the neural network is difficult to obtain information of appearance change, and the face recognition result obtained after sequencing is very different from the actual vehicle in the picture to be detected.

Disclosure of Invention

The invention aims to solve the defects in the prior art and provides a vehicle face recognition method aiming at a license plate shielding.

The aim of the invention can be achieved by adopting the following technical scheme:

a face recognition method for a license plate, the face recognition method comprising the steps of:

s1, inputting a certain face picture P in a data set into a backbone network to obtain a feature map F _p ∈R ^C×W×H Wherein C represents the number of channels of the feature map, W and H are the width and height of the feature map, and R represents the real number domain;

s2, mapping the characteristic diagram F _p The method comprises the steps of inputting the color of the vehicle into a color prediction module to predict the color of the vehicle, obtaining a predicted vehicle color result i, wherein the color prediction result i is used for step S6;

s3, mapping the characteristic diagram F _p Inputting to global perception module for prediction to obtain vector

Where D is the dimension of the column vector, vector Q ₁ The global information of the face picture is contained;

s4, mapping the characteristic diagram F _p Inputting to a detail perception module for prediction to obtain a vector

Vector Q ₂ The detail information of the face picture is contained;

s5, vector Q ₁ Sum vector Q ₂ Splicing to obtain a vector Q _r ∈R ^D×1 Vector Q _r Global information and detail information of the face picture are contained;

s6, according to the predicted vehicle color result i of the step S2, the vector Q is calculated _r A folder named i;

s7, sequentially and repeatedly executing the operations from the step S1 to the step S6 on all the face pictures in the picture library in the data set to obtain a vector set Q _i ＝{Q _i,k |1≤k≤I _n }, wherein I _n For the number of vectors in folder i, Q _i,k Refers to a vector corresponding to a kth picture in the folder i;

s8, inputting face pictures to be detected, and sequentially executing the step S1 to obtain a feature map F _q ；

S9, inputting a feature map F _q Executing step S2 to obtain a color prediction result of i _q The color prediction result i is used in step S11;

s10, inputting a feature map F _q Sequentially executing the operations of the steps S3, S4 and S5 to obtain a spliced vector Q _q ∈R ^D×1 Vector Q _q Global information and detail information of the face picture to be detected are contained;

s11, vector Q _q And file i _q Cosine distance calculation is carried out on all vectors in the set to obtain a set

I _nq For folder i _q Number of medium vectors, ++>

Refers to a folder i _q The corresponding vector of the g-th picture, and the color prediction classification processing enables the identification of the face picture to be more targeted;

s12, collecting

And sequencing the images according to the distance from large to small, and selecting the images corresponding to the top 5 vectors as prediction results.

Further, the backbone network structure is sequentially connected from an input layer to an output layer as follows: convolutional layers conv1_1, relu layer conv1_relu, convolutional layer conv1_2, relu layer conv1_2_relu, pooling layer max_pooling1, convolutional layer conv2_1, relu layer conv2_1_relu, convolutional layer conv2_2, BN layer conv2_2_bn, relu layer conv2_2_relu, pooling layer max_pooling2, convolutional layer conv3_1, relu layer conv3_1_relu, convolutional layer conv3_2 Relu layer conv3_2_relu, convolution layer conv3_3, relu layer conv3_3_relu, pooling layer max_pooling3, convolution layer conv4_1, relu layer conv4_1_relu, convolution layer conv4_2, relu layer conv4_2_relu, convolution layer conv4_3, relu layer conv4_3_relu, pooling layer max_pooling4, convolution layer conv5_1, relu layer conv5_1_relu, convolution layer conv5_2 Relu layer conv5_2_Relu, convolution layer conv5_3, relu layer conv5_3_Relu, pooling layer max_pooling5, convolution layer fc6, relu layer fc6_Relu, convolution layer fc7, relu layer fc7_Relu, convolution layer conv6_1, relu layer conv6_1_Relu, convolution layer conv6_2, relu layer conv6_2_Relu, convolution layer conv7_1, relu layer conv7_1_Relu, convolution layer conv7_2_Relu, convolution layer conv8_1, relu layer conv8_1_Relu, convolution layer conv8_2, relu layer conv8_2_Relu, convolution layer conv9_1, convolution layer conv9_2_conv2_conv10_con2_conv10_con2, convolution layer conv10_2_conv10_con2_con2.

Further, the color prediction module structure is sequentially connected from the input layer to the output layer as follows: pooling layers global_pooling_1, fc layer fc_1, fc layer fc_2.

Further, the global perception module structure is sequentially connected from the input layer to the output layer as follows: pooling layer global_pooling_2, multidimensional normalization layer conv1_1_in_bn, convolution layer conv11_1, relu layer conv11_1_relu, convolution layer conv11_2, relu layer conv11_2_relu; the multidimensional normalization layer conv1_1_in_bn is composed of a BN layer conv11_1_bn and an IN layer conv11_1_in.

Further, the detail perception module structure is sequentially connected from the input layer to the output layer as follows: feature detail compression layer horizontal_working_1, BN layer conv12_1_bn, convolution layer conv12_1, relu layer conv12_1_relu, convolution layer conv12_2, relu layer conv12_2_relu.

Further, the step S2 is as follows:

s21, feature map F _p Inputting to global average pooling layer global_pooling_1 to obtain feature map E _a ∈R ^C×1 For characteristic diagram F _p Is compressed according to the information;

s22, combining the characteristic diagram E _a Input to fc layer fc_1, fc layer fc_2, and obtain vector E by softmax function _f ∈R ^{N ×1} Wherein N is the number of vehicle colors in the data set, and the vehicle colors are predicted through the full-connection structure of the two fc layers;

s23, at vector E _f The position index i corresponding to the highest value is obtained, and a vehicle color result i is output;

further, the step S3 is as follows:

s31, feature map F _p Input to global_pooling_2 layer to obtain vector V e R ^D×1 For characteristic diagram F _p Is compressed according to the information;

s32, inputting the vector V into the BN layer conv11_1_bn to obtain the vector V _b ∈R ^D×1 The method comprises the steps of carrying out a first treatment on the surface of the Vector V _b The vector V is normalized from the batch direction;

s33, inputting the vector V into the IN layer conv11_1_in to obtain the vector V _i ∈R ^D×1 The method comprises the steps of carrying out a first treatment on the surface of the Vector V _i Is the vector V normalized from the channel directionResults of the formatting;

s34, vector V _b Vector V of AND _i Splicing to obtain a vector V _c ∈R ^2D×1 ；

S35, vector V _c Vector Q is obtained by convolving layers conv11_1, relu layer conv11_1_relu, convolving layers conv11_2, relu layer conv11_2_relu ₁ Thus vector Q ₁ And the data spliced by the vector V from batch normalization and channel normalization are obtained, so that the loss of overfitting and wind pattern characteristics is avoided.

Further, the step S4 is as follows:

s41, feature map F _p Input into a feature detail compression layer horizontal_compressing_1 for compression, and input into a BN layer conv12_1_bn for normalization to obtain a feature diagram J _s ∈R ^C×H×1 ；

S42, feature map J _s Input to convolution layers conv12_1, relu layer conv12_1_relu, convolution layers conv12_2, relu layer conv12_2_relu to obtain vector Q ₂ 。

Further, the feature detail compression layer is used for compressing detail features to facilitate subsequent identification, and the working process is as follows:

assuming that the input of the feature detail compression layer is a feature map M E R ^C×W×H Output is a feature map Y ε R ^C×H×1 ，

For the feature map M, the feature map M is segmented into a feature map set A with the number of H in the height direction of the feature map ^* ＝{A _m 1.ltoreq.m.ltoreq.H, where A _m ∈R ^C×W×1 Refers to the set A ^* A feature map of the m-th bit in the list;

for set A ^* A certain characteristic diagram A _z Compressing to obtain result k _z The compression formula is as follows:

wherein A is _z,x,y For set A ^* Feature map A of the z-th position of (2) _z An xth row vector with the middle channel number of y;

for set A ^* Each feature map in the list is calculated by a formula (1) and is collected into a column vector T E R ^H×1 Wherein the value of the T-th position in the column vector T represents the set a ^* The feature map of the t-th position of (c) is calculated by the formula (1).

Further, the cosine distance is used for calculating the similarity of two vectors, and the calculation process is as follows:

assume that vector J and vector B perform cosine distance calculation to obtain a value l, and the calculation expression is as follows:

l＝1-J·B ^T 。

in the formula, the symbol "·" represents the bitwise multiplication of elements in the vector, and the right-hand superscript "T" represents the vector transpose operation.

Compared with the prior art, the invention has the following advantages and effects:

(1) According to the invention, 5 vehicles similar to the picture of the shielded license plate can be found out from the picture library, and the picture of the non-shielded vehicle matched with the picture can be found out from the 5 pictures.

(2) Compared with the traditional method, the method improves the normalization of the neural network and the feature vector, and improves the expression capability of the feature vector on the face picture.

(3) Compared with the traditional method, the invention adds the color prediction module, as the auxiliary of the later recognition, classifies the vehicle faces according to the colors of the vehicle faces and stores the vehicle faces in folders with different colors, so that the recognition of the vehicle faces is more targeted.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiments of the invention and together with the description serve to explain the invention and do not constitute a limitation on the invention. In the drawings:

fig. 1 is a flowchart of a method for recognizing a face for blocking a license plate disclosed in the present invention.

Fig. 2 is a schematic diagram of a face recognition method for a license plate shielding method disclosed in the present invention.

FIG. 3 is a bar graph of accuracy of the present method versus other methods using the VehicleID dataset under the precondition of example 1;

FIG. 4 is a bar graph of accuracy of the present method versus other methods using the VeRi dataset with example 2;

fig. 5 is a block diagram of a backbone network in the present invention.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Example 1

The Vehiccleid dataset was selected for use in this example 1. Each picture in the dataset carries an id tag corresponding to an identity in the real world. The training set is 13164 vehicles, and 113346 pictures are taken in total. The test set is 2400 vehicles, and 19777 pictures are taken.

The method comprises the following steps:

step S1, acquiring a number N of vehicle color categories in a data set, wherein n=9, the number of color categories being used for dimension setting in an fc layer fc_2 in a color prediction module; taking 1000 pictures randomly from the test set according to the id tag of the vehicle to serve as a picture library to be tested;

in step S2, the experimental environment in this embodiment only needs a general hardware configuration and a graphics processing unit (Graphics Processing Unit, GPU) capable of increasing the computing speed to perform the acceleration operation. The model is built, trained and the test of training results are completed under a Pytorch deep learning framework and a FastRIID tool, and a calculation unified architecture (Compute Unified Device Architecture, CUDA) is used to enable the GPU to solve the complex calculation problem. The specific operating environment configuration required for the experiment in this example is shown in table 1.

TABLE 1 experiment operation Environment configuration Table of this embodiment

Step S3, constructing a vehicle face recognition method aiming at a license plate shielding, wherein a structural diagram is shown in FIG. 2, and the specific construction steps are as follows:

step S31, a backbone network is constructed, as shown in fig. 5, where the backbone network structure is sequentially connected from an input layer to an output layer: convolutional layers conv1_1, relu layer conv1_relu, convolutional layer conv1_2, relu layer conv1_2_relu, pooling layer max_pooling1, convolutional layer conv2_1, relu layer conv2_1_relu, convolutional layer conv2_2, BN layer conv2_2_bn, relu layer conv2_2_relu, pooling layer max_pooling2, convolutional layer conv3_1, relu layer conv3_1_relu, convolutional layer conv3_2 Relu layer conv3_2_relu, convolution layer conv3_3, relu layer conv3_3_relu, pooling layer max_pooling3, convolution layer conv4_1, relu layer conv4_1_relu, convolution layer conv4_2, relu layer conv4_2_relu, convolution layer conv4_3, relu layer conv4_3_relu, pooling layer max_pooling4, convolution layer conv5_1, relu layer conv5_1_relu, convolution layer conv5_2 Relu layer conv5_2_relu, convolution layer conv5_3, relu layer conv5_3_relu, pooling layer max_pooling5, convolution layer fc6, relu layer fc6_relu, convolution layer fc7, relu layer fc7_relu, convolution layer conv6_1, relu layer conv6_1_relu, convolution layer conv6_2, relu layer conv6_2_relu, convolution layer conv7_1, relu layer conv7_1_relu convolutionally layer conv7_2, reeu layer conv7_2_reelu, convolutionally layer conv8_1_reelu, convolutionally layer conv8_2, reeu layer conv8_2_reelu, convolutionally layer conv9_1, reeu layer conv9_1_reelu, convolutionally layer conv9_2, reeu layer conv9_2_reelu, convolutionally layer conv10_1, reeu layer conv10_reelu, convolutionally layer conv10_2, reeu layer conv10_2_reelu;

step S32, constructing a color prediction module, wherein the color prediction module structure is sequentially connected from an input layer to an output layer as follows: pooling layers global_pooling_1, fc layer fc_1, fc layer fc_2;

step S33, constructing a global perception module, wherein the global perception module structure is sequentially connected from an input layer to an output layer as follows: pooling layer global_pooling_2, multidimensional normalization layer conv1_1_in_bn, convolution layer conv11_1, relu layer conv11_1_relu, convolution layer conv11_2, relu layer conv11_2_relu; the multi-dimensional normalization layer conv1_1_in_bn is composed of a BN layer conv11_1_bn and an IN layer conv11_1_in;

step S34, constructing a detail perception module, wherein the detail perception module structure is sequentially connected from an input layer to an output layer as follows: feature detail compression layer horizontal_working_1, BN layer conv12_1_bn, convolution layer conv12_1, relu layer conv12_1_relu, convolution layer conv12_2, relu layer conv12_2_relu;

and S4, dividing the pictures of the training set into a picture library and a library to be queried according to the vehicle ids, wherein the vehicle ids in the library to be queried are unique, and the picture library can contain a plurality of pictures with the same vehicle ids. The training set pictures after being processed are trained by the method. In the training process, the optimizer used was Adam, the learning rate was set to 0.00035, the learning rate momentum was set to 0.0005, the training batch size was set to 64, epoch was set to 60, and the learning rate was decreased by a factor of 0.1 at epoch of 30 and 50.

S5, inputting a certain face picture P in the data set into a backbone network to obtain a feature map F _p ∈R ^C×W×H Wherein C represents the number of channels of the feature map, W and H are the width and height of the feature map, and R represents the real number domain;

step S6, feature map F _p The color prediction module is used for predicting the color of the vehicle to obtain a predicted vehicle color result i, wherein i is more than or equal to 1 and less than or equal to N;

the specific process is as follows:

s61, feature map F _p Inputting to global average pooling layer global_pooling_1 to obtain feature map E _a ∈R ^C×1 For characteristic diagram F _p Is compressed according to the information;

s62, feature map E _a Input to fc layer fc_1, fc layer fc_2, and obtain vector E by softmax function _f ∈R ^{N ×1} Predicting the color of the vehicle through the full-connection structure of the two fc layers;

s63, at vector E _f Is taken inThe position index i corresponding to the highest value outputs a vehicle color result i

Step S7, feature map F _p Inputting to global perception module for prediction to obtain vector

Where D is the dimension of the column vector, vector Q ₁ The global information of the face picture is contained; in example 1, d=2048;

the specific process is as follows:

s71, feature map F _p Input to global_pooling_2 layer to obtain vector V e R ^D×1 ；

S72, inputting the vector V into the BN layer conv11_1_bn to obtain the vector V _b ∈R ^D×1 ；

S73, inputting the vector V into the IN layer conv11_1_in to obtain the vector V _i ∈R ^D×1 ；

S74, vector V _b Vector V of AND _i Splicing to obtain a vector V _c ∈R ^2D×1 ；

S75, vector V _c Vector Q is obtained by convolving layers conv11_1, relu layer conv11_1_relu, convolving layers conv11_2, relu layer conv11_2_relu ₁ 。

Step S8, feature map F _p Inputting to a detail perception module for prediction to obtain a vector

Vector Q ₂ The detail information of the face picture is contained;

the specific process is as follows:

s41, feature map F _p Input into a feature detail compression layer horizontal_compressing_1 for compression, and input into a BN layer conv12_1_bn for normalization to obtain a feature diagram J _s ∈R ^C×H×1 ；

S42, feature map J _s Input to convolution layers conv12_1, relu layer conv12_1_relu, convolution layers conv12_2, relu layer conv12_2_relu to obtain vector Q ₂ 。

The working process of the characteristic detail compression layer in the detail perception module is as follows:

assuming that the input of the feature detail compression layer is a feature map M E R ^C×W×H Output is a feature map Y ε R ^C×H×1 ：

For the feature map M, the feature map M is segmented into a feature map set A with the number of H in the height direction of the feature map ^* ＝{A _m 1.ltoreq.m.ltoreq.H, where A _m ∈R ^C×W×1 Refers to the set A ^* A feature map of the m-th bit in the list;

for set A ^* A certain characteristic diagram A _z Compressing to obtain result k _z The compression formula is as follows:

wherein A is _z,x,y For set A ^* Feature map A of the z-th position of (2) _z An xth row vector with the middle channel number of y;

for set A ^* Each feature map in the list is calculated by a formula (1) and is collected into a column vector T E R ^H×1 Wherein the value of the T-th position in the column vector T represents the set a ^* The feature map of the t-th position of (c) is calculated by the formula (1).

Step S9, vector Q ₁ Sum vector Q ₂ Splicing to obtain a vector Q _r ∈R ^D×1 Vector Q _r Global information and detail information of the face picture are contained;

step S10, according to the predicted vehicle color result i of step S6, vector Q is calculated _r A folder named i;

step S11, all face pictures in the picture library in the test set are sequentially and completely executed with the operations from step S5 to step S10 to obtain a vector set Q _i ＝{Q _i,k |1≤k≤I _n }, wherein I _n For the number of vectors in folder i, Q _i,k Refers to a vector corresponding to a kth picture in the folder i;

step S12, inputting the face picture to be detected, and sequentially executing step S5 to obtain a feature map F _q ；

Step S13, input a feature map F _q Executing step S6 to obtain a color prediction result of i _q The color prediction result i is used in step S11;

step S14, inputting a feature map F _q Sequentially executing the operations of the steps S7, S8 and S9 to obtain a spliced vector Q _q ∈R ^{D ×1} Vector Q _q Global information and detail information of the face picture to be detected are contained;

step S15, vector Q _q And file i _q Cosine distance calculation is carried out on all vectors in the set to obtain a set

I _nq For folder i _q Number of medium vectors, ++>

Refers to a folder i _q The corresponding vector of the g-th picture, and the color prediction classification processing enables the identification of the face picture to be more targeted;

the cosine distance is calculated as follows:

assume that vector J and vector B perform cosine distance calculation to obtain a value l, and the calculation expression is as follows:

l＝1-J·B ^T 。

in the formula, the symbol "·" represents the bitwise multiplication of elements in the vector, and the right-hand superscript "T" represents the vector transpose operation.

Step S16, gathering

And sequencing the images according to the distance from large to small, and selecting the images corresponding to the top 5 vectors as prediction results.

Step S17, sequentially executing steps S12, S13, S14 and S15 on the picture libraries to be tested in the test set, wherein the specific flow is shown in FIG. 1, obtaining all prediction results of the picture libraries to be tested in the test set, and calculating the accuracy.

The experimental comparison result of example 1 is shown in fig. 3, and example 1 is compared with other methods on a 1000-number picture library to be tested, the 5 pictures contain the corresponding picture marks as accurate, the method obtains 88.1% of accuracy, 70.21% higher than the BOW-CN method, 62.47% higher than the LOMO method, 27.61% higher than the FACT method, 27.38% higher than the NuFACT method, 17.81% higher than the VAML, and 4.73% higher than the QD-DLF, the effectiveness of the method is proved, and the corresponding unoccluded license plate can be successfully identified.

Example 2

In this example 2, the disclosed VeRi dataset was selected, which contained more than 50,000 images of 776 vehicles, including 37778 training sets, 11579 pictures in the library, and 1678 pictures to be tested. Each picture in the dataset carries an id tag corresponding to an identity in the real world.

The method comprises the following steps:

step S1, acquiring a number N of vehicle color categories in a data set, wherein n=9, the number of color categories being used for dimension setting in an fc layer fc_2 in a color prediction module;

in step S2, the experimental environment in this embodiment only needs a general hardware configuration and a graphics processing unit (Graphics Processing Unit, GPU) capable of increasing the computing speed to perform the acceleration operation. The model is built, trained and the test of training results are completed under a Pytorch deep learning framework and a Fasteid tool, and a calculation unified architecture (Compute Unified Device Architecture, CUDA) is used, so that the GPU can solve the complex calculation problem. Specifically, the hardware and software adopted in this embodiment are configured to: the GPU version is GeForce RTX 2080Ti; the CPU version is Intel (R) Xeon (R) Silver 4216CPU@2.10GHz; the version of the operating system is centOS 8.3.2011; python version 3.6.13; CUDA version 11.2.

Step S3, constructing a vehicle face recognition method aiming at a license plate shielding, wherein a structural diagram is shown in FIG. 2, and the specific construction steps are as follows:

step S31, a backbone network is constructed, as shown in fig. 5, where the backbone network structure is sequentially connected from an input layer to an output layer: convolutional layers conv1_1, relu layer conv1_relu, convolutional layer conv1_2, relu layer conv1_2_relu, pooling layer max_pooling1, convolutional layer conv2_1, relu layer conv2_1_relu, convolutional layer conv2_2, BN layer conv2_2_bn, relu layer conv2_2_relu, pooling layer max_pooling2, convolutional layer conv3_1, relu layer conv3_1_relu, convolutional layer conv3_2 Relu layer conv3_2_relu, convolution layer conv3_3, relu layer conv3_3_relu, pooling layer max_pooling3, convolution layer conv4_1, relu layer conv4_1_relu, convolution layer conv4_2, relu layer conv4_2_relu, convolution layer conv4_3, relu layer conv4_3_relu, pooling layer max_pooling4, convolution layer conv5_1, relu layer conv5_1_relu, convolution layer conv5_2 Relu layer conv5_2_relu, convolution layer conv5_3, relu layer conv5_3_relu, pooling layer max_pooling5, convolution layer fc6, relu layer fc6_relu, convolution layer fc7, relu layer fc7_relu, convolution layer conv6_1, relu layer conv6_1_relu, convolution layer conv6_2, relu layer conv6_2_relu, convolution layer conv7_1, relu layer conv7_1_relu convolutionally layer conv7_2, reeu layer conv7_2_reelu, convolutionally layer conv8_1_reelu, convolutionally layer conv8_2, reeu layer conv8_2_reelu, convolutionally layer conv9_1, reeu layer conv9_1_reelu, convolutionally layer conv9_2, reeu layer conv9_2_reelu, convolutionally layer conv10_1, reeu layer conv10_reelu, convolutionally layer conv10_2, reeu layer conv10_2_reelu;

step S32, constructing a color prediction module, wherein the color prediction module structure is sequentially connected from an input layer to an output layer as follows: pooling layers global_pooling_1, fc layer fc_1, fc layer fc_2;

step S33, constructing a global perception module, wherein the global perception module structure is sequentially connected from an input layer to an output layer as follows: pooling layer global_pooling_2, multidimensional normalization layer conv1_1_in_bn, convolution layer conv11_1, relu layer conv11_1_relu, convolution layer conv11_2, relu layer conv11_2_relu; the multi-dimensional normalization layer conv1_1_in_bn is composed of a BN layer conv11_1_bn and an IN layer conv11_1_in;

step S34, constructing a detail perception module, wherein the detail perception module structure is sequentially connected from an input layer to an output layer as follows: feature detail compression layer horizontal_working_1, BN layer conv12_1_bn, convolution layer conv12_1, relu layer conv12_1_relu, convolution layer conv12_2, relu layer conv12_2_relu;

and S4, dividing the pictures of the training set into a picture library and a library to be queried according to the vehicle ids, wherein the vehicle ids in the library to be queried are unique, and the picture library can contain a plurality of pictures with the same vehicle ids. The training set pictures after being processed are trained by the method. In the training process, the optimizer used was SGD, the learning rate was set to 0.001, the learning rate momentum was set to 0.0005, the training batch size was set to 64, epoch was set to 60, and the learning rate was decreased by a factor of 0.1 at epoch of 30 and 50.

S5, inputting a certain face picture P in the data set into a backbone network to obtain a feature map F _p ∈R ^C×W×H Wherein C represents the number of channels of the feature map, W and H are the width and height of the feature map, and R represents the real number domain;

step S6, feature map F _p The color prediction module is used for predicting the color of the vehicle to obtain a predicted vehicle color result i, wherein i is more than or equal to 1 and less than or equal to N;

the specific process is as follows:

s21, feature map F _p Inputting to global average pooling layer global_pooling_1 to obtain feature map E _a ∈R ^C×1 For characteristic diagram F _p Is compressed according to the information;

s22, combining the characteristic diagram E _a Input to fc layer fc_1, fc layer fc_2, and obtain vector E by softmax function _f ∈R ^{N ×1} Predicting the color of the vehicle through the full-connection structure of the two fc layers;

s23, at vector E _f And (5) taking the position index i corresponding to the highest value, and outputting a vehicle color result i.

Step S7, feature map F _p Inputting to global perception module for prediction to obtain vector

Where D is the dimension of the column vector, vector Q ₁ The global information of the face picture is contained; in example 2, d=2048;

the specific process is as follows:

s71, feature map F _p Input to global_pooling_2 layer to obtain vector V e R ^D×1 ；

S72, inputting the vector V into the BN layer conv11_1_bn to obtain the vector V _b ∈R ^D×1 ；

S73, inputting the vector V into the IN layer conv11_1_in to obtain the vector V _i ∈R ^D×1 ；

S74, vector V _b Vector V of AND _i Splicing to obtain a vector V _c ∈R ^2D×1 ；

S75, vector V _c Vector Q is obtained by convolving layers conv11_1, relu layer conv11_1_relu, convolving layers conv11_2, relu layer conv11_2_relu ₁ 。

Step S8, feature map F _p Inputting to a detail perception module for prediction to obtain a vector

Vector Q ₂ The detail information of the face picture is contained;

the specific process is as follows:

s41, feature map F _p Input into a feature detail compression layer horizontal_compressing_1 for compression, and input into a BN layer conv12_1_bn for normalization to obtain a feature diagram J _s ∈R ^C×H×1 ；

S42, feature map J _s Input to convolution layers conv12_1, relu layer conv12_1_relu, convolution layers conv12_2, relu layer conv12_2_relu to obtain vector Q ₂ ；

The working process of the characteristic detail compression layer in the detail perception module is as follows:

presuming feature detail compressionThe layer input is a feature map M E R ^C×W×H Output is a feature map Y ε R ^C×H×1 ：

For the feature map M, the feature map M is segmented into a feature map set A with the number of H in the height direction of the feature map ^* ＝{A _m 1.ltoreq.m.ltoreq.H, where A _m ∈R ^C×W×1 Refers to the set A ^* A feature map of the m-th bit in the list;

for set A ^* A certain characteristic diagram A _z Compressing to obtain result k _z The compression formula is as follows:

wherein A is _z,x,y For set A ^* Feature map A of the z-th position of (2) _z An xth row vector with the middle channel number of y;

for set A ^* Each feature map in the list is calculated by a formula (1) and is collected into a column vector T E R ^H×1 Wherein the value of the T-th position in the column vector T represents the set a ^* The feature map of the t-th position of (c) is calculated by the formula (1).

Step S9, vector Q ₁ Sum vector Q ₂ Splicing to obtain a vector Q _r ∈R ^D×1 Vector Q _r Global information and detail information of the face picture are contained;

step S10, according to the predicted vehicle color result i of step S6, vector Q is calculated _r A folder named i;

step S11, all face pictures in the picture library are sequentially and completely executed with the operations from step S5 to step S10 to obtain a vector set Q _i ＝{Q _i,k |1≤k≤I _n }, wherein I _n For the number of vectors in folder i, Q _i,k Refers to a vector corresponding to a kth picture in the folder i;

step S12, inputting the face picture to be detected, and sequentially executing step S5 to obtain a feature map F _q ；

Step S13, input a feature map F _q Step S6 is executed to obtain a color prediction resultIs i _q The color prediction result i is used in step S11;

step S14, inputting a feature map F _q Sequentially executing the operations of the steps S7, S8 and S9 to obtain a spliced vector Q _q ∈R ^{D ×1} Vector Q _q Global information and detail information of the face picture to be detected are contained;

step S15, vector Q _q And file i _q Cosine distance calculation is carried out on all vectors in the set to obtain a set

I _nq For folder i _q Number of medium vectors, ++>

Refers to a folder i _q The corresponding vector of the g-th picture, and the color prediction classification processing enables the identification of the face picture to be more targeted;

the cosine distance is calculated as follows:

assume that vector J and vector B perform cosine distance calculation to obtain a value l, and the calculation expression is as follows:

l＝1-J·B ^T 。

in the formula, the symbol "·" represents the bitwise multiplication of elements in the vector, and the right-hand superscript "T" represents the vector transpose operation.

Step S16, gathering

And sequencing the images according to the distance from large to small, and selecting the images corresponding to the top 5 vectors as prediction results.

Step S17, sequentially executing steps S12, S13, S14 and S15 on the picture libraries to be tested in the test set, wherein the specific flow is shown in FIG. 1, obtaining all prediction results of the picture libraries to be tested in the test set, and calculating the accuracy.

The experimental comparison result of example 2 is shown in fig. 4, and example 2 is compared with other methods on a 1678-number picture library to be tested, the corresponding picture marks are included in 5 pictures to be accurate, the method obtains 97.3% of accuracy, which is 43.61% higher than the BOW-CN method, 50.82% higher than the LOMO method, 24.42% higher than the FACT method, 5.88% higher than the NuFACT method, 6.48% higher than the VAML, and 2.84% higher than the QD-DLF, the effectiveness of the method is proved, and the corresponding unoccluded license plate can be successfully identified.

In summary, the above embodiments disclose a face recognition method for shielding license plates. According to the invention, color prediction information and picture vector information are obtained according to the color prediction module, the global perception module and the detail perception module, then a corresponding folder is found according to the color prediction information, cosine distance calculation is sequentially carried out on the corresponding folder and the picture vector information in the folder, and the first 5 are selected as recognition results according to the sequence from big to small. The technical problem of current discernment shielding license plate is solved. Compared with the traditional method, the method has the advantages that the auxiliary flow of vehicle color classification is increased, the global perception module and the detail perception module are used for carrying out recognition, multiple normalization calculation is carried out in the global perception module from batches and channels, the searching capability and the content expression of the model on the shielded vehicles can be improved, and the vehicles in the corresponding non-shielded license plate states can be successfully found out. The high recognition rate of the method can be represented by the experimental results of example 1 and example 2.

The above examples are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above examples, and any other changes, modifications, substitutions, combinations, and simplifications that do not depart from the spirit and principle of the present invention should be made in the equivalent manner, and the embodiments are included in the protection scope of the present invention.

Claims (10)

1. The vehicle face recognition method for the license plate shielding is characterized by comprising the following steps of:

s1, inputting a certain face picture P in a data set into a backbone network to obtain a feature map F _p ∈R ^C×E×H Wherein C represents the number of channels of the feature map, W and H are the width and height of the feature map, and R represents the real number domain;

s2, mapping the characteristic diagram F _p Input to colorThe method comprises the steps that a prediction module predicts the color of a vehicle to obtain a predicted vehicle color result i;

s3, mapping the characteristic diagram F _p Inputting to global perception module for prediction to obtain vector

Wherein D is the dimension of the column vector;

s4, mapping the characteristic diagram F _p Inputting to a detail perception module for prediction to obtain a vector

S5, vector Q ₁ Sum vector Q ₂ Splicing to obtain a vector Q _r ∈R ^D×1 ；

S6, according to the predicted vehicle color result i of the step S2, the vector Q is calculated _r A folder named i;

s7, sequentially and repeatedly executing the operations from the step S1 to the step S6 on all the face pictures in the picture library in the data set to obtain a vector set Q _i ＝{Q _i,k |1≤k≤I _n }, wherein I _n For the number of vectors in folder i, Q _i,k Refers to a vector corresponding to a kth picture in the folder i;

s8, inputting face pictures to be detected, and sequentially executing the step S1 to obtain a feature map F _q ；

S9, inputting a feature map F _q Executing step S2 to obtain a color prediction result of i _q ；

S10, inputting a feature map F _q Sequentially executing the operations of the steps S3, S4 and S5 to obtain a spliced vector Q _q ∈R ^D×1 ；

S11, vector Q _q And file i _q Cosine distance calculation is carried out on all vectors in the set to obtain a set

I _nq For folder i _q Number of medium vectors, ++>

Refers to a folder i _q Vector corresponding to the g-th picture;

s12, collecting

And sequencing the images according to the distance from large to small, and selecting the images corresponding to the top 5 vectors as prediction results.

2. The method for recognizing the face of a vehicle for shielding a license plate according to claim 1, wherein the backbone network structure is sequentially connected from an input layer to an output layer as follows: convolutional layers conv1_1, relu layer conv1_relu, convolutional layer conv1_2, relu layer conv1_2_relu, pooling layer max_pooling1, convolutional layer conv2_1, relu layer conv2_1_relu, convolutional layer conv2_2, BN layer conv2_2_bn, relu layer conv2_2_relu, pooling layer max_pooling2, convolutional layer conv3_1, relu layer conv3_1_relu, convolutional layer conv3_2 Relu layer conv3_2_relu, convolution layer conv3_3, relu layer conv3_3_relu, pooling layer max_pooling3, convolution layer conv4_1, relu layer conv4_1_relu, convolution layer conv4_2, relu layer conv4_2_relu, convolution layer conv4_3, relu layer conv4_3_relu, pooling layer max_pooling4, convolution layer conv5_1, relu layer conv5_1_relu, convolution layer conv5_2 Relu layer conv5_2_Relu, convolution layer conv5_3, relu layer conv5_3_Relu, pooling layer max_pooling5, convolution layer fc6, relu layer fc6_Relu, convolution layer fc7, relu layer fc7_Relu, convolution layer conv6_1, relu layer conv6_1_Relu, convolution layer conv6_2, relu layer conv6_2_Relu, convolution layer conv7_1, relu layer conv7_1_Relu, convolution layer conv7_2_Relu, convolution layer conv8_1, relu layer conv8_1_Relu, convolution layer conv8_2, relu layer conv8_2_Relu, convolution layer conv9_1, convolution layer conv9_2_conv2_conv10_con2_conv10_con2, convolution layer conv10_2_conv10_con2_con2.

3. The method for recognizing a vehicle face aiming at a license plate shielding according to claim 1, wherein the color prediction module structure is sequentially connected from an input layer to an output layer as follows: pooling layers global_pooling_1, fc layer fc_1, fc layer fc_2.

4. The method for recognizing the face of a vehicle aiming at shielding a license plate according to claim 1, wherein the global perception module structure is sequentially connected from an input layer to an output layer as follows: pooling layer global_pooling_2, multidimensional normalization layer conv1_1_in_bn, convolution layer conv11_1, relu layer conv11_1_relu, convolution layer conv11_2, relu layer conv11_2_relu; the multidimensional normalization layer conv1_1_in_bn is composed of a BN layer conv11_1_bn and an IN layer conv11_1_in.

5. The method for recognizing the face of the vehicle for shielding the license plate according to claim 1, wherein the detail perception module structure is sequentially connected from the input layer to the output layer as follows: feature detail compression layer horizontal_working_1, BN layer conv12_1_bn, convolution layer conv12_1, relu layer conv12_1_relu, convolution layer conv12_2, relu layer conv12_2_relu.

6. A method for recognizing a vehicle face for shielding a license plate according to claim 3, wherein the step S2 is as follows:

s21, feature map F _p Inputting to global average pooling layer global_pooling_1 to obtain feature map E _a ∈R ^C×1 ；

S22, combining the characteristic diagram E _a Input to fc layer fc_1, fc layer fc_2, and obtain vector E by softmax function _f ∈R ^N×1 Where N is the number of vehicle colors in the dataset;

s23, at vector E _f And (5) taking the position index i corresponding to the highest value, and outputting a vehicle color result i.

7. The method for recognizing a vehicle face for shielding a license plate according to claim 4, wherein the step S3 is as follows:

s31, feature map F _p Input to global_pooling_2 layer to obtain vector V e R ^D×1 ；

S32, inputting the vector V into the BN layer conv11_1_bn to obtain the vector V _b ∈R ^D×1 ；

S33, inputting the vector V into the IN layer conv11_1_in to obtain the vector V _i ∈R ^D×1 ；

S34, vector V _b Vector V of AND _i Splicing to obtain a vector V _c ∈R ^2D×1 ；

S35, vector V _c Vector Q is obtained by convolving layers conv11_1, relu layer conv11_1_relu, convolving layers conv11_2, relu layer conv11_2_relu ₁ 。

8. The method for recognizing the face of the vehicle for shielding the license plate according to claim 5, wherein the step S4 is as follows:

s41, feature map F _p Inputting to feature detail compression layer horizontal_compressing_1 and BN layer conv12_1_bn to obtain feature diagram J _s ∈R ^C×H×1 ；

S42, feature map J _s Input to convolution layers conv12_1, relu layer conv12_1_relu, convolution layers conv12_2, relu layer conv12_2_relu to obtain vector Q ₂ 。

9. The method for recognizing the face of a vehicle for shielding a license plate according to claim 5, wherein the working process of the feature detail compression layer is as follows:

assuming that the input of the feature detail compression layer is a feature map M E R ^C×W×H Output is a feature map Y ε R ^C×H×1 ，

For the feature map M, the feature map M is segmented into a feature map set A with the number of H in the height direction of the feature map ^* ＝{A _m 1.ltoreq.m.ltoreq.H, where A _m ∈R ^C×W×1 Refers to the set A ^* A feature map of the m-th bit in the list;

for set A ^* A certain characteristic diagram A _z Compressing to obtain result k _z The compression formula is as follows:

wherein A is _z,x,y For set A ^* Characteristic map A of the z-th position in (b) _z An xth row vector with the middle channel number of y;

for set A ^* Each feature map in the list is calculated by a formula (1) and is collected into a column vector T E R ^H×1 Wherein the value of the T-th position in the column vector T represents the set a ^* The feature map of the t-th position of (c) is calculated by the formula (1).

10. The method for recognizing a car face aiming at a license plate shielding according to claim 1, wherein the cosine distance is calculated as follows:

assume that vector J and vector B perform cosine distance calculation to obtain a value l, and the calculation expression is as follows:

l＝1-J·B ^T

in the formula, the symbol "·" represents the bitwise multiplication of elements in the vector, and the right-hand superscript "T" represents the vector transpose operation.

Images