JP7246104B2 - License plate identification method based on text line identification - Google Patents

Description

The present invention relates to a license plate identification technique based on image processing and mode identification, and more particularly to a license plate identification method based on text line identification.

License plate detection and identification is a typical computer vision field task and has broad application potential in intelligent traffic systems. With the development of modern transportation systems, traffic volume increases rapidly, and license plate identification systems can assist traffic management, public safety, and so on.

Over the last decade or so, the license plate identification problem has received widespread attention in the industry. Considering various factors affecting image quality, such as shooting environment (lighting, position, defocus, etc.), picture quality (resolution, etc.) and complex shooting backgrounds, the task of license plate identification in arbitrary scenes remains challenging. facing.

The identification methods of some existing license plate identification systems mainly include the steps of license plate detection, character segmentation and scene character identification. License plate identification can be summarized in two parts: detecting the position of the license plate from the natural image and identifying the textual information on the detected license plate. Among the workflows of some existing license plate identification systems, some focus on achieving a complete workflow from the input natural image to the output text content, while another focuses on the accuracy of identification. Vehicle detection is added before license plate detection to improve

Existing license plate identification methods can be divided into deep learning-based methods and non-deep learning-based methods. Before the development of deep learning, the license plate is roughly identified, generally based on color information, text information, or edge information of the license plate. The methods used are typically Finite Boltzmann Machines or Support Vector Machines.

In recent years, with the development of deep learning, the license plate identification method based on character segmentation is relatively popular. Character segmentation-based methods require pre-segmented training data, which makes tagging the training data difficult, and it utilizes font files to automatically generate images. The effectiveness and robustness of license plate identification is relatively low.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a license plate identification method based on text line identification with improved identification effectiveness and robustness, so as to overcome the deficiencies of the above prior art.

The object of the present invention may be achieved by the following technical solutions.

A license plate identification method based on text line identification, comprising:
S1 for acquiring an original image;
a license plate detection step S2 of detecting a license plate portion in the original image to obtain a license plate image;
a text line detection step S3 of detecting text lines on the license plate by the CPTN network to obtain a license plate text line image;
a text line identification step S4 of inputting the license plate text line image into the license plate text line identification network and finally outputting the character sequence of the license plate text line to complete the license plate identification.

Further, in step S2, the license plate portion in the original image is detected by the YOLOv3 network.

Furthermore, the YOLOv3 network reduces the feature map dimension of the original image five times to reduce the first feature map, second feature map, third feature map, fourth feature map, and fifth feature map, respectively. After that, the third feature map, the fourth feature map and the fifth feature map are respectively spliced after going through upsampling, and finally output the feature tensor to complete the detection of the license plate.

Further, step S3 is specifically:
S31 of predicting the vertical detection frame of the CPTN network using a regression model;
performing boundary optimization S32 on the predicted vertical detection frame that can prevent incorrect horizontal positioning that can occur in CPTN networks;
The vertical detection frames that reach the threshold for which the degree of overlap in the vertical direction is set are merged into one detection frame to obtain the final vertical detection frame. S33 that can prevent splitting into two parts;
The CPTN network includes S34 performing text line detection by the vertical detection frame to obtain a license plate text line image.

Further, the calculation method of the center position _tc and the height _th of the vertical frame detection frame is as follows.

where c ^b _y is the center position of the boundary frame, h ^b is the height of the boundary frame, c ^a _y is the center of the anchor frame, ^ha is the height of the anchor frame, and the boundary optimal Each vertical frame detection frame calculates one horizontal detection offset _tw , and the calculation formula of the offset _tw is as follows.

Where x ^a _side is the coordinate closest to the actual license plate horizontal boundary, c ^a _x is the x coordinate of the vertical detection frame center position, and w ^a is the width of the vertical detection frame.

Further, said license plate text identification network comprises a correction network and a text identification network, respectively realizing correction and character identification for a license plate text line image, said correction network correcting distorted and twisted text by two-dimensional transformation. and the text identification network employs a seq2seq network with built-in attention mechanisms of the encoder-decoder paradigm.

Further, the correction network includes a positioning network, which performs prediction on the control point vectors C of the original text line and, by backpropagation gradient, the control point vectors A ^r of the corrected text line. Obtained by regression analysis, the correction network performs a two-dimensional transformation on the original text line image based on the relationship between the original text line control point vector group A and the corrected text line control point vector group ^Ar. , to obtain the corrected text line image.

More preferably, the positioning network comprises six convolution filter layers, five max pool layers and two fully connected layer group layers, and the control points comprise five, each of It is the intersection of the four vertices and the diagonals.

デコーダは、コンテキストベクトル、デコーダ内部状態及び前ステップの出力を利用し、アテンションメカニズムとＧＲＵサイクルネットワークユニットによって、各キャラクタとシーケンスターミネータの確率を出力し、現在のテキストシンボルを予測するＳ４３と、を含む。 Further, the text identification network performing character identification on the license plate text line image specifically includes:
The encoder uses a convolutional neural network to extract the text feature map from the corrected license plate text line image, then divides the text feature map and inputs it to the bi-directional LSTM network to obtain the text feature sequence. S41;

The decoder uses the context vector, the decoder internal state and the output of the previous step, outputs the probability of each character and the sequence terminator and predicts the current text symbol by means of the attention mechanism and the GRU cycle network unit S43. .

Further, the calculation formula of the decoder is as follows.

Among them, Equation 4 is the current text symbol prediction probability output by the decoder at step t ₂ , Equation 5 is the decoder's internal state vector at step t ₂ , and Equation 6 is the decoder's internal at step t ₂ -1. is the state vector, Eq. 7 is the context vector of step t ₂ of the decoder, Eq. 8 is the output of step t ₂ -1 of the decoder, rnn is the GRU cycle network unit, and Eq. , where W ₀ is the fully connected network parameter and b ₀ is the fully connected network offset amount.

The context vector equation 7 of step _t2 of the decoder is obtained by the attention mechanism, and the equation is as follows.

where W _conv is the convolution network parameter, Equation 11 is the complete connectivity network parameter, Equation 12 is the text feature sequence of the encoder step _t1 , Equation 13 is the weighting parameter, and T is the input sequence 14 is the k-time weighting parameter, v, W, and V are all fully connected network parameters, BLSTM is the bi-directional LSTM network, and Equation 15 is the text feature of the encoder step t ₁ is the vector sequence obtained after partitioning the figure, and Equation 16 is the text feature sequence for encoder step t ₁ -1.

Compared with the prior art, the present invention has the following advantages.
1) The present invention proposes a whole new license plate identification method without character splitting, and introduces license plate text line detection instead of character splitting. Detecting as a whole, improving the effectiveness of subsequent identification models on training, and compared to existing license plate identification methods, the method of the present invention improves the robustness of the model and improves the accuracy of the model's identification. .
2) The present invention turns the license plate identification problem into a classical computer vision problem--image-based sequence identification, and thus the training data contains only the two-dimensional coordinates of the license plate and the necessary character sequences to identify. , saving time and cost in model training.
3) With license plate text line detection, the present invention can be applied to license plates with multiple lines of text, and can identify different license plates in multiple countries and regions, so the present invention can be used in daily urban traffic management. It can be used not only in urban areas, but also in inter-city and even international traffic management, and has become an important component of smart city construction. promotion is significant.

1 is an overall flow chart of the method of the present invention; 1 is a schematic diagram of a YOLOv3 network structure for license plate detection; FIG. 1 is a schematic diagram of a network model for license plate text line detection; FIG. 1 is a schematic diagram of a network model for license plate text line correction; FIG. 1 is a schematic diagram of a positioning network for predicting license plate text line control points; FIG. 1 is a schematic diagram of a network model for license plate text line identification; FIG.

The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. Apparently, the described embodiments are some but not all embodiments of the present invention. All other embodiments obtained by persons skilled in the art based on the embodiments in the present invention without creative efforts shall fall within the protection scope of the present invention.

Example;
As shown in Fig. 1, the present invention provides a license plate identification method based on text line identification, which can identify license plates in natural scenes, and the method is based on convolution neural network and cycle neural network, mainly It includes three steps: license plate detection (LPD), license plate text detection (LP Text Detection) and license plate text recognition (LP Text Recognition and Recognition).

Among them, in the license plate detection step, the license plate part in the original image is detected by the YOLOv3 network. For example, in FIG. Extract a sub-image of part of the license plate.

In the license plate text line detection step, the text lines on the license plate are segmented by the CPTN network, and the text on the license plate commonly seen internationally may be divided into single lines and multiple lines. For multiple lines of text, it is necessary to first split the two lines of text into multiple single lines of text to facilitate subsequent identification work. For a single line of text, intuitively from the image, this step is also essential because license plate detection in the previous step cannot necessarily accurately locate the position of the text line on the license plate. . As shown in FIG. 1, one two-line license plate text is split into two lines, top and bottom, and sent singly to the subsequent network for text identification.

The license plate text line identification step identifies the characters in the text line by the correction network based on TPS and the identification network based on the Seq2Seq model, including the attention mechanism, to complete the license plate identification, and the effects of issues such as shooting angle , the license plate may be distorted in the image, and in order to improve the effect of text identification, it is necessary to perform correction operations on the text lines before identification. In FIG. 1, finally, after identifying each of the two text lines as corrected, a complete license plate identification result is obtained.

The specific execution process of the three steps is as follows.

(1) License plate detection step When using the YOLOv3 network to detect the license plate part in the original image, the original input image needs to be divided into grids first, and if the center of the license plate is in the grid unit, then the grid is responsible for license plate detection.

As shown in Figure 2, the backbone network of the YOLOv3 network is the classical Darknet-53, which is mainly composed of 53 layers of convolutional networks, including bottom-up paths, top-down paths and side connections.

The present invention sets the resolution of the input image to 608*608 and reduces the dimension of the feature map five times according to the network structure of Darknet-53: 304, 152, 76, 38, 19. To improve the network's effectiveness in detecting targets of different sizes, the YOLOv3 network detects license plates using three different dimensional feature maps, respectively 76, 38, and 19; Feature tensors of different sizes are spliced after upsampling, and the finally output feature tensor not only has high accuracy, but also has higher significance. In order to reduce the regression complexity of boundary frames, the present invention introduces the concept of Anchor Box in Faster-RCNN or the concept of Prior Box in SSD, and uses the k-means clustering method to obtain the Prior Box.

(2) License plate text line detection step The license plate identification method of the present invention is applied to license plates of multiple countries and regions. However, considering that other countries' license plates have multiple lines of characters, it may be necessary to detect the license plate characters line by line to facilitate subsequent character identification. For a single-line text license plate, this step can improve the IoU values of the detection area and the actual area.

Unlike typical detection targets, a line of text is a single character sequence and has a coherent meaning. A Region Generation Network (RPN) employs the CTPN model to detect license plate text lines, since it is relatively difficult to locate the start and end positions of license plate text lines.

The CTPN network introduces a vertical frame to detect lines of text, a vertical frame is a set of equal-width detection frames, their heights are different, and one vertical frame has a center position and a height. It may be determined by two measures of tightness. In the CPTN network, one regression model is used to predict vertical frames. The method of calculating the center position t _c and height t _h of the vertical frame is as follows.

Among them, c ^b _y and h ^b represent the center position and height of the boundary frame respectively, and c ^a _y and ^ha can be pre-calculated based on the input image to aid the calculation. However, because the image is horizontally divided into equal-width 16-pixel areas, it cannot be guaranteed that the text line detection frame can completely cover the actual license plate area horizontally as well. , a situation may arise in which the positioning in the horizontal direction is not accurate. In order to solve this problem, a method of boundary optimization is introduced to calculate the offset amount of horizontal detection for each vertical frame, and the method of calculating the offset amount is as follows.

Among them, x ^a _side represents the coordinate closest to the actual license plate horizontal boundary, c ^a _x represents the x coordinate of the vertical frame center position, and w ^a represents the width of the vertical frame.

As shown in Figure 3, the backbone network of the CTPN model uses the VGG16 network, the input image can be of any size, and the size of the feature map output by VGG16 is the size of the input image. depends on Features are extracted through multiple rounds of convolution, and finally a feature map of W*H*N is obtained, where N is the number of feature channels, and W and H are the width and height of the feature map, respectively. Then 256 3*3 convolution cores slide on the feature map, extract 256-dimensional feature vectors for each pixel point, and combine multiple 256-dimensional vectors extracted within the same row in the picture into one sequence. and introduced into the BLSTM module, after which the 512-dimensional fully connected layer and the output layer are connected.

The CTPN network may split the same line of text into two parts, and the present invention performs subsequent processing by introducing detection frame merging so that the two detections overlap to some extent in the vertical direction. , they are merged into one detection frame, specifically, one threshold is set, and if the overlapping part in the vertical direction is higher than the threshold, both are merged.

(3) License plate text line identification step This step needs to complete identification for already detected text lines on the license plate, but before identification, the text lines need to be corrected. Due to the problem of the shooting angle of view, the characters on the picture may appear distorted, and some correction can make the distorted characters as regular as possible, thus improving the identification accuracy rate.

The present invention uses the Seq2Seq network to perform text identification, in which classical attention mechanisms are involved. For text correction, the present invention is implemented by fitting an STN network into a text identification network to correct distorted and twisted text by 2D transformation.

６つのコンボリューションフィルタレイヤ、５つの最大プールレイヤ及び二つの完全接続レイヤで構成される。一つの１０次元のベクトルを出力し、５つの２次元ベクトルに再構成し、５つの制御点座標に対応させる。制御点の座標は、正規化を経て、つまり、左上のの頂点座標が（０、０）であり、右下の頂点座標が（１、１）である。 As shown in FIG. 4, the main idea of STN networks is to model the spatial transformation operation into a neural network model. In the image to be corrected, five control points located at the intersections of the four vertices and the diagonal lines of the rectangular frame are determined. Assume that the input picture is I, the output corrected image is _Ir , the vector group consisting of the coordinates of the five control points of the original image is represented by A, and the five A vector group consisting of control points is represented by ^Ar , and the coordinates of each control point in the control point vector group A of the original text line are specifically represented by Equation (19). The essence of a two-dimensional transform is to approximate a single interpolation function f and satisfy A ^r =f(A). The TPS (Thin-Plate-Spline) model has proven to be very effective in the process of distorted text correction, and the correction task of license plate pictures can be reduced to the task of predicting five control point positions. A positioning network is often used to predict the control points on image I. The positioning network recursively analyzes the control points of the output image by backpropagation gradients and automatically labels the five control points of the output image. ,

It consists of 6 convolution filter layers, 5 max pool layers and 2 fully connected layers. A 10-dimensional vector is output and reconstructed into 5 2-dimensional vectors corresponding to 5 control point coordinates. The coordinates of the control points are normalized, that is, the upper left vertex coordinates are (0,0) and the lower right vertex coordinates are (1,1).

The coordinates of the p point are expressed as [x _p , y _p ], and the corresponding corrected coordinates of the point p′ can be calculated according to the following method.

Among them, Φ(x)=x ² log(x) is the kernel function applied to the Euclidean distance between the point p and the kth control point.

After solving the parameters of the TPS by solving the linear system, the finally obtained formula of the output corrected image is:

Among them, V is the down sampler, I is the input picture, _Ir is the corrected picture, and the pixel points of the original map and the corrected map are down-sampled to finally obtain the corrected image.

As shown in FIG. 6, a license plate text identification network is used to output the character sequence of a license plate text line, the network being one seq2seq frame and built-in depending on the encoder-decoder paradigm. have an attention mechanism that

First, the encoder uses a convolutional neural network to extract features from the corrected license plate text line image, and the image size is 32*100. The convolution network for extracting features is an improvement based on ResNet-50, the moving step size of the convolution core for the last three downsampling layers is (2, 1), so that This is to ensure that the feature map on each feature channel is one vector, so the final feature map size is 1*25*512 (h*w*n). Then divide the feature map to _obtain a vector group consisting of a vector sequence, expressed as X=[x ₁ , x ₂ , . Each of the feature width w in the size and each vector in the vector group has 512 dimensions, that is, the number of feature channels in the feature map size n.

Bidirectional LSTM (BLSTM) networks can obtain long-range dependencies of feature sequences in two directions, so applying BLSTM to the feature sequences obtained in the previous step yields a richer Get feature sequences with contextual relationships. The new _feature sequence output by BLSTM is represented as H=[h ₁ , h ₂ , .

In any one step of BLSTM, the decoder finally outputs one probability diagram based on the context vector C, the internal state s of the decoder, the output y of the previous step, and this probability diagram is for each character and the probability of a sequence terminator (EOS). The context vector C is the aggregated information of H, denoted as C=[c ₁ ,c ₂ ,...,c _T ],C=q(H), where q is the attention mechanism, Eq. may be expressed as
Among them, Equation 25 is obtained by calculating hidden state Equation 26 at step t ₁ of the encoder and hidden state Equation 27 at step t ₂ -1 of the decoder, and W, V, and b are all It is a trainable weighting.

The output of the encoder also enters the decoder as input, and the decoder computes one output vector z and one new state vector s.

Wherein y is the one-hot form, rnn represents the GRU cycle network unit, and the output z is used to predict the current text symbol.

To operate the concept of maximum likelihood estimation and maximize the conditional probability of the output sequence, the target function that needs to be optimized is:

If the output exceeds the maximum length, or if an EOS symbol is obtained, indicating that the output sequence has ended and finally obtained the identification result of the license plate text line in the image, this embodiment uses the Beam The Research algorithm is used, in which the Beam size parameter is set to 5.

We have verified the high robustness and high performance of our proposed method by training and testing with AOLP and UFPR-ALPR datasets.

In the license plate detection step, if the IoU value is greater than 0.5, the license plate is considered successfully detected, and the IoU formula is as follows:

Among them, R _det is the detection frame and R _gt is the mark frame.

In the license plate text line detection task, IoU is used to assess the accuracy of detection. We also evaluated the performance using F ₁ -score in the license plate text identification task and some license plate text detection tasks, and the formula is as follows.

This index simultaneously considered the accuracy rate precision and the recall rate recall.

In this example, two data sets are used to validate each. After each step is finished, each one checks its effect to ensure that each step has high performance and high robustness. Unlike the UFPR-ALPR dataset, the AOLP dataset itself does not split the training and test sets, so of the three subsets, two are used as the training set and one as the test set. For example, a license plate recognition model may be trained using the LE and AC subsets and tested using the RP subset. See Tables 1 through 6 for detailed test results on the three main steps for each of the two datasets.

What have been described above are only specific embodiments of the present invention, and the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily come up with various equivalent modifications or replacements within the technical scope posted in the present invention, and any of these modifications or replacements should be included in the protection scope of the present invention. is. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (7)

A license plate identification method based on text line identification, comprising:
S1 for acquiring an original image;
a license plate detection step S2 of detecting a license plate portion in the original image to obtain a license plate image;
a text line detection step S3 of detecting text lines on the license plate by a text detection network to obtain a license plate text line image;
a text line identification step S4 of inputting the license plate text line image into the license plate text line identification network and finally outputting the character sequence of the license plate text line to complete the license plate identification ;
The license plate text line identification network includes a correction network and a text identification network, respectively, to achieve correction and character identification for the license plate text line image, the correction network corrects text for distortion and twist by two-dimensional transformation. , the text identification network employs a seq2seq network with built-in attention mechanisms of the encoder-decoder paradigm;
Specifically, the text identification network performing character identification on the license plate text line image includes:
The encoder uses a convolutional neural network to extract the text feature map from the corrected license plate text line image, then divides the text feature map and feeds it into a bi-directional LSTM network, after which the text feature sequence h _t S41 to obtain
inputting S42 the text feature sequence ht _to the decoder;
the decoder uses the context vector, the decoder internal state and the output of the previous step, outputs the probability of each character and the sequence terminator and predicts the current text symbol by means of the attention mechanism and the GRU cycle network unit S43; including
The calculation formula of the decoder is as shown in Formula 1,

Among them, Equation 2 is the current text symbol prediction probability output by the decoder at step t ₂ , Equation 3 is the decoder's internal state vector at step t ₂ , and Equation 4 is the decoder's internal at step t ₂ -1. Equation 5 is the context vector of step t ₂ of the decoder, Equation 6 is the output of step t ₂ -1 of the decoder, rnn is the GRU cycle network unit, and Equation 7 is the character classification probability diagram. , where W ₀ is the fully connected network parameter, b ₀ is the fully connected network offset amount,

The context vector Equation 5 of step t2 of the decoder _is obtained by the attention mechanism, and the equation is as Equation 8,

where w _conv is the convolution network parameter, Equation 9 is the complete connectivity network parameter, Equation 10 is the text feature sequence of the encoder step t1 _, Equation 11 is the weighting parameter, and T is the 12 is the k-time weighting parameter, v, W, and V are all fully connected network parameters, BLSTM is the bi-directional LSTM network, and Equation 13 is the text feature of encoder step t ₁ is the vector sequence obtained after partitioning the figure, and Equation 14 is the text feature sequence of encoder step t ₁ -1,

A license plate identification method based on text line identification, characterized by: The license plate identification method based on text line identification according to claim 1, characterized in that in said step S2, the license plate portion in the original image is detected by the YOLOv3 network. The YOLOv3 network reduces the dimension of the feature map of the original image five times to obtain the first, second, third, fourth, and fifth feature maps, respectively. After that, the third feature map, the fourth feature map and the fifth feature map are spliced after upsampling, and finally the feature tensor is output to complete the detection of the license plate. 3. The license plate identification method based on text line identification according to claim 2, wherein: Specifically, the step S3 is
S31 using a regression model to predict the vertical detection frame of the CTPN network;
S32 performing boundary optimization on the vertical detection frame obtained by prediction;
a step S33 of merging the vertical detection frames reaching a threshold for which the degree of superimposition in the vertical direction is set into one detection frame to obtain a final vertical detection frame;
The method for license plate identification based on text line identification according to claim 1, characterized in that the CTPN network performs text line detection by vertical detection frames, and includes S34 to obtain a license plate text line image. The method of calculating the center position t _c and height t _h of the vertical detection frame is as shown in Equation 15 and Equation 16 , where c ^b _y is the center position of the boundary frame and h ^b is the height of the boundary frame. is the height of the anchor frame, c ^a _y is the center of the anchor frame, and ^ha is the height of the anchor frame;

In the boundary optimization, each vertical detection frame calculates a horizontal detection offset _tw , and the offset _tw is calculated by Equation 17 ,

wherein x ^a _side is the coordinate closest to the actual license plate horizontal boundary, c ^a _x is the x coordinate of the vertical detection frame center position, and w ^a is the width of the vertical detection frame frame. A license plate identification method based on text line identification according to claim 4. The correction network comprises a positioning network, the positioning network performs prediction on the control point vectors A of the original text line, and recursively analyzes the control point vectors A ^r of the corrected text line by backpropagation gradient. , the correction network performs a two-dimensional transformation on the original text line image based on the relationship between the original text line control point vector group C and the corrected text line control point vector group ^Ar , and corrects The license plate identification method based on text line identification according to claim 1 , characterized in that a later text line image is obtained. The positioning network includes six convolution filter layers, five max pool layers and two fully connected layer group layers, and the control points include five, each corresponding to the four vertices of license plate text lines. The license plate identification method based on text line identification according to claim 6 , characterized in that it is the intersection of diagonal lines.

Images