CN112329881B - License plate recognition model training method, license plate recognition method and device - Google Patents

License plate recognition model training method, license plate recognition method and device Download PDF

Info

Publication number
CN112329881B
CN112329881B CN202011317882.6A CN202011317882A CN112329881B CN 112329881 B CN112329881 B CN 112329881B CN 202011317882 A CN202011317882 A CN 202011317882A CN 112329881 B CN112329881 B CN 112329881B
Authority
CN
China
Prior art keywords
license plate
vehicle
double
license
recognition
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011317882.6A
Other languages
Chinese (zh)
Other versions
CN112329881A (en
Inventor
邓练兵
李大铭
李皓
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhuhai Dahengqin Technology Development Co Ltd
Original Assignee
Zhuhai Dahengqin Technology Development Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhuhai Dahengqin Technology Development Co Ltd filed Critical Zhuhai Dahengqin Technology Development Co Ltd
Priority to CN202011317882.6A priority Critical patent/CN112329881B/en
Publication of CN112329881A publication Critical patent/CN112329881A/en
Application granted granted Critical
Publication of CN112329881B publication Critical patent/CN112329881B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/24—Aligning, centring, orientation detection or correction of the image
    • G06V10/243—Aligning, centring, orientation detection or correction of the image by compensating for image skew or non-uniform image deformations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/14—Image acquisition
    • G06V30/148—Segmentation of character regions
    • G06V30/153—Segmentation of character regions using recognition of characters or words
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
    • G06V20/625—License plates
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Multimedia (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Biomedical Technology (AREA)
  • Evolutionary Biology (AREA)
  • Health & Medical Sciences (AREA)
  • Character Discrimination (AREA)
  • Image Analysis (AREA)
  • Traffic Control Systems (AREA)

Abstract

The invention discloses a license plate recognition model training method, a license plate recognition method and a license plate recognition device, wherein the model training method comprises the following steps: constructing a double-license plate vehicle data set and a license plate data set; training yolov 3-tiny network model based on double-license vehicle data set; testing a training model of yolov 3-tiny and reserving the weight of the yolov 3-tiny network model with the highest precision; training an LPS/CR-NET network model based on the license plate data set, carrying out network test on the training model of the LPS/CR-NET, and preliminarily determining weights of a plurality of network models with better precision; the two-license plate recognition accuracy under each initially determined LPS/CR-NET network model weight is calculated based on yolov 3-tiny double-license plate detection results and a character sorting method, and the LPS/CR-NET network model weight with the highest accuracy is reserved. A fast and efficient license plate positioning method is provided for double-license plate recognition, and the precision of double-license plate recognition is effectively improved.

Description

License plate recognition model training method, license plate recognition method and device
Technical Field
The invention relates to the technical field of license plate recognition, in particular to a license plate recognition model training method based on deep learning, and a license plate recognition method and device.
Background
Vehicles passing through an electronic purse net area have the problems of diversified vehicle license plate formats (different formats of common cars, embassy cars, police cars, military cars and Hongkou and Australia license plates) in China and high difficulty in Chinese character recognition, and also need to consider different formats of vehicle license plates of other countries and different vehicle types and different license plate hanging positions, so that new requirements are provided for license plate recognition technology. The license plate recognition technology requires that the license plate of the moving automobile can be extracted and recognized from a complex background, and the information of the license plate number, the color and the like of the automobile can be recognized through the technologies of license plate extraction, license plate recognition and the like.
Due to the opening of the Guangdong, hong Kong and Macao bridge, partial vehicles in hong Kong, Macao and Guangdong regions can be driven across areas, and the condition that the vehicles hang double license plates is gradually common. However, the existing algorithms for recognizing the license plates only consider that one vehicle has only one license plate, neglects the condition that the vehicle hangs double license plates, and neglects the detection and recognition algorithm of the vehicle with double license plates.
Therefore, how to construct a double-license plate recognition training model and realize double-license plate detection and recognition of vehicle pictures is a technical problem which needs to be solved urgently at present.
Disclosure of Invention
Therefore, the technical problem to be solved by the invention is to overcome the defect that the license plate detection and recognition model in the prior art cannot perform license plate recognition and detection on a vehicle with two suspended license plates, and the accuracy is low, so that the license plate recognition model training method is provided.
In order to solve the technical problems, the technical scheme of the invention is as follows:
in a first aspect, a license plate recognition model training method includes:
constructing a double-license plate vehicle data set B and a license plate data set C according to the collected vehicle picture samples with double license plates;
training a yolov 3-tiny network model structure based on the double-license vehicle data set B, and positioning a license plate area in a vehicle picture;
performing network test on each training model of yolov 3-tiny according to a double-license vehicle test set containing a plurality of test pictures with double license plates, and reserving a yolov 3-tiny network model weight file with the highest precision;
training a network model structure of LPS/CR-NET character segmentation and recognition based on the license plate data set C, positioning the position of each character in a license plate picture and recognizing each character;
performing network test on each training model of the LPS/CR-NET according to a license plate test set containing a plurality of license plate test pictures, and preliminarily determining a plurality of network model weight files with better precision;
the double-license plate identification accuracy of the vehicle pictures under each initially determined LPS/CR-NET network model weight file is calculated based on the yolov 3-tiny double-license plate detection result and the character sorting method, and the LPS/CR-NET network model weight file with the highest accuracy is reserved.
Further, the construction of a double-license plate vehicle data set B and a license plate data set C according to the collected vehicle picture samples with double license plates comprises the following steps:
collecting a vehicle picture sample with double license plates;
respectively carrying out position calibration on the vehicle, the license plate and the characters in the collected vehicle picture sample by adopting a rectangular frame to form a vehicle region rectangular frame, a license plate region rectangular frame and a character region rectangular frame;
rotating the vehicle picture sample subjected to position calibration to obtain a data set A subjected to data amplification;
according to the vehicle area rectangular frame, cutting a vehicle picture in each vehicle picture sample of the data set A to construct a double-license vehicle data set B; each labeling file for cutting out the vehicle picture in the double-license plate vehicle data set B comprises a license plate region rectangular frame and a character region rectangular frame;
calculating the license plate number corresponding to the license plate by using a character sorting algorithm according to the characters in the rectangular frame of the license plate area;
cutting out a license plate picture from each constructed vehicle picture of the double-license plate vehicle data set B according to the license plate region rectangular frame to construct a license plate data set C; and the labeling file for cutting out the license plate picture from the license plate data set C comprises a license plate number corresponding to the license plate, a character area rectangular frame and a character type.
Further, in the step of respectively calibrating the positions of the vehicle, the license plate and the characters in the collected vehicle picture sample by using the rectangular frame, a license plate region rectangular frame is formed in a clockwise or anticlockwise direction by taking an angular point of the upper left corner of the license plate in the vehicle picture sample with the double license plates as a starting point.
Further, after the step of constructing a double-license plate vehicle data set B and a license plate data set C according to the collected vehicle picture samples with double license plates, the method further comprises the following steps:
and converting the double-license plate vehicle data set B and the license plate data set C into a standard YOLO series data format.
Further, the method for detecting the double-license plate and sorting the characters based on yolov 3-tiny calculates the double-license plate recognition accuracy of the vehicle picture under each initially determined LPS/CR-NET network model weight file, and keeps the LPS/CR-NET network model weight file with the highest accuracy, and comprises the following steps:
detecting a license plate area through yolov 3-tiny network;
inputting the detected license plate area into an LPS/CR-NET for character segmentation and recognition;
sorting characters of the character segmentation recognition result, and outputting a license plate number;
if the position of the license plate area is detected to be larger than 0.5 and the number of the license plate is consistent with the true value IoU, the license plate is predicted and identified successfully; if all license plates in the vehicle picture are successfully identified, the vehicle picture is successfully identified; and calculating the identification accuracy of the vehicle picture, and determining the LPS/CR-NET network model weight file with the highest identification accuracy of the vehicle picture.
Further, the character sorting of the character segmentation recognition result and the output of the license plate number include:
calculating the average width of the bounding box of the n character regions
Figure BDA0002790172580000031
And average height
Figure BDA0002790172580000032
Water having coordinates of center position of bounding box by character areaArranging x coordinates in the horizontal direction in an ascending order; let the i-th character region rectangular frame be Ki，i＝＝1，...，n；xiIs KiThe horizontal direction x coordinate of the center point of (a);
defaulting all characters on a second line;
calculating any two adjacent character area rectangular frames KiAnd KjHorizontal distance Δ w ofi,j＝xj-xiAnd a vertical distance Δ hi,j＝yj-yi(i 1, 2., n-1, j +1), constructing a discriminant;
Figure BDA0002790172580000041
Figure BDA0002790172580000042
wherein, P>0.8 and Δ hi,j<0, then K isjIn the first row; if P>0.8 and Δ hi,j>0, then K isiIn the first row;
calculating Ki－1，KiHorizontal distance Δ w ofi-1,i＝xi-xi-1And a vertical distance Δ hi-1,i＝yi-yi-1(i ═ 1, 2., n-1, j ═ i +1), calculating P; if P is less than 0.7, then Ki－1At KiThe row position of the position;
and serially connecting the characters of the first line and the second line to output the license plate number.
In a second aspect, a license plate recognition method includes:
obtaining a vehicle picture;
inputting the vehicle picture into a model, and performing double-license plate recognition on the vehicle picture to obtain a double-license plate recognition result; the model is obtained by training through the license plate recognition model training method.
In a third aspect, a license plate recognition device includes:
the acquisition module is used for acquiring a vehicle picture;
and the recognition module is used for inputting the vehicle picture into a model and performing double-license plate recognition on the vehicle picture, wherein the model is obtained by training through the license plate recognition model training method.
In a fourth aspect, an electronic device includes: the license plate recognition system comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein the processor realizes the license plate recognition model training method or the license plate recognition method when executing the computer program.
In a fifth aspect, a computer-readable storage medium storing computer instructions for causing a computer to perform the license plate recognition model training method as described above or the license plate recognition method of claim 7.
The technical scheme of the invention has the following advantages:
1. according to the license plate recognition model training method provided by the invention, a yolov 3-tiny network model structure is adopted to carry out license plate detection on collected vehicle picture samples with double license plates, and the yolov 3-tiny network model structure is a One-Stage target detection network, so that the license plate detection efficiency can be greatly improved while the requirement on accuracy is met. Meanwhile, when the characters of the license plate are recognized, a network model structure of LPS/CR-NET character segmentation and recognition is adopted, and a plurality of license plate characters are recognized in a mode of constructing a plurality of convolution connection layers, so that a license plate recognition process is simplified, license plate recognition time is reduced, and license plate recognition efficiency is improved. The license plate recognition method simplifies the license plate recognition process into two processes of license plate detection and license plate character recognition, and carries out recognition through two models with corresponding functions, so that the license plate recognition method not only can meet the requirement of accuracy, but also can carry out license plate detection and recognition at a higher speed, and realizes the accurate recognition of the license plates in the double-license plate vehicle.
2. The license plate recognition model training method provided by the invention combines yolov 3-tiny target detection network algorithm and LPS/CR-NET network together to realize the detection and recognition of double license plates based on the existing algorithm; in addition, aiming at the disordered character segmentation and classification result output by a target detection network such as LPS/CR-NET, a character sorting algorithm is designed to output license plate numbers instead of each isolated character while detecting license plate areas, and meanwhile, the characters with abnormal positions are effectively eliminated and the license plate recognition effect is improved by the character sorting method.
3. In the training stage, training a yolov 3-tiny target detection network based on a double-license vehicle data set B to position a license plate region in a vehicle picture; and then, training a target detection network for LPS/CR-NET character segmentation and recognition based on the license plate data set C, and realizing the positioning of each character block position and the recognition of each character in the license plate image block. In the testing stage, firstly, a double-license vehicle testing set is utilized to perform precision calculation on each training model of yolov 3-tiny, and then a yolov 3-tiny model weight file with the best precision is reserved; secondly, the accuracy of each training model of the LPS/CR-NET is evaluated by utilizing a license plate test set, and a plurality of weight models with better accuracy are preliminarily determined; and then calculating the double-license plate recognition accuracy and precision of the vehicle pictures under each preliminarily determined LPS/CR-NET model weight based on the yolov 3-tiny double-license plate detection result and the character sorting algorithm, and keeping the LPS/CR-NET model weight with the highest accuracy. The algorithm adopts light yolov 3-tiny to detect the double license plates of the vehicle, provides a fast and efficient license plate positioning method for the license plate identification of the electronic purse net monitoring video, and simultaneously, the constructed character sorting method comprehensively considers the problem of the sorting of the characters of the license plate with double rows of characters and the problem of abnormal character positions, thereby effectively improving the identification precision of the double license plates.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a flowchart illustrating an implementation of a license plate recognition model training method according to an embodiment of the present disclosure;
FIG. 2 is a Yolov 3-tiny license plate target detection network training model diagram in the embodiment of the present invention;
FIG. 3 is a diagram of a LPS/CR-NET character segmentation recognition network model according to an embodiment of the present invention;
FIG. 4 is a diagram illustrating the operation result of a Modified Fast-Yolov 2+ CR-Net algorithm in the prior art;
FIG. 5 is a graph of the results of the prior art operation based on Fast-Yolo + LPS/CR-NET algorithm;
fig. 6 is a diagram of the operation result of the operation algorithm according to the embodiment of the present invention.
Detailed Description
The technical solutions of the present invention will be described clearly and completely with reference to the accompanying drawings, and it should be understood that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In a first embodiment, a license plate recognition model training method as shown in fig. 1 to 6 includes the following steps:
step S10, constructing a double-license-plate vehicle data set B and a license-plate data set C according to the collected vehicle picture samples with double license plates;
step S20, training a yolov 3-tiny network model structure based on the double-license vehicle data set B, and positioning a license plate discharging area in a vehicle picture;
step S30, performing network test on each training model of yolov 3-tiny according to a double-license vehicle test set containing a plurality of test pictures with double licenses, and reserving a yolov 3-tiny network model weight file with the highest precision;
step S40, training a network model structure of LPS/CR-NET character segmentation and recognition based on the license plate data set C, positioning the position of each character in a license plate picture and recognizing each character;
step S50, performing network test on each training model of LPS/CR-NET according to a license plate test set containing a plurality of license plate test pictures, and preliminarily determining a plurality of network model weight files with better precision;
and step S60, calculating the double-license recognition accuracy of the vehicle pictures under each initially determined LPS/CR-NET network model weight file based on the yolov 3-tiny double-license detection result and the character sorting method, and keeping the LPS/CR-NET network model weight file with the highest accuracy.
The license plate recognition model training method mainly comprises two stages: 1) training phase, 2) testing phase. The yolov 3-tiny target detection network algorithm and the LPS/CR-NET network are combined together based on the existing algorithm to realize the detection and identification of double license plates; in addition, aiming at the disordered character segmentation and classification result output by a target detection network such as LPS/CR-NET, a character sorting algorithm is designed to output license plate numbers instead of each isolated character while detecting license plate areas, and meanwhile, the characters with abnormal positions are effectively eliminated and the license plate recognition effect is improved by the character sorting method.
In the training stage, training a yolov 3-tiny target detection network based on a double-license-plate vehicle data set B to position a license plate region in a vehicle picture; and then, training a target detection network for LPS/CR-NET character segmentation and recognition based on the license plate data set C, and realizing the positioning of each character block position and the recognition of each character in the license plate image block.
In the testing stage, firstly, a double-license vehicle testing set is utilized to perform precision calculation on each training model of yolov 3-tiny, and then a yolov 3-tiny model weight file with the best precision is reserved; secondly, the accuracy of each training model of the LPS/CR-NET is evaluated by utilizing a license plate test set, and a plurality of weight models with better accuracy are preliminarily determined; and then calculating the double-license plate recognition accuracy and precision of the vehicle pictures under each preliminarily determined LPS/CR-NET model weight based on the yolov 3-tiny double-license plate detection result and the character sorting algorithm, and keeping the LPS/CR-NET model weight with the highest accuracy. The proposed algorithm has shown a significant performance improvement compared to the prior art.
The algorithm adopts light yolov 3-tiny to detect the double license plates of the vehicle, provides a fast and efficient license plate positioning method for the license plate identification of the electronic purse net monitoring video, and simultaneously, the constructed character sorting method comprehensively considers the problem of the sorting of the characters of the license plate with double rows of characters and the problem of abnormal character positions, thereby effectively improving the identification precision of the double license plates.
In this embodiment, a double-license vehicle data set B and a license plate data set C are constructed, where the double-license vehicle data set B is used for training a license plate target detection network, the license plate data set C is used for training a character segmentation recognition network, and the step S10 specifically includes the following steps:
and S101, collecting a vehicle picture sample with double license plates. Specifically, a vehicle picture sample with double license plates, hong Kong license plates and Macau license plates is collected (considering that there are few double license plates in China, in order to effectively identify hong Kong and Macau license plates in double license plates, images of hong Kong and Macau single license plates are collected at the same time).
Step S102, respectively carrying out position calibration on the vehicle, the license plate and the character in the collected vehicle picture sample by adopting a rectangular frame to form a vehicle region rectangular frame, a license plate region rectangular frame and a character region rectangular frame; in the process of calibrating the license plate region rectangular frame, the license plate region rectangular frame is formed in a clockwise or anticlockwise direction by taking an angular point of the upper left corner of the license plate in a vehicle picture sample with double license plates as an initial point.
Step S103, rotating the vehicle picture sample subjected to position calibration to obtain a data set A of data amplification;
step S104, cutting out vehicle pictures from each vehicle picture sample of the data set A according to the vehicle area rectangular frame to construct a double-license vehicle data set B; each labeling file for cutting out the vehicle picture in the double-license plate vehicle data set B comprises a license plate region rectangular frame and a character region rectangular frame;
and S105, calculating the license plate number corresponding to the license plate by using a character sorting algorithm according to the characters in the rectangular frame of the license plate area, and marking the license plate number in the attribute of the vehicle target.
Before calculation by character sorting algorithm, the marked document Xml is converted into Xxt document whose content is '0, x' because there is only one kind (license plate)1，y1，x2，y2V n "in which x, y have been normalized, e.g.
Figure BDA0002790172580000091
(w is picture width, x)1Coordinates of a rectangular frame of the license plate area in the picture).
The core idea of the character sorting algorithm is that for adjacent character blocks in the horizontal direction, if the distance between the two character blocks in the vertical direction is greater than a certain threshold, the two characters are not in the same row; otherwise, it is in the same row. Based on the result of the license plate character segmentation and recognition, the specific steps of the character sorting algorithm are as follows:
a, calculating the average width w and the average height h of a surrounding frame of the n character areas;
b, arranging the x coordinates in the horizontal direction of the coordinates of the central position of the enclosing frame of the character area in an ascending order; let the i-th character region rectangular frame be Ki，i＝＝1，...，n；xiIs KiThe horizontal direction x coordinate of the center point of (a);
c, defaulting all characters on a second line;
d, calculating any two adjacent character area rectangular frames KiAnd KjHorizontal distance Δ w ofi,j＝xj-xiAnd a vertical distance Δ hi,j＝yj-yi(i 1, 2., n-1, j +1), constructing a discriminant;
Figure BDA0002790172580000092
Figure BDA0002790172580000093
wherein，P>0.8 and Δ hi,j<0, then K isjIn the first row; if P>0.8 and Δ hi,j>0, then K isiIn the first row;
e, calculating Ki－1，KiHorizontal distance Δ w ofi-1,i＝xi-xi-1And a vertical distance Δ hi-1,i＝yi-yi-1(i ═ 1, 2., n-1, j ═ i +1), calculating P; if P is less than 0.7, then Ki－1At KiThe row position of the position;
and f, serially connecting the characters of the first line and the second line and outputting the license plate number.
Step S106, cutting out license plate pictures from each constructed vehicle picture of the double-license plate vehicle data set B according to the license plate area rectangular frame, and constructing a license plate data set C; and the labeling file for cutting out the license plate picture from the license plate data set C comprises a license plate number corresponding to the license plate, a character area rectangular frame and a character type.
In steps S20 to S60, the operation flow of training yolov 3-tiny network model structure based on the two-license-plate vehicle data set B and training LPS/CR-NET character segmentation and recognition network model structure based on the license-plate data set C, and finally determining the LPS/CR-NET network model weight file with the highest accuracy is as follows:
A) converting the data in the double-license plate vehicle data set B and the license plate data set C into data formats commonly used in a standard YOLO series;
B) respectively adopting a K-means clustering algorithm to obtain respective anchor anchoring values;
C) yolov 3-tiny license plate target detection network training, wherein the network structure is shown in FIG. 2;
D) and performing Yolov 3-tiny network test, calculating the correct detection rate of the license plate of the vehicle picture, and determining the weight of the model, wherein the correct detection rate of the license plate LPobj_accuracyExpressed as:
Figure BDA0002790172580000101
ncorrectnumber of vehicle pictures indicating that the vehicle has successfully detected all license plates, NpictureRepresenting the total number of vehicle pictures.
E) LPS/CR-NET character segmentation recognition network training, wherein the network structure is shown in figure 3;
F) LPS/CR-NET network test, calculating the complete segmentation and recognition accuracy of license plate characters, primarily screening model weight, and the complete segmentation and recognition accuracy LP of license plate charactersCH_accutacyExpressed as:
Figure BDA0002790172580000102
wherein n iscorrectNumber of license plates representing all characters in license plate picture successfully recognized, NpictureRepresenting the total number of license plate pictures.
G) And determining the LPS/CR-NET network model weight by combining the Yolov 3-tiny network. Firstly, detecting a license plate area through a Yolov 3-tiny network; then inputting the detected license plate area into an LPS/CR-NET for character segmentation and recognition; sorting characters of the character segmentation recognition result, and outputting a license plate number; if the number plate prediction frame is larger than 0.5 and the number plate number is consistent with the true value IoU, the number plate prediction identification is successful; thirdly, if all license plates in the vehicle are successfully identified, the vehicle picture is successfully identified; and calculating the identification accuracy of the vehicle picture. And determining the LPS/CR-NET network model weight with the highest accuracy.
In the model training process, Yolov 3-tiny adopts the vehicle picture data set made by the project to carry out double-license target detection, the network training stage iterates 100000 times totally, the picture input size is 416 ANG 416, the batch size of batch is 8, and the weight loss parameter is 5e－4Momentum is 0.9, learning rate is initialized to 0.001, decay is 0.0001 and 0.00001 after 33000 and 66000 iterations, and anchors parameters of two yolos output layers are shown as follows;
#16
【yolo】
mask＝5，6，7，8，9
anchors＝65，27，54，44，80，38，79，53，105，46，113，36，132，46，104，62，131，58，134，80；
#23
【yolo】
mask＝0，1，2，3，4
anchors＝65，27，54，44，80，38，79，53，105，46，113，36，132，46，104，62，131，58，134，80。
the LPS/CR-NET adopts a license plate picture data set manufactured by the project to carry out character segmentation and recognition, the network training stage is iterated for 60000 times totally, the picture input size is 240 ANG 80, the batch size is 32, the weight loss parameter is 5 e-4, the momentum is 0.9, the learning rate is initialized to 0.001, the learning rate is respectively 0.01, 0.001 and 0.0001 after 100, 48000 and 54000 iterations, and the anchors parameter of a region output layer is shown as follows;
【region】
anchors＝2.9195，5.2152，3.6705，4.3885，5.7145，3.5746，4.3904，4.8669，3.7476，5.7960。
during the test, the test is carried out on Python version 3.6, and Python version 3.6 is provided with the following modules, pitorch, opencv, numpy, os, time, pickle, xml, imutilis and the like. Windows operating system. The Darknet configuration in the Windows environment is configured. Since Yolov 3-tiny and LPS/CR-NET are both yolo series target detection network algorithms, the training of the two networks is realized by adopting a dark NET configuration environment.
In order to perform quantitative precision evaluation on the detection result, precision, call and accuracy of the license plate are mainly used as evaluation indexes for evaluation. The calculation formulas of the recall rate recall and the accuracy rate precision are as follows:
Figure BDA0002790172580000121
Figure BDA0002790172580000122
wherein, TPiNumber of license plates for representing correct recognition of ith pictureFPi shows the number of license plate data which the ith picture shows to be recognized by mistake, TNiAnd (4) indicating the number of the license plate data missed in detection of the ith picture.
Accuracy is defined as:
Figure BDA0002790172580000123
wherein n iscorrectNumber of license plates representing all characters in license plate picture successfully recognized, NpictureRepresenting the total number of license plate pictures.
Figure BDA0002790172580000124
The second embodiment provides a license plate identification method, which comprises the following steps:
1) obtaining a vehicle picture;
2) inputting the vehicle picture into a model, and performing double-license plate recognition on the vehicle picture to obtain a double-license plate recognition result; the model is obtained by training through the license plate recognition model training method in the first embodiment;
3) adopting a character sorting algorithm to identify the double license plates obtained in the step 2)
4) Two license plate numbers are output.
In a third embodiment, a double-license plate recognition apparatus includes:
the acquisition module is used for acquiring a vehicle picture;
and the recognition module is used for inputting the vehicle picture into a model and performing double-license plate recognition on the vehicle picture, wherein the model is obtained by training through the license plate recognition model training method in the embodiment I.
In a fourth embodiment, an electronic device includes: the license plate recognition system comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein the processor executes the computer program to realize the license plate recognition model training method in the first embodiment or the license plate recognition method in the second embodiment.
The processor may be a Central Processing Unit (CPU). The Processor may also be other general purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, or a combination thereof.
The memory, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as program instructions/modules corresponding to the license plate recognition model training method or the license plate recognition method in the embodiments of the present invention. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, the license plate recognition model training method or the license plate recognition method in the above method embodiments is realized.
The memory may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created by the processor, and the like. Further, the memory may include high speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory optionally includes memory located remotely from the processor, and such remote memory may be coupled to the processor via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD), a Solid State Drive (SSD), or the like; the storage medium may also comprise a combination of memories of the kind described above.
In summary, the license plate recognition model training method adopts the yolov 3-tiny network model structure to carry out license plate detection on collected vehicle picture samples with double license plates, and the yolov 3-tiny network model structure is a One-Stage target detection network, so that the license plate detection efficiency can be greatly improved while the requirement on accuracy is met. Meanwhile, when the characters of the license plate are recognized, a network model structure of LPS/CR-NET character segmentation and recognition is adopted, and a plurality of license plate characters are recognized in a mode of constructing a plurality of convolution connection layers, so that a license plate recognition process is simplified, license plate recognition time is reduced, and license plate recognition efficiency is improved. The license plate recognition method simplifies the license plate recognition process into two processes of license plate detection and license plate character recognition, and carries out recognition through two models with corresponding functions, so that the license plate recognition method not only can meet the requirement of accuracy, but also can carry out license plate detection and recognition at a higher speed, and realizes the accurate recognition of the license plates in the double-license plate vehicle.
It should be understood that the above examples are only for clarity of illustration and are not intended to limit the embodiments. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. And obvious variations or modifications therefrom are within the scope of the invention.

Claims (8)

1. A license plate recognition model training method is characterized by comprising the following steps:
constructing a double-license plate vehicle data set B and a license plate data set C according to the collected vehicle picture samples with double license plates;
training a yolov 3-tiny network model structure based on the double-license vehicle data set B, and positioning a license plate area in a vehicle picture;
performing network test on each training model of yolov 3-tiny according to a double-license vehicle test set containing a plurality of test pictures with double license plates, and reserving a yolov 3-tiny network model weight file with the highest precision;
training a network model structure of LPS/CR-NET character segmentation and recognition based on the license plate data set C, positioning the position of each character in a license plate picture and recognizing each character;
performing network test on each training model of the LPS/CR-NET according to a license plate test set containing a plurality of license plate test pictures, and preliminarily determining a plurality of network model weight files with highest precision;
calculating the double-license plate recognition accuracy of the vehicle pictures under each initially determined LPS/CR-NET network model weight file based on the yolov 3-tiny double-license plate detection result and the character sorting method, and keeping the LPS/CR-NET network model weight file with the highest accuracy;
the double-license plate detection result and character sorting method based on yolov 3-tiny calculates the double-license plate recognition accuracy of the vehicle picture under each initially determined LPS/CR-NET network model weight file, and reserves the LPS/CR-NET network model weight file with the highest accuracy, and comprises the following steps:
detecting a license plate area through yolov 3-tiny network;
inputting the detected license plate area into an LPS/CR-NET for character segmentation and recognition;
sorting characters of the character segmentation recognition result, and outputting a license plate number;
if the position of the license plate area is detected to be larger than 0.5 and the number of the license plate is consistent with the true value IoU, the license plate is predicted and identified successfully; if all license plates in the vehicle picture are successfully identified, the vehicle picture is successfully identified; calculating the identification accuracy of the vehicle picture, and determining an LPS/CR-NET network model weight file with the highest identification accuracy of the vehicle picture;
the operation flow of training yolov 3-tiny network model structure based on the double-license-plate vehicle data set B and training LPS/CR-NET character segmentation and recognition network model structure based on the license plate data set C and finally determining the LPS/CR-NET network model weight file with the highest accuracy is as follows:
A) converting the data in the double-license plate vehicle data set B and the license plate data set C into data formats commonly used in a standard YOLO series;
B) respectively adopting a K-means clustering algorithm to obtain respective anchor anchoring values;
C) yolov 3-tiny license plate target detection network training;
D) yolov 3-tiny network test, calculating the correct detection rate of the license plate of the vehicle picture, and determining the weight of the model, wherein the correct detection rate LP of the license plateobj_accuracyExpressed as:
Figure FDA0003387409430000021
ncorrectnumber of vehicle pictures indicating that the vehicle has successfully detected all license plates, NpictureRepresenting a total number of vehicle pictures;
E) training an LPS/CR-NET character segmentation recognition network;
F) LPS/CR-NET network test, calculating the complete segmentation and recognition accuracy of license plate characters, primarily screening model weight, and the complete segmentation and recognition accuracy LP of license plate charactersCH-accutacyExpressed as:
Figure FDA0003387409430000022
wherein, n'correctRepresenting the number, N ', of license plates successfully recognized by all characters in the license plate picture'pictureRepresenting the total number of the license plate pictures;
determining the weight of an LPS/CR-NET network model by combining a yolov 3-tiny network, and detecting a license plate region through a yolov 3-tiny network; then inputting the detected license plate area into an LPS/CR-NET for character segmentation and recognition; sorting characters of the character segmentation recognition result, and outputting a license plate number; if the number plate prediction frame is larger than 0.5 and the number plate number is consistent with the true value IoU, the number plate prediction identification is successful; if all license plates in the vehicle are successfully identified, the vehicle picture is successfully identified; and calculating the identification accuracy of the vehicle picture, and determining the LPS/CR-NET network model weight with the highest accuracy.
2. The method for training the license plate recognition model of claim 1, wherein the constructing a double-license plate vehicle data set B and a license plate data set C according to the collected vehicle picture samples with double license plates comprises:
collecting a vehicle picture sample with double license plates;
respectively carrying out position calibration on the vehicle, the license plate and the characters in the collected vehicle picture sample by adopting a rectangular frame to form a vehicle region rectangular frame, a license plate region rectangular frame and a character region rectangular frame;
rotating the vehicle picture sample subjected to position calibration to obtain a data set A subjected to data amplification;
according to the vehicle area rectangular frame, cutting a vehicle picture in each vehicle picture sample of the data set A to construct a double-license vehicle data set B; each labeling file for cutting out the vehicle picture in the double-license plate vehicle data set B comprises a license plate region rectangular frame and a character region rectangular frame;
calculating the license plate number corresponding to the license plate by using a character sorting algorithm according to the characters in the rectangular frame of the license plate area;
cutting out a license plate picture from each constructed vehicle picture of the double-license plate vehicle data set B according to the license plate region rectangular frame to construct a license plate data set C; and the labeling file for cutting out the license plate picture from the license plate data set C comprises a license plate number corresponding to the license plate, a character area rectangular frame and a character type.
3. The license plate recognition model training method of claim 2, wherein in the step of performing position calibration on the collected vehicle, license plate and character in the vehicle picture sample by using the rectangular frame, a license plate region rectangular frame is formed in a clockwise or counterclockwise direction by using an angular point of an upper left corner of the license plate in the vehicle picture sample with double license plates as a starting point.
4. The license plate recognition model training method of claim 1, wherein the step of performing character sorting on the character segmentation recognition result and outputting the license plate number comprises the steps of:
calculating the average width of the bounding box of the n character regions
Figure FDA0003387409430000031
And average height
Figure FDA0003387409430000032
Arranging the x coordinates in the horizontal direction of the coordinates of the central position of the enclosing frame of the character area in an ascending order; let the i-th character region rectangular frame be Ki，i＝1，...，n；xiIs KiThe horizontal direction x coordinate of the center point of (a);
defaulting all characters on a second line;
calculating any two adjacent character area rectangular frames KiAnd KjHorizontal distance Δ w ofi,j＝xj-xiAnd a vertical distance Δ hi,j＝yj-yi(i 1, 2., n-1, j +1), constructing a discriminant;
Figure FDA0003387409430000041
Figure FDA0003387409430000042
wherein, P>0.8 and Δ hi,j<0, then K isjIn the first row; if P>0.8 and Δ hi,jIf greater than 0, then K is addediIn the first row;
calculating Ki－1，KiHorizontal distance ofΔwi-1,i＝xi-xi-1And a vertical distance Δ hi-1,i＝yi-yi-1(i ═ 1, 2., n-1, j ═ i +1), calculating P; if P is less than 0.7, then Ki－1At KiThe row position of the position;
and serially connecting the characters of the first line and the second line to output the license plate number.
5. A license plate recognition method is characterized by comprising the following steps:
obtaining a vehicle picture;
inputting the vehicle picture into a model, and performing double-license plate recognition on the vehicle picture to obtain a double-license plate recognition result; the license plate recognition model is obtained by training through the license plate recognition model training method of any one of claims 1-4.
6. A license plate recognition device, comprising:
the acquisition module is used for acquiring a vehicle picture;
the recognition module is used for inputting the vehicle pictures into a model and performing double-license plate recognition on the vehicle pictures, wherein the model is obtained by training through the license plate recognition model training method of any one of claims 1 to 4.
7. An electronic device, comprising: a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the license plate recognition model training method of any one of claims 1-4 or the license plate recognition method of claim 5 when executing the computer program.
8. A computer-readable storage medium storing computer instructions for causing a computer to execute the license plate recognition model training method of any one of claims 1 to 4 or the license plate recognition method of claim 5.
CN202011317882.6A 2020-11-20 2020-11-20 License plate recognition model training method, license plate recognition method and device Active CN112329881B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011317882.6A CN112329881B (en) 2020-11-20 2020-11-20 License plate recognition model training method, license plate recognition method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011317882.6A CN112329881B (en) 2020-11-20 2020-11-20 License plate recognition model training method, license plate recognition method and device

Publications (2)

Publication Number Publication Date
CN112329881A CN112329881A (en) 2021-02-05
CN112329881B true CN112329881B (en) 2022-02-15

Family

ID=74321954

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011317882.6A Active CN112329881B (en) 2020-11-20 2020-11-20 License plate recognition model training method, license plate recognition method and device

Country Status (1)

Country Link
CN (1) CN112329881B (en)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113159153A (en) * 2021-04-13 2021-07-23 华南理工大学 License plate recognition method based on convolutional neural network
CN113436180A (en) * 2021-07-07 2021-09-24 京东科技控股股份有限公司 Method, device, system, equipment and medium for detecting spray codes on production line
CN113643448A (en) * 2021-07-28 2021-11-12 深圳市捷顺科技实业股份有限公司 Multi-certificate passing processing method and device
CN113989794B (en) * 2021-11-12 2022-08-02 珠海安联锐视科技股份有限公司 License plate detection and recognition method
CN114511899A (en) * 2021-12-30 2022-05-17 武汉光庭信息技术股份有限公司 Street view video fuzzy processing method and system, electronic equipment and storage medium
CN114821230A (en) * 2022-04-18 2022-07-29 网易有道信息技术（北京）有限公司 Training method of target detection model, text line grouping method and related products

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110991444A (en) * 2019-11-19 2020-04-10 复旦大学 Complex scene-oriented license plate recognition method and device
CN111027534A (en) * 2018-10-09 2020-04-17 杭州海康威视数字技术股份有限公司 Compact double-license-plate detection method and device
CN111666938A (en) * 2020-05-21 2020-09-15 珠海大横琴科技发展有限公司 Two-place double-license-plate detection and identification method and system based on deep learning

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110378331B (en) * 2019-06-10 2022-10-04 南京邮电大学 End-to-end license plate recognition system and method based on deep learning
CN111914839B (en) * 2020-07-28 2024-03-19 特微乐行(广州)技术有限公司 Synchronous end-to-end license plate positioning and identifying method based on YOLOv3

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111027534A (en) * 2018-10-09 2020-04-17 杭州海康威视数字技术股份有限公司 Compact double-license-plate detection method and device
CN110991444A (en) * 2019-11-19 2020-04-10 复旦大学 Complex scene-oriented license plate recognition method and device
CN111666938A (en) * 2020-05-21 2020-09-15 珠海大横琴科技发展有限公司 Two-place double-license-plate detection and identification method and system based on deep learning

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Real-Time Brazilian License Plate Detection and Recognition Using Deep Convolutional Neural Networks;Sergio Montazzolli et al;《2017 30th SIBGRAPI Conference on Graphics, Patterns and Images》;20171231;第55-62页 *
基于改进YOLOv3和BGRU的车牌识别系统;史建伟 等;《计算机工程与设计》;20200831;第41卷(第8期);第2345-2351页 *

Also Published As

Publication number Publication date
CN112329881A (en) 2021-02-05

Similar Documents

Publication Publication Date Title
CN112329881B (en) License plate recognition model training method, license plate recognition method and device
US11455805B2 (en) Method and apparatus for detecting parking space usage condition, electronic device, and storage medium
CN107944450B (en) License plate recognition method and device
CN109033950B (en) Vehicle illegal parking detection method based on multi-feature fusion cascade depth model
CN113822247B (en) Method and system for identifying illegal building based on aerial image
CN112307989B (en) Road surface object identification method, device, computer equipment and storage medium
WO2020258077A1 (en) Pedestrian detection method and device
CN104239867A (en) License plate locating method and system
CN110781980B (en) Training method of target detection model, target detection method and device
CN110826429A (en) Scenic spot video-based method and system for automatically monitoring travel emergency
CN114140683A (en) Aerial image target detection method, equipment and medium
CN111435446A (en) License plate identification method and device based on L eNet
CN113343985B (en) License plate recognition method and device
CN110942456B (en) Tamper image detection method, device, equipment and storage medium
CN112906471A (en) Traffic signal lamp identification method and device
CN111444911B (en) Training method and device of license plate recognition model and license plate recognition method and device
CN112329886A (en) Double-license plate recognition method, model training method, device, equipment and storage medium
CN115880260A (en) Method, device and equipment for detecting base station construction and computer readable storage medium
US20240203128A1 (en) Systems and methods for determining target event
CN111553183A (en) Ship detection model training method, ship detection method and ship detection device
CN111008554A (en) Dynamic traffic zebra crossing interior impersonation pedestrian identification method based on deep learning
CN110909674A (en) Traffic sign identification method, device, equipment and storage medium
CN111931721B (en) Method and device for detecting color and number of annual inspection label and electronic equipment
CN112784494B (en) Training method of false positive recognition model, target recognition method and device
CN117765485A (en) Vehicle type recognition method, device and equipment based on improved depth residual error network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant