CN112906643A - License plate number identification method and device - Google Patents
License plate number identification method and device Download PDFInfo
- Publication number
- CN112906643A CN112906643A CN202110302814.0A CN202110302814A CN112906643A CN 112906643 A CN112906643 A CN 112906643A CN 202110302814 A CN202110302814 A CN 202110302814A CN 112906643 A CN112906643 A CN 112906643A
- Authority
- CN
- China
- Prior art keywords
- license plate
- character recognition
- character
- correction
- recognition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
- G06V20/54—Surveillance or monitoring of activities, e.g. for recognising suspicious objects of traffic, e.g. cars on the road, trains or boats
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/24—Aligning, centring, orientation detection or correction of the image
- G06V10/243—Aligning, centring, orientation detection or correction of the image by compensating for image skew or non-uniform image deformations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
- G06V20/63—Scene text, e.g. street names
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
- G06V20/625—License plates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
- Character Discrimination (AREA)
Abstract
The application discloses a license plate number recognition method and a license plate number recognition device, wherein the method comprises the steps of carrying out license plate domain recognition on an image in a law enforcement instrument video by using a trained license plate detector, and carrying out coarse correction on a license plate domain by using Radon transformation; inputting the license plate domain after rough correction into a license plate key point feature model for license plate key point feature positioning to obtain a precisely positioned license plate; carrying out secondary accurate correction on the license plate by affine transformation and a projection accurate correction mode in sequence; and (3) performing license plate number correction recognition on the corrected license plate by using an end-to-end license plate character recognition model and a license plate type matching template. According to the license plate correction method, license plates are corrected respectively through Radon transformation, affine transformation and projection accurate correction, the defect of single correction is overcome, and the correction efficiency is improved; through the end-to-end matching mechanism of the license plate character recognition model and the license plate type matching template, the problem of character false generation or character missing generation is avoided, and the complexity of license plate repeated cycle matching is reduced.
Description
Technical Field
The invention belongs to the technical field of license plate number recognition, and relates to a license plate number recognition method and device.
Background
The intelligent license plate recognition method is a license plate number recognition technology based on license plate features in a video image. The technology is widely applied to the fields of traffic vehicle violation supervision, traffic accident evidence obtaining and rechecking and the like. The technical route of vehicle license plate recognition at a conventional checkpoint can be summarized as follows: for an input processed image or a section of video, firstly, a vehicle detector is adopted to carry out primary detection on a vehicle in the video image, then, a vehicle region is subjected to secondary vehicle license plate region detection, then, a region to be detected of a vehicle license plate is input into a vehicle license plate key point feature model to extract key point features of the image of the vehicle license plate region, then, the key points of the vehicle license plate are corrected through affine transformation, horizontal projection and the like, and finally, the corrected vehicle license plate is subjected to character detection and number recognition technology, so that accurate recognition of the vehicle license plate number of a vehicle captured by a conventional checkpoint is realized.
However, the mobile wearable law enforcement instrument has complex and changeable case handling environment, the video shooting is easily interfered by factors such as illumination, jitter, wearing height and angle, the vehicle characteristics in the video image are seriously lost, the deviation angle and deformation of the license plate are large, and the difficulty in license plate identification of the video image of the law enforcement instrument is increased. The accurate identification of the license plate number is crucial to traffic violation and accident verification and evidence collection, and the problems encountered by license plate identification of a law enforcement instrument cannot be effectively solved by a conventional checkpoint vehicle primary to license plate domain secondary detection mode and a single inclined license plate angle correction method by means of license plate feature extraction of a traditional method, so that an intelligent license plate identification technical scheme aiming at the law enforcement instrument is urgently needed.
Disclosure of Invention
The application provides a license plate number recognition method and a license plate number recognition device, and the technical scheme is as follows:
in a first aspect, a license plate number recognition method is provided, and the method includes:
utilizing a trained license plate detector to perform license plate domain recognition on an image in a law enforcement instrument video, and utilizing Radon transformation to perform rough correction on a license plate domain;
inputting the license plate domain after rough correction into a license plate key point feature model for license plate key point feature labeling to obtain a license plate containing the license plate key point features labeled;
respectively carrying out secondary accurate correction on the license plate by sequentially utilizing affine transformation and a projection accurate correction mode;
and performing license plate number correction recognition on the license plate after secondary accurate correction by using a trained end-to-end license plate character recognition model and a license plate type matching template, wherein the end-to-end license plate character recognition model label corresponds to the character category number of the license plate number.
Optionally, before the license plate key point feature is input to the license plate key point feature model for license plate key point feature labeling, the method further includes:
and carrying out random external expansion on the marked coordinates of the license plate key points to obtain a license plate domain with the coordinates of the external expanded license plate key points.
Optionally, before the license plate number correction recognition is performed on the license plate after the secondary accurate correction by using the trained end-to-end license plate character recognition model and the license plate type matching template, the method further includes:
acquiring a sample set of character recognition boxes of various labels, wherein characters in each character recognition box are subjected to character pre-labeling, and the sample set comprises difficult samples;
training the end-to-end license plate character recognition model by using the sample set, wherein a CBAM (CBAM) feature attention module is added in a convolutional neural network of the license plate character recognition model, and the difficult sample is used for carrying out online adjustment on the attention module;
and stopping training when the correct character recognition rate of the license plate character recognition model reaches a preset recognition rate.
Optionally, the obtaining a sample set of character recognition boxes of various types of labels includes:
identifying the license plate type of a license plate in a sample license plate image, and acquiring a license plate template corresponding to the license plate type;
acquiring a character recognition frame in which each character contained in the license plate to be recognized is respectively positioned in the license plate image;
adding character recognition frames which are not generated and deleting character recognition frames which are generated mistakenly according to the license plate template to obtain a character recognition frame set;
and performing character pre-labeling on each character recognition box in the character recognition box set to obtain the sample set.
Optionally, the adding the character recognition box with missed generation includes:
and determining the size of the character recognition frame which is not generated according to the aspect ratio of the license plate to be recognized, and adding the character recognition frame which is not generated and has the size.
Optionally, the method further comprises:
extracting frames from the video frames to obtain a vehicle image containing vehicle characteristics;
screening license plate images of license plates with different inclination angles of the same vehicle from the vehicle images with complete license plates;
randomly selecting a preset number of license plate images from the screened license plate images as a license plate image data set, and increasing the proportion of difficult samples in the license plate image data set in a random copying, rotating and noise adding manner in a training process, wherein each license plate image in the license plate image data set is pre-labeled with the characteristics of license plate key points;
training the license plate detector by using the license plate image data set, wherein the license plate detector is used for identifying an image containing a license plate, and a CBAM convolution attention is added in a network model of the license plate detector;
when the license plate detector detects that the verification accuracy rate of the license plate image reaches a preset threshold value, stopping training;
and migrating the trained network model of the license plate detector into a network model which is reserved with a main SSD framework and has lower complexity, so as to obtain the trained license plate detector.
In a second aspect, the present application provides a license plate number recognition device, the device comprising:
the license plate recognition module is configured to perform license plate domain recognition on the image in the video of the law enforcement instrument by using the trained license plate detector;
the rough correction module is configured to firstly perform Radon transformation rough correction on the license plate domain identified by the license plate identification module;
the key point marking module is configured to input the license plate domain after the coarse correction of the coarse correction module into a license plate key point feature model for license plate key point feature marking, so as to obtain a license plate with the license plate key point features marked thereon;
the secondary correction module is configured to respectively perform secondary accurate correction on the license plate by sequentially utilizing affine transformation and a projection accurate correction mode;
and the number recognition module is configured to perform license plate number correction recognition on the license plate corrected by the secondary correction module by using an end-to-end license plate character recognition model and a license plate type matching template, wherein a label used by the end-to-end license plate character recognition model corresponds to the character category number of the license plate number.
Optionally, the apparatus further comprises:
and the outward expansion module is configured to perform random outward expansion on the marked coordinates of the license plate key points to obtain a license plate domain with the coordinates of the outward expanded license plate key points.
Optionally, the apparatus further comprises a first training module configured to:
acquiring a sample set of character recognition boxes of various labels, wherein characters in each character recognition box are subjected to character pre-labeling, and the sample set comprises difficult samples;
training the end-to-end license plate character recognition model by using the sample set, wherein a CBAM (CBAM) feature attention module is added in a convolutional neural network of the license plate character recognition model, and the difficult sample is used for carrying out online adjustment on the attention module;
and stopping training when the correct character recognition rate of the license plate character recognition model reaches a preset recognition rate.
Optionally, the first training module is configured to:
identifying the license plate type of a license plate in a sample license plate image, and acquiring a license plate template corresponding to the license plate type;
acquiring a character recognition frame in which each character contained in the license plate to be recognized is respectively positioned in the license plate image;
adding character recognition frames which are not generated and deleting character recognition frames which are generated mistakenly according to the license plate template to obtain a character recognition frame set;
and performing character pre-labeling on each character recognition box in the character recognition box set to obtain the sample set.
Optionally, the first training module is configured to:
and determining the size of the character recognition frame which is not generated according to the aspect ratio of the license plate to be recognized, and adding the character recognition frame which is not generated and has the size.
Optionally, the apparatus further comprises a second training module configured to:
extracting frames from the video frames to obtain a vehicle image containing vehicle characteristics;
screening license plate images of license plates with different inclination angles of the same vehicle from the vehicle images with complete license plates;
randomly selecting a preset number of license plate images from the screened license plate images as a license plate image data set, and increasing the proportion of difficult samples in the license plate image data set in a random copying, rotating and noise adding manner in the training process;
each license plate image in the license plate image data set is pre-labeled with the characteristics of the key points of the license plate;
training the license plate detector by using the license plate image data set, wherein the license plate detector is used for identifying an image containing a license plate, and a CBAM convolution attention is added in a network model of the license plate detector;
when the license plate detector detects that the verification accuracy rate of the license plate image reaches a preset threshold value, stopping training;
and migrating the trained network model of the license plate detector into a network model which is reserved with a main SSD framework and has lower complexity, so as to obtain the trained license plate detector.
The application can at least realize the following beneficial effects:
the license plate domain and the license plate are respectively corrected by the radon transform, affine transform and projection accurate correction modes, so that the defect of single correction is overcome, and the correction efficiency is improved; through the end-to-end matching mechanism of the license plate character recognition model and the license plate type matching template, the problem of character false generation or character missing generation is avoided, and the complexity of license plate repeated cycle matching is reduced.
Through the online circulation adjustment training of the difficult samples and the addition mechanism of the feature attention module, the learning strength of the difficult samples is strengthened by the convolutional network layer.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
FIG. 1 is a flowchart of a method for license plate number recognition provided in an embodiment of the present application;
FIG. 2A is a flowchart of a method for license plate number recognition provided in another embodiment of the present application;
FIG. 2B is a schematic flow chart of a license plate detector training process according to an embodiment of the present disclosure;
FIG. 2C is a schematic diagram illustrating a license plate character recognition model trained according to an embodiment of the present disclosure;
FIG. 3A is a schematic structural diagram of a license plate number recognition device according to an embodiment of the present disclosure;
fig. 3B is a schematic structural diagram of a license plate number recognition device according to another embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the invention, as detailed in the appended claims.
Fig. 1 is a flowchart of a method for identifying a license plate number provided in an embodiment of the present application, where the method for identifying a license plate number provided in the present application may include the following steps:
103, respectively carrying out secondary accurate correction on the license plate by sequentially utilizing affine transformation and a projection accurate correction mode;
the projection precision correction mode can comprise horizontal direction correction and vertical direction correction.
And step 104, utilizing an end-to-end license plate character recognition model and a license plate type matching template to perform license plate number correction recognition on the license plate subjected to secondary accurate correction.
The labels used by the end-to-end license plate character recognition model described herein correspond to the number of character classes of the license plate number.
In summary, the license plate number identification method provided by the application corrects the license plate domain and the license plate respectively through radon transformation, affine transformation and projection accurate correction, makes up the defect of single correction, and improves the correction efficiency; through the end-to-end matching mechanism of the license plate character recognition model and the license plate type matching template, the problem of character false generation or character missing generation is avoided, and the complexity of license plate repeated cycle matching is reduced.
Fig. 2A is a flowchart of a method for identifying a license plate number provided in another embodiment of the present application, where the method for identifying a license plate number provided in the present application may include the following steps:
before the license plate number is identified, a license plate detector needs to be trained, so that the trained license plate detector can be used for detecting and identifying the license plate image of the video frame.
In a possible implementation manner, please refer to fig. 2B, which is a schematic flow chart of a license plate detector training process provided in an embodiment of the present application, where the license plate detector training process in the present application may include the following steps:
the frame extraction method can extract key frames in video frames, or extract one frame of video frame every other predetermined number of video frames, obviously, in practical application, other frame extraction methods can also be adopted, which is not limited by the present application, as long as it is ensured that vehicle images containing vehicle features can be extracted from the video frames.
the license plates with different inclination angles are selected, so that the training of the license plate detector has better adaptability, and the recognition success rate of license plate recognition can be improved.
by adding the online adjusting mechanism of the difficult samples in the training process of the license plate detector model, the learning strength of the license plate detector model to the difficult samples can be improved.
And each license plate image in the license plate image data set is pre-labeled with the key point characteristics of the license plate.
In the training stage, the key point characteristics of the license plate can be artificially pre-labeled for each license plate image.
in the phase of training the license plate detector, the SSD network is used as a basic network to pre-train and detect the license plate detector, so that the feature recognition interference caused by the similar color ambiguity and the large inclination angle of the license plate domain is reduced, the attention of the network to the license plate feature is improved, a proper amount of CBAM convolution attention blocks are added into the network, the more important deep feature information of the license plate feature map is conveniently mined, the feature extraction capability of the network is enhanced, and the more accurate license plate channel and space feature information is obtained.
when the license plate detector detects that the verification accuracy of the license plate image reaches a preset threshold, the verification accuracy of the license plate detector is high, and continuous training can be stopped at the moment.
And step 2016, migrating the trained network model of the license plate detector to a network model with a reserved main SSD framework and lower complexity, and obtaining the trained license plate detector.
In order to reduce embedded model parameters and complexity of an application network model and improve timeliness, a knowledge distillation mode is used for transferring the trained network model into a network model which keeps a main SSD framework and is lower in complexity.
please refer to fig. 2C, which is a schematic diagram of a license plate character recognition model training process provided in an embodiment of the present application, where the training of the license plate character recognition model may include the following steps:
optionally, obtaining a sample set of character recognition boxes of various types of labels includes:
s1, recognizing the license plate type of the license plate in the sample license plate image, and acquiring a license plate template corresponding to the license plate type;
in practical application, a license plate type pre-labeling model needs to be trained in advance, and then the trained license plate type pre-labeling model is used for identifying the license plate type of a license plate in a license plate image.
After the license plate type of the license plate in the license plate image is identified, the license plate template corresponding to the license plate type can be obtained, so that the matching calculation complexity during matching verification of the character recognition frame is reduced.
In order to solve the problems of missing generation of a small number of character recognition frames and cyclic matching of a license plate and multiple templates, the recognized license plate types are efficiently matched with corresponding license plate templates, wherein the number of the license plate type matching templates is 5, and the number is respectively as follows: the single-layer blue cards and the yellow cards are of a character distribution standard, and the double-layer yellow cards, the police cards, the Hongkong and Macao cards and the new energy license plates are respectively provided.
S2, acquiring a character recognition frame where each character contained in the license plate to be recognized is located in the license plate image;
s3, adding character recognition frames which are not generated and deleting character recognition frames which are generated mistakenly according to the license plate template to obtain a character recognition frame set;
according to the license plate template, when the character identification frame is judged to be generated in a missing mode, the character identification frame which is generated in a missing mode can be added; when it is determined that the character recognition frame is generated by mistake, the character recognition frame generated by mistake can be deleted, and finally, a character recognition frame set is obtained.
In a possible implementation manner, when the character recognition frames which are not generated are added, the size of the character recognition frames which are not generated can be determined according to the aspect ratio of the license plate to be recognized, and the character recognition frames which are not generated and have the size are added.
And S4, performing character pre-labeling on each character recognition frame in the character recognition frame set to obtain a sample set.
the CBAM characteristic attention module is added in the convolutional neural network of the license plate character recognition model, and the difficult sample is used for carrying out online adjustment on the attention module.
In practical application, each character in the license plate number is a label amount in a license plate character recognition model, taking china as an example, 23 provinces (temporarily unavailable taiwan license plates), 5 municipalities, 4 municipalities in direct jurisdiction and 2 special administrative districts, 34 small units, 10 numbers 0-9, 25 capital letters (I is unused and O is used in a small amount), an alarm word of an additional alarm plate, a learning word of a coach plate and a hanging word of a double-layer yellow plate, and 72 label categories are totally adopted in china.
And step 2023, stopping training when the correct character recognition rate of the license plate character recognition model reaches a preset recognition rate.
After the training of the license plate detector and the license plate character recognition model is completed, the following license plate recognition process can be performed by using the models after the training.
and (3) carrying out license plate domain recognition on the image in the video of the law enforcement instrument by using the trained license plate detector to obtain a recognized license plate domain, and then carrying out coarse correction on the recognized license plate domain by using Radon transformation.
in practical application, the license plate key point marking model needs to be trained in advance, and then the trained license plate key point marking model is used for marking the license plate key point characteristics of the license plate domain.
in practical application, in order to increase the key point positioning robustness of a complex scene and a multi-scale license plate image, pixel point values are randomly expanded according to the coordinates of key points of the license plate (0-30), (5-60) and (10-90), and a license plate detection data set is randomly screened in a 0.2 proportion to train a license plate key point positioning model and a license plate type pre-labeling model.
The license plate key point model positioning model adopts a base network of SqueezeNet, in order to strengthen the learning attention of the inclined large-angle license plate features and improve the regression accuracy of the inclined license plate key point features, CBAM attention modules are correspondingly added among network modules, online difficult samples are adopted for noise adding, a slight rotation mode is adopted for data enhancement, and the online difficult data adjustment circulation is carried out until the online verification model positioning accuracy reaches the preset requirement. In addition, a knowledge distillation mode is also adopted to transfer the trained network model into a simple network of Fire and CBAM modules in a descending SqueezeNet structure, and then the optimal light-weight network model with the highest accuracy and the lowest model parameters is selected.
that is to say, firstly, pixel point values corresponding to random outward expansion of license plate key point feature coordinates are used as license plate key point feature data, and then rough correction is carried out on the license plate key point feature data by adopting Radon transformation, so that the difficulty of subsequent affine transformation is reduced, and the accuracy of license plate identification is improved.
Firstly, the license plate image which is accurately positioned is corrected by adopting affine transformation, and then the inclined deformation in the vertical direction is corrected again in a projection mode, so that the license plate with a slight or no inclined angle is obtained.
And step 207, utilizing an end-to-end license plate character recognition model and a license plate type matching template to perform license plate number correction recognition on the corrected license plate.
The labels of the convolutional neural network used by the end-to-end license plate character recognition model correspond to the characters of the license plate number.
For the corrected license plate feature image, a network architecture of a license plate character recognition model is based on an Fcos algorithm, multi-label target detection is achieved in a pixel-by-pixel prediction mode, complex calculation related to a predefined anchor frame is avoided, and video memory and model parameters are reduced. In addition, a proper amount of CBAM feature attention modules are added into the network framework, and online difficult data attention adjustment is also adopted to obtain the best license plate recognition model.
The method comprises the steps of utilizing a corresponding national standard template and a license plate type to match the needle vehicle license plate character frame, wherein the actual scale scaling is carried out by taking the width of a detected character as a scaling proportion, missing character frames are added according to corresponding intervals, and in addition, redundant wrong character frames are deleted according to template requirements under the condition that an additionally generated character frame is not matched with the template.
In summary, the license plate number identification method provided by the application corrects the license plate domain and the license plate respectively through radon transformation, affine transformation and projection accurate correction, makes up the defect of single correction, and improves the correction efficiency; through the end-to-end matching mechanism of the license plate character recognition model and the license plate type matching template, the problem of character false generation or character missing generation is avoided, and the complexity of license plate repeated cycle matching is reduced.
Through the online circulation adjustment training of the difficult samples and the addition mechanism of the feature attention module, the learning strength of the difficult samples is strengthened by the convolutional network layer.
Referring to fig. 3A, which is a schematic structural diagram of a license plate number recognition device provided in an embodiment of the present application, the license plate number recognition device provided in the present application may include: the license plate identification module 310, the rough correction module 320, the key point labeling module 330, the secondary correction module 340 and the number identification module 350.
The license plate recognition module 310 may be configured to perform license plate domain recognition on images in the law enforcement instrument video using a trained license plate detector;
the rough rectification module 320 is configured to perform Radon transformation rough rectification on the license plate domain identified by the license plate identification module 310;
the key point labeling module 330 may be configured to input the license plate domain after the rough correction by the rough correction module 320 into a license plate key point feature model for license plate key point feature labeling, so as to obtain a license plate including the license plate key point features labeled;
the secondary correction module 340 may be configured to perform secondary accurate correction on the license plate obtained by the key point labeling module 330 in a sequential affine transformation and projection accurate correction manner;
the number recognition module 350 may be configured to perform license plate number correction recognition on the license plate corrected by the secondary correction module 340 by using a trained end-to-end license plate character recognition model and the license plate type matching template, where the end-to-end license plate character recognition model label corresponds to the character type of the license plate number.
In a possible implementation manner, please refer to fig. 3B, which is a schematic structural diagram of a license plate number recognition device provided in another embodiment of the present application, the license plate number recognition device provided in the present application may further include: and an external expansion module 360.
The outward expansion module 360 may be configured to perform random outward expansion on the coordinates of the labeled license plate key points to obtain a license plate with the coordinates of the expanded license plate key points.
Still referring to fig. 3B, the license plate number recognition apparatus provided by the present application may further include a first training module 370, and the first training module 370 may be configured to perform the following operations:
acquiring a sample set of character recognition boxes of various labels, wherein characters in each character recognition box are subjected to character pre-labeling, and the sample set comprises difficult samples;
training the end-to-end license plate character recognition model by using the sample set, wherein a CBAM (CBAM) feature attention module is added in a convolutional neural network of the license plate character recognition model, and the difficult sample is used for carrying out online adjustment on the attention module;
and stopping training when the correct character recognition rate of the license plate character recognition model reaches a preset recognition rate.
Optionally, the first training module 370 may be further configured to perform the following operations:
identifying the license plate type of a license plate in a sample license plate image, and acquiring a license plate template corresponding to the license plate type;
acquiring a character recognition frame in which each character contained in the license plate to be recognized is respectively positioned in the license plate image;
adding character recognition frames which are not generated and deleting character recognition frames which are generated mistakenly according to the license plate template to obtain a character recognition frame set;
and performing character pre-labeling on each character recognition box in the character recognition box set to obtain the sample set.
Optionally, the first training module 370 may be further configured to perform the following operations:
and determining the size of the character recognition frame which is not generated according to the aspect ratio of the license plate to be recognized, and adding the character recognition frame which is not generated and has the size.
Still referring to fig. 3B, the license plate number recognition apparatus provided by the present application may further include a second training module 380, and the second training module 380 may be configured to perform the following operations:
extracting frames from the video frames to obtain a vehicle image containing vehicle characteristics;
screening license plate images of license plates with different inclination angles of the same vehicle from the vehicle images with complete license plates;
randomly selecting a preset number of license plate images from the screened license plate images as a license plate image data set, and increasing the proportion of difficult samples in the license plate image data set in a random copying, rotating and noise adding manner in the training process;
each license plate image in the license plate image data set is pre-labeled with the characteristics of the key points of the license plate;
training the license plate detector by using the license plate image data set, wherein the license plate detector is used for identifying an image containing a license plate, and a CBAM convolution attention is added in a network model of the license plate detector;
when the license plate detector detects that the verification accuracy rate of the license plate image reaches a preset threshold value, stopping training;
and migrating the trained network model of the license plate detector into a network model which is reserved with a main SSD framework and has lower complexity, so as to obtain the trained license plate detector.
In summary, the license plate number recognition device provided by the application corrects the license plate domain through radon transformation, affine transformation and accurate projection correction, makes up the defect of single correction, and improves the correction efficiency; through the end-to-end matching mechanism of the license plate character recognition model and the license plate type matching template, the problem of character false generation or character missing generation is avoided, and the complexity of license plate repeated cycle matching is reduced.
Through the online circulation adjustment training of the difficult samples and the addition mechanism of the feature attention module, the learning strength of the difficult samples is strengthened by the convolutional network layer.
Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the invention and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.
It will be understood that the invention is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the invention is limited only by the appended claims.
Claims (12)
1. A license plate number recognition method is characterized by comprising the following steps:
utilizing a trained license plate detector to carry out license plate domain recognition on images in the video of the law enforcement instrument;
firstly carrying out Radon transformation coarse correction on the identified license plate domain, inputting the license plate domain into a license plate key point feature model for carrying out license plate key point feature labeling, and obtaining a license plate containing the license plate key point feature labeled;
respectively carrying out secondary accurate correction on the license plate by sequentially utilizing affine transformation and a projection accurate correction mode;
and performing license plate number correction recognition on the license plate after secondary accurate correction by using a trained end-to-end license plate character recognition model and a license plate type matching template, wherein a label used by the end-to-end license plate character recognition model corresponds to the character category number of the license plate number.
2. The method of claim 1, wherein before the inputting to the license plate key point feature model for license plate key point feature labeling, the method further comprises:
and carrying out random external expansion on the marked coordinates of the license plate key points to obtain a license plate domain with the coordinates of the external expanded license plate key points.
3. The method of claim 1, wherein before the license plate number revision recognition is performed on the secondarily and accurately corrected license plate by using the trained end-to-end license plate character recognition model and the template matched with the license plate type, the method further comprises:
acquiring a sample set of character recognition boxes of various labels, wherein characters in each character recognition box are subjected to character pre-labeling, and the sample set comprises difficult samples;
training the end-to-end license plate character recognition model by using the sample set, wherein a CBAM (CBAM) feature attention module is added in a convolutional neural network of the license plate character recognition model, and the difficult sample is used for carrying out online adjustment on the attention module;
and stopping training when the correct character recognition rate of the license plate character recognition model reaches a preset recognition rate.
4. The method of claim 3, wherein obtaining a sample set of character recognition boxes for each type of label comprises:
identifying the license plate type of a license plate in a sample license plate image, and acquiring a license plate template corresponding to the license plate type;
acquiring a character recognition frame in which each character contained in the license plate to be recognized is respectively positioned in the license plate image;
adding character recognition frames which are not generated and deleting character recognition frames which are generated mistakenly according to the license plate template to obtain a character recognition frame set;
and performing character pre-labeling on each character recognition box in the character recognition box set to obtain the sample set.
5. The method of claim 4, wherein the adding of the missed character recognition box comprises:
and determining the size of the character recognition frame which is not generated according to the aspect ratio of the license plate to be recognized, and adding the character recognition frame which is not generated and has the size.
6. The method of claim 1, further comprising:
extracting frames from the video frames to obtain a vehicle image containing vehicle characteristics;
screening license plate images of license plates with different inclination angles of the same vehicle from the vehicle images with complete license plates;
randomly selecting a preset number of license plate images from the screened license plate images as a license plate image data set, and increasing the proportion of difficult samples in the license plate image data set in a random copying, rotating and noise adding manner in the training process;
each license plate image in the license plate image data set is pre-labeled with the characteristics of the key points of the license plate;
training the license plate detector by using the license plate image data set, wherein the license plate detector is used for identifying an image containing a license plate, and a CBAM convolution attention is added in a network model of the license plate detector;
when the license plate detector detects that the verification accuracy rate of the license plate image reaches a preset threshold value, stopping training;
and migrating the trained network model of the license plate detector into a network model which is reserved with a main SSD framework and has lower complexity, so as to obtain the trained license plate detector.
7. A license plate number recognition apparatus, characterized in that the apparatus comprises:
the license plate recognition module is configured to perform license plate domain recognition on the image in the video of the law enforcement instrument by using the trained license plate detector;
the rough correction module is configured to firstly perform Radon transformation rough correction on the license plate domain identified by the license plate identification module;
the key point marking module is configured to input the license plate domain after the coarse correction of the coarse correction module into a license plate key point feature model for license plate key point feature marking, so as to obtain a license plate with the license plate key point features marked thereon;
the secondary correction module is configured to respectively perform secondary accurate correction on the license plate by sequentially utilizing affine transformation and a projection accurate correction mode;
and the number recognition module is configured to perform license plate number correction recognition on the license plate corrected by the secondary correction module by using an end-to-end license plate character recognition model and a license plate type matching template, wherein a label used by the end-to-end license plate character recognition model corresponds to the character category number of the license plate number.
8. The apparatus of claim 7, further comprising:
and the outward expansion module is configured to perform random outward expansion on the marked coordinates of the license plate key points to obtain a license plate domain with the coordinates of the outward expanded license plate key points.
9. The apparatus of claim 7, further comprising a first training module configured to:
acquiring a sample set of character recognition boxes of various labels, wherein characters in each character recognition box are subjected to character pre-labeling, and the sample set comprises difficult samples;
training the end-to-end license plate character recognition model by using the sample set, wherein a CBAM (CBAM) feature attention module is added in a convolutional neural network of the license plate character recognition model, and the difficult sample is used for carrying out online adjustment on the attention module;
and stopping training when the correct character recognition rate of the license plate character recognition model reaches a preset recognition rate.
10. The apparatus of claim 9, wherein the first training module is configured to:
identifying the license plate type of a license plate in a sample license plate image, and acquiring a license plate template corresponding to the license plate type;
acquiring a character recognition frame in which each character contained in the license plate to be recognized is respectively positioned in the license plate image;
adding character recognition frames which are not generated and deleting character recognition frames which are generated mistakenly according to the license plate template to obtain a character recognition frame set;
and performing character pre-labeling on each character recognition box in the character recognition box set to obtain the sample set.
11. The apparatus of claim 10, wherein the first training module is configured to:
and determining the size of the character recognition frame which is not generated according to the aspect ratio of the license plate to be recognized, and adding the character recognition frame which is not generated and has the size.
12. The apparatus of claim 7, further comprising a second training module configured to:
extracting frames from the video frames to obtain a vehicle image containing vehicle characteristics;
screening license plate images of license plates with different inclination angles of the same vehicle from the vehicle images with complete license plates;
randomly selecting a preset number of license plate images from the screened license plate images as a license plate image data set, and increasing the proportion of difficult samples in the license plate image data set in a random copying, rotating and noise adding manner in the training process;
each license plate image in the license plate image data set is pre-labeled with the characteristics of the key points of the license plate;
training the license plate detector by using the license plate image data set, wherein the license plate detector is used for identifying an image containing a license plate, and a CBAM convolution attention is added in a network model of the license plate detector;
when the license plate detector detects that the verification accuracy rate of the license plate image reaches a preset threshold value, stopping training;
and migrating the trained network model of the license plate detector into a network model which is reserved with a main SSD framework and has lower complexity, so as to obtain the trained license plate detector.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110302814.0A CN112906643A (en) | 2021-03-22 | 2021-03-22 | License plate number identification method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110302814.0A CN112906643A (en) | 2021-03-22 | 2021-03-22 | License plate number identification method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112906643A true CN112906643A (en) | 2021-06-04 |
Family
ID=76106316
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110302814.0A Pending CN112906643A (en) | 2021-03-22 | 2021-03-22 | License plate number identification method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112906643A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114898353A (en) * | 2022-07-13 | 2022-08-12 | 松立控股集团股份有限公司 | License plate identification method based on video sequence image characteristics and information |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106650732A (en) * | 2016-12-27 | 2017-05-10 | 深圳市捷顺科技实业股份有限公司 | License plate identification method and apparatus |
| US20170177965A1 (en) * | 2015-12-17 | 2017-06-22 | Xerox Corporation | Coarse-to-fine cascade adaptations for license plate recognition with convolutional neural networks |
| CN109492642A (en) * | 2018-09-25 | 2019-03-19 | 平安科技（深圳）有限公司 | Licence plate recognition method, device, computer equipment and storage medium |
| CN111310850A (en) * | 2020-03-02 | 2020-06-19 | 杭州雄迈集成电路技术股份有限公司 | License plate detection model construction method and system and license plate detection method and system |
| CN111709884A (en) * | 2020-04-29 | 2020-09-25 | 高新兴科技集团股份有限公司 | License plate key point correction method, system, equipment and storage medium |
| CN111950546A (en) * | 2020-08-06 | 2020-11-17 | 广东飞翔云计算有限公司 | License plate recognition method and device, computer equipment and storage medium |
| CN112052850A (en) * | 2020-09-03 | 2020-12-08 | 平安科技（深圳）有限公司 | License plate recognition method and device, electronic equipment and storage medium |
-
2021
- 2021-03-22 CN CN202110302814.0A patent/CN112906643A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170177965A1 (en) * | 2015-12-17 | 2017-06-22 | Xerox Corporation | Coarse-to-fine cascade adaptations for license plate recognition with convolutional neural networks |
| CN106650732A (en) * | 2016-12-27 | 2017-05-10 | 深圳市捷顺科技实业股份有限公司 | License plate identification method and apparatus |
| CN109492642A (en) * | 2018-09-25 | 2019-03-19 | 平安科技（深圳）有限公司 | Licence plate recognition method, device, computer equipment and storage medium |
| CN111310850A (en) * | 2020-03-02 | 2020-06-19 | 杭州雄迈集成电路技术股份有限公司 | License plate detection model construction method and system and license plate detection method and system |
| CN111709884A (en) * | 2020-04-29 | 2020-09-25 | 高新兴科技集团股份有限公司 | License plate key point correction method, system, equipment and storage medium |
| CN111950546A (en) * | 2020-08-06 | 2020-11-17 | 广东飞翔云计算有限公司 | License plate recognition method and device, computer equipment and storage medium |
| CN112052850A (en) * | 2020-09-03 | 2020-12-08 | 平安科技（深圳）有限公司 | License plate recognition method and device, electronic equipment and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114898353A (en) * | 2022-07-13 | 2022-08-12 | 松立控股集团股份有限公司 | License plate identification method based on video sequence image characteristics and information |
| CN114898353B (en) * | 2022-07-13 | 2022-10-25 | 松立控股集团股份有限公司 | License plate recognition method based on video sequence image characteristics and information |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107944450B (en) | License plate recognition method and device | |
| CN105787466B (en) | A kind of fine recognition methods and system of type of vehicle | |
| CN104616009A (en) | Character cutting and recognizing method | |
| CN104809481A (en) | Natural scene text detection method based on adaptive color clustering | |
| CN110569856B (en) | Sample labeling method and device, and damage category identification method and device | |
| CN102289686A (en) | Method for identifying classes of moving targets based on transfer learning | |
| CN112052850A (en) | License plate recognition method and device, electronic equipment and storage medium | |
| CN112766218B (en) | Cross-domain pedestrian re-recognition method and device based on asymmetric combined teaching network | |
| CN104881665A (en) | Chip character identification and verification method and apparatus | |
| CN111191611A (en) | Deep learning-based traffic sign label identification method | |
| CN110675637A (en) | Vehicle illegal video processing method and device, computer equipment and storage medium | |
| CN111368682A (en) | Method and system for detecting and identifying station caption based on faster RCNN | |
| CN114782770A (en) | License plate detection and recognition method and system based on deep learning | |
| CN112906643A (en) | License plate number identification method and device | |
| CN115019294A (en) | Pointer instrument reading identification method and system | |
| CN116343237A (en) | Bill identification method based on deep learning and knowledge graph | |
| CN112949523A (en) | Method and system for extracting key information from identity card image picture | |
| CN116740335A (en) | Equipment indication recognition method, device and system | |
| CN108734158B (en) | Real-time train number identification method and device | |
| CN111402185A (en) | Image detection method and device | |
| CN115116073A (en) | Washing label identification method and device, storage medium and electronic device | |
| CN112380970B (en) | Video target detection method based on local area search | |
| CN111639640B (en) | License plate recognition method, device and equipment based on artificial intelligence | |
| CN114519791A (en) | License plate character recognition method and device | |
| CN116704518A (en) | Text recognition method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |