CN113723408B - License plate recognition method and system and readable storage medium - Google Patents
License plate recognition method and system and readable storage medium Download PDFInfo
- Publication number
- CN113723408B CN113723408B CN202111288251.0A CN202111288251A CN113723408B CN 113723408 B CN113723408 B CN 113723408B CN 202111288251 A CN202111288251 A CN 202111288251A CN 113723408 B CN113723408 B CN 113723408B
- Authority
- CN
- China
- Prior art keywords
- license plate
- character
- vehicle
- model
- image data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Character Discrimination (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a license plate recognition method, a license plate recognition system and a readable storage medium, wherein the method comprises the following steps: s1 preparing a data set: collecting vehicle image data, and labeling each character in the vehicle and the license plate image data thereof according to a first rule; s2, selecting a basic model and carrying out secondary design, wherein the secondary design step comprises the following steps: performing enlargement processing on the size of input image data of the model; performing cluster analysis on the minimum circumscribed rectangular frame of the object marked in the step S1 to obtain anchors of the data set; and carrying out post-processing and rewriting on the model, and sequencing the disordered recognition result output by the model according to a second rule. Thereby realizing that the vehicle license plate recognition problem is solved by a process by adopting a single deep learning model.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a method, a system and a readable storage medium for recognizing a license plate by adopting the artificial intelligence technology through deep learning of a specific model.
Background
The automatic license plate number identification technology in the prior art is mainly cross application of a computer vision technology in the field of transportation, and has wide application markets in the fields of road violation detection, intelligent parking lots and the like.
In order to achieve the purpose of effectively recognizing the license plate, the prior art is mainly divided into: the identification scheme based on the traditional image processing and the identification scheme combining the traditional image processing with deep learning are two categories, however, the traditional image processing scheme is relatively slow in identification speed, low in identification accuracy and easy to be influenced by factors such as illumination weather angles.
The traditional image processing technology and the deep learning combined scheme need to be carried out in two processes, namely, the license plate is positioned by using a deep learning method, and then the license plate number at the position of the license plate is identified by using a traditional image processing method. Although the introduction of deep learning brings certain speed and precision improvement, the upper limit of the precision and speed is still limited because the traditional image processing method is still available.
Therefore, in the prior art, a deep learning and deep learning method has been provided, that is, a deep learning method is used to locate the license plate position, and then a deep learning method is used to identify the license plate number at the obtained position.
Meanwhile, the two models are inherently in a disadvantage in recognition speed, and if the two models are applied to edge equipment, the disadvantage can result in that the two models cannot operate at all; furthermore, the two models are susceptible to interaction, and optimization of the models is therefore more difficult.
Disclosure of Invention
Therefore, the present invention is directed to a license plate recognition method, a system thereof, and a readable storage medium thereof, for solving a vehicle license plate recognition problem through one process using a single deep learning model.
In order to achieve the above object, according to a first aspect of the present invention, there is provided a license plate recognition method, including the steps of: s1 preparing a data set: collecting vehicle image data, and labeling each character in the vehicle and the license plate image data thereof according to a first rule; s2, selecting a basic model and carrying out secondary design, wherein the secondary design step comprises the following steps: the increasing the size of the input image data of the model includes: processing the area near the license plate number by a data enhancement processing means to increase the resolution of the local area near the license plate number; performing cluster analysis on the minimum circumscribed rectangle of the object labeled in the step S1 to obtain anchors of the data set and setting branches in the headers of the model, which are responsible for predicting license plate characters, to increase the size of the output characteristic diagram of the model; and carrying out post-processing and rewriting on the model, and sequencing the disordered recognition result output by the model according to a second rule.
In a possible preferred embodiment, said first law comprises: the vehicle image data is sorted according to a preset proportion into: the method comprises a training set, a verification set and a test set, wherein only each character in the vehicle and the license plate image data in the training set and the verification set is labeled.
In a possible preferred embodiment, the labeling step comprises: labeling a vehicle type in the image data and labeling each character in a license plate of a corresponding vehicle, and the labels include: the type of each character in the vehicle and the license plate thereof, and the coordinates of the fixed point at the upper left corner and the fixed point at the lower right corner of the minimum circumscribed rectangle of each character in the vehicle and the corresponding license plate thereof.
In a possible preferred embodiment, the secondary design step further comprises: setting backsbone of the model, comprising the following steps: deleting p1, reducing the number of p 2-p 4 by one time, and keeping the p5 unchanged; s1 closest to the input image is retained, while the convolution step sizes of s2 and s4 are changed to 1, and s5 close to the output remains unchanged.
In a possible preferred embodiment, the secondary design step further comprises: setting a detection head of the model, wherein the steps comprise: making the first branch in the detection head compensate, modifying convolution step sizes in h1_2 and h1_4 and making the convolution step sizes responsible for detection of the vehicle types; the second branch in the detection head is responsible for detecting the vehicle type and the license plate characters; let the third branch in the inspection head be responsible for the inspection of license plate characters, where h1_2 means the layer 2 structure of the first inspection head of yolov3, and h1_4 means the layer 4 structure of the first inspection head of yolov 3.
In a possible preferred embodiment, said second law comprises: judging which vehicle target is at each license plate character target position; and sorting the character targets in the same vehicle target according to the position result of the upper left corner of the character targets.
In a possible preferred embodiment, the method further comprises a step S3 of model training and optimization, which comprises the steps of: training the secondarily designed model in the step S2 by preset hyper-parameter combinations; testing each trained model on a test set; comparing the test results to optimize; and repeating the steps until the model reaches the standard.
In order to achieve the above object, according to a second aspect of the present invention, there is provided a license plate recognition method, including the steps of:
s1 preparing a data set: collecting vehicle image data, and labeling each character in the vehicle and the license plate image data thereof according to a first rule; wherein the labeling step comprises: labeling a vehicle type in the image data and labeling each character in a license plate of a corresponding vehicle, and the labels include: the type of each character in the vehicle and the license plate thereof, and the coordinates of the fixed point at the upper left corner and the fixed point at the lower right corner of the minimum circumscribed rectangle of each character in the vehicle and the corresponding license plate thereof.
S2, selecting a basic model and carrying out secondary design, wherein the secondary design step comprises the following steps: the increasing the size of the input image data of the model includes: processing the area near the license plate number by a data enhancement processing means to increase the resolution of the local area near the license plate number; performing clustering analysis on the minimum circumscribed rectangular frame of the object labeled in the step S1 to obtain anchors of the data set, setting branches in the headers of the model, which are responsible for predicting license plate characters, to increase the size of the output characteristic diagram of the model, and outputting a disordered recognition result;
s3 post-processing the model to rewrite: sequencing the disordered results output in the step S2, and setting each license plate character to be detected and output as
Each vehicle of which the kind is detected is
Each of which
Four position coordinates are set:
，
，
，
are respectively a character
The x coordinate and the y coordinate of the vertex of the upper left corner of the rectangle and the x coordinate and the y coordinate of the fixed point of the lower right corner are positioned; each one of which is
Setting four position coordinates
，
，
，
Are respectively a character
The x coordinate and the y coordinate of the vertex at the upper left corner of the rectangle and the x coordinate and the y coordinate of the fixed point at the lower right corner, if the coordinates are the same
≧ 
≧ 
And is
≧ 
≧ 
Then the character is judged
Belonging to vehicles
One of the license plate characters of (1) then for all belonging vehicles
Is a character of
According to which
Is sorted by size.
In order to achieve the above object, according to a third aspect of the present invention, there is also provided a license plate recognition system including: the main controller, the camera, the said camera is connected with main controller, in order to transmit the vehicle image data to the main controller, wherein store the number plate recognition method step as in any claim 1 to 8 in the said main controller, in order to carry out the number plate number in the recognition vehicle image data in the main controller.
In order to achieve the above object, according to a fourth aspect of the present invention, there is also provided a readable storage medium having stored thereon a computer program, which when executed by a processor, performs the steps of the license plate recognition method as recited in any one of the above.
The license plate recognition method, the license plate recognition system and the readable storage medium have the advantages that a single deep learning model is adopted, the license plate recognition problem is solved through one process, and the recognition speed and the recognition accuracy are better than those of a traditional image processing method or a method combining traditional image processing and deep learning; meanwhile, the identification accuracy of the scheme can be infinitely close to 100 percent along with the increase of the data sets, and the method is not limited by the precision and the speed of the traditional processing method; in addition, the scheme can meet the requirements of detecting and positioning different types of vehicles, license plates with different colors and numbers of different license plates through one model, and special processing is not required to be carried out on special license plate colors, so that the method has remarkable progress compared with the prior art.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the invention and, together with the description, serve to explain the invention and not to limit the invention. In the drawings:
FIG. 1 is a schematic flow chart illustrating steps of a first embodiment of the present invention;
FIG. 2 is a diagram illustrating a definition of a minimum bounding rectangle according to a first embodiment of the present invention;
fig. 3 is a schematic diagram illustrating an example of vehicle and license plate recognition according to a first embodiment of the present invention.
Detailed Description
The following describes in detail embodiments of the present invention. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications can be made by persons skilled in the art without departing from the spirit of the invention. All falling within the scope of the present invention.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present invention will be described in detail below with reference to the embodiments with reference to the attached drawings.
In order to make those skilled in the art better understand the technical solution of the present invention, the technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, shall fall within the scope of the present invention.
It should be noted that the terms "first", "second", "S1", "S2", and the like in the description and claims of the present invention and the above-described drawings are used for distinguishing similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions.
(A)
Referring to fig. 1, to achieve the object of the present invention, an embodiment of the present invention provides a license plate recognition method, which includes:
step S1 creates a data set
Wherein the manufacturing process of the data set comprises:
step S11 acquires a vehicle image: examples are as in the common vehicle categories: the method is characterized in that the method comprises the following steps of collecting public images of vehicles through a search engine or any method in the prior art, extracting the four images from various types of data sets of open sources, and collecting videos accessed by cameras at intersections.
The collected data sets are then sorted according to a predetermined ratio, such as 6: 2: the proportion of 2 is divided into three parts which are respectively: training set, verifying set and testing set; the training set is used for training the model and directly influences the quality of the model; the verification set is used for evaluating the optimization degree of training in the training process; the test set is generally allocated in this proportion for evaluating the model after training, but if it is used in industrial application, the test set needs to occupy a larger proportion, and even the test set can be compared with the training set 1: 1, and therefore the relevant sorting ratio in this embodiment is only an example and not a limitation, and those skilled in the art can adjust the ratio according to the needs of actual situations.
The collected vehicle images can contain images of one or more vehicles, the vehicles are not required to be complete, but the license plate numbers are required to be complete, and the images can be provided under various scenes such as different angles, different illumination conditions, different weather backgrounds and the like, but the license plate numbers are required to be visible to the naked eyes.
Step S12 image data annotation: the test set only needs to be labeled, and the test set does not need to be labeled; it should be noted that, in the preferred embodiment, the label preferably includes the type of the vehicle and each character, and the coordinates of the top left corner and the bottom right corner of the minimum bounding rectangle of the vehicle and each character.
Examples are illustrated by using 4 types of license plates of cars (car), buses (bus), trucks (truck), motorcycles (motorbike) in china as an example, wherein the license plate comprises 38 chinese characters (four of 34 provinces correspond to two chinese characters, such as kansu has a word and a sweet, sichuan has a chinese character and a holy, guizhou has a black character and a precious character, Yunnan has a Yunnan character and a cloud), 24 english letters (without I and O), 10 numbers (0-9), 4 vehicles and 76 types in total, wherein if english characters are used in the label, pinyin can be used to replace the chinese characters, and considering that the pronunciation of the chinese characters is the same and that four provinces have two corresponding chinese characters, for distinguishing, the 38 chinese characters in the example can be used with the names of the corresponding provinces, and underlined and the chinese characters are added, such as jingsu _ su in the label corresponding to "su", and then, a table of representing characters from the Chinese characters to the label is made for subsequent use when the Chinese characters are needed. For example, an image of only one passenger car, the number plate of the passenger car is: thre E05EV8, the corresponding labels for this image are:
bus left top right bottom
Jiangsu_su left top right bottom
E left top right bottom
0 left top right bottom
5 left top right bottom
E left top right bottom
V left top right bottom
8 left top right bottom
the left is the x coordinate of the top left corner vertex of the minimum circumscribed rectangle of the object of the corresponding type, the top is the y coordinate of the top left corner fixed point, the right is the x coordinate of the bottom right corner vertex, and the bottom is the y coordinate of the bottom right corner vertex.
To further illustrate the definition of the minimum bounding rectangle, as shown in FIG. 2, the coordinate system of the image is the X-axis to the right and the Y-axis down, for example: the vertex of the upper left corner of an image is the origin of the coordinate system, the coordinates are (0, 0), as shown in fig. 2, a character (jing) is a character in the license plate, the character is enclosed by a rectangle in the present case, and the smallest rectangle in the rectangles capable of enclosing the character is the smallest circumscribed rectangle of the character, namely the rectangle at the outer layer of the rectangle in the figure (jing).
The position of the rectangle in the image coordinate system is used for representing the position of the character in the image coordinate system, the position of the rectangle can be represented by the coordinates of only two points, the coordinates of the upper left vertex and the coordinates of the lower right vertex of the rectangle are represented by left, the coordinates of the upper left vertex and the coordinates of the lower right vertex are represented by right, the coordinates of the upper left vertex and the coordinates of the lower right vertex are represented by top, and bottom represents the coordinates of the lower right vertex. Therefore, (left, top) is the coordinates of the top left vertex, and (right, bottom) is the coordinates of the bottom right vertex. The four values of these two points determine the position of the character in the image.
Therefore, the design scheme of the circumscribed rectangle is obviously different from the scheme of marking the whole license plate in the prior art, and the method for marking each specific character in the license plate can effectively realize that each specific license plate symbol is identified and positioned in one process.
Step S2 base model selection
In theory, the scheme of the invention supports all mainstream target detection basic models, such as: yolov3, yolov4, yolov5, centernet, etc., but since the aforementioned labeling method is to label each character as an individual target, there arises a problem that the target to be recognized is a very small target, that is, the area of the target occupies a small proportion of the whole image, which results in that a satisfactory processing result cannot be obtained when the method is directly applied to the basic models.
Therefore, the problem needs to be purposefully designed secondarily based on the basic model. Although yolov3 is taken as an example to illustrate the basic model, and yolov3 is designed twice for the problem, it should be noted that the present embodiment does not limit the kind of the basic model, and any secondary design scheme made for other basic models according to the concept of the present embodiment and equivalent to the present invention is within the scope of the present embodiment.
Step S3 model quadratic design
Because the scheme of the embodiment proposes the labeling training with a single character as a target, three problems will be brought:
firstly, the detection is difficult because the single character object is too small, i.e. most character types are very small objects.
Secondly, each character in the license plate is closely arranged and has a small spacing distance, and the characters have basically the same size, so that the challenge caused by the characteristic is greater, and the problem that the label is rewritten is serious. Since each anchor of each grid specified by yolo3 is responsible for predicting the characteristics of an object, when such objects are encountered in a data set with close distance and the same size, it is easy to have a plurality of license plate characters allocated to a certain anchor of the grid, and since one anchor receives only one object, the rest of the license plate characters are ignored as background.
Third, the original yolov3 model would output all detected targets, but these targets are unordered and the license plate numbers are ordered.
Therefore, in a preferred embodiment, and therefore to solve the above problem, the step of model quadratic design includes:
step S31: the size of the input image of the model is increased from 416X416 to 1280X1280 in the example, so that certain information can be kept after the feature extraction of the multilayer network structure. It is worth mentioning that the solution of increasing the size of the input image of the model in step S31 can also improve or even eliminate the problem that a plurality of license plate numbers may appear in a grid, thereby causing the label rewriting phenomenon, because in the process of increasing the size of the input image, the data enhancement processing means such as cutting and splicing is usually adopted to process the area near the license plate numbers, thereby increasing the resolution of the local area near the license plate numbers, thereby utilizing the phenomenon to reduce the probability that a plurality of license plate numbers are assigned to a grid, and thereby eliminating and improving the label rewriting problem when there are not so many targets assigned in a grid.
Step S32:
step a1 processes the labels in the training set, for example, performing cluster analysis on the minimum bounding rectangle of each labeled object to obtain anchors belonging to the training set, where the obtained anchors may be, for example:
[8,6, 10,7, 11,10]
[28,20, 47,62, 71,132]
[108,96, 137,168, 356,284]
three groups of anchors correspond to three branches of yolov3, namely 3 heads, and each head outputs a respective detection result;
step A2: the yolov3 backbone was modified to the original backbone structure as follows:
# [from, number, module, args]
[
[-1, 1, Conv, [32, 3, 1]],
[-1, 1, Conv, [64, 3, 2]], #s1
[-1, 1, Bottleneck, [64]], #p1
[-1, 1, Conv, [128, 3, 2]], #s2
[-1, 2, Bottleneck, [128]], #p2
[-1, 1, Conv, [256, 3, 2]], #s3
[-1, 8, Bottleneck, [256]], #p3
[-1, 1, Conv, [512, 3, 2]], #s4
[-1, 8, Bottleneck, [512]], #p4
[-1, 1, Conv, [1024, 3, 2]], #s5
[-1, 4, Bottleneck, [1024]]#p5
]
considering that most characters are very small targets, when the design of the backbone is carried out, the influence of the depth of the network structure on the small targets is considered, but the model is required to be kept to have enough fitting capability, and because the size of an input image is increased to 1280, the calculation amount is increased by multiple times, so that the depth of the model is required to be reduced, the calculation amount is also required to be reduced, through experiments, p1 is preferably deleted, the number of p 2-p 4 is reduced by one time, and for better fitting, p5 is kept unchanged; for s 1-s 5, in order to retain more information, s1 closest to the input image is retained, the convolution step size of s2 and s4 is changed to 1, and s5 close to the output end is kept unchanged; the modified backbone structure is as follows:
# [from, number, module, args]
[
[-1, 1, Conv, [32, 3, 1]],
[-1, 1, Conv, [64, 3, 2]], #s1
[-1, 1, Conv, [128, 3, 1]], #s2
[-1, 1, Bottleneck, [128]], #p2
[-1, 1, Conv, [256, 3, 2]], #s3
[-1, 4, Bottleneck, [256]], #p3
[-1, 1, Conv, [512, 3, 1]], #s4
[-1, 4, Bottleneck, [512]], #p4
[-1, 1, Conv, [1024, 3, 2]], #s5
[-1, 4, Bottleneck, [1024]]#p5
]
step A3: the detection head of yolov3 was modified: the original test head structure is as follows:
head:
[[-1, 1, Bottleneck, [1024, False]],#h1_1
[-1, 1, Conv, [512, [1, 1]]], #h1_2
[-1, 1, Conv, [1024, 3, 1]],
[-1, 1, Conv, [512, 1, 1]], #h1_4
[-1, 1, Conv, [1024, 3, 1]],
[-2, 1, Conv, [256, 1, 1]], #h2_1
[-1, 1, nn.Upsample, [None, 2, 'nearest']],
[[-1, 8], 1, Concat, [1]], #h2_3
[-1, 1, Bottleneck, [512, False]],
[-1, 1, Bottleneck, [512, False]],
[-1, 1, Conv, [256, 1, 1]],
[-1, 1, Conv, [512, 3, 1]],
[-2, 1, Conv, [128, 1, 1]],
[-1, 1, nn.Upsample, [None, 2, 'nearest']],
[[-1, 6], 1, Concat, [1]], #h3_3
[-1, 1, Bottleneck, [256, False]],
[-1, 2, Bottleneck, [256, False]],
[[27, 22, 15], 1, Detect, [nc, anchors]], #h3_6
]
for the first branch in the headers, the characteristic diagram with the smaller size is originally obtained by the first branch and is responsible for detecting the targets with the larger size, but the modification is to increase the output size and reduce the convolution step sizes of s2 and s4, so that the detection of the targets of the four types of vehicles is influenced, so that compensation is made here, and the convolution step sizes in h1_2 and h1_4 are changed to 2 to be responsible for detecting the four types of vehicles;
for the second branch in the heads, the second branch originally obtains a feature map with medium size, which is responsible for detecting the target with medium size, considering that most of four vehicles in the image are license plate characters of small targets and four vehicles of large targets, even if some vehicles appear in the image in the form of medium size, the proportion of the condition is smaller, so the second branch is modified appropriately to be responsible for the target with medium size and small size, and similarly, when two feature outputs are combined in h2_3, one is the previous layer, and the other is changed into the output of s3 in the backbone, namely [ -1,8] is changed into [ -1,6 ];
for the third branch in the headers, the feature map with the larger size is originally obtained by the third branch, and is responsible for detecting the target with the smaller size, and corresponding to the license plate character in the invention, the reasons of modification are the same as those of the second branch, wherein [ -1,6] in h3_3 is changed into [ -1,4], and the nc assignment 76 in h3_6 represents the total number of categories.
The improved heads structure is therefore:
head:
[[-1, 1, Bottleneck, [1024, False]],#h1_1
[-1, 1, Conv, [512, [1, 2]]], #h1_2
[-1, 1, Conv, [1024, 3, 1]],
[-1, 1, Conv, [512, 1, 2]], #h1_4
[-1, 1, Conv, [1024, 3, 1]],
[-2, 1, Conv, [256, 1, 1]], #h2_1
[-1, 1, nn.Upsample, [None, 2, 'nearest']],
[[-1, 6], 1, Concat, [1]], #h2_3
[-1, 1, Bottleneck, [512, False]],
[-1, 1, Bottleneck, [512, False]],
[-1, 1, Conv, [256, 1, 1]],
[-1, 1, Conv, [512, 3, 1]],
[-2, 1, Conv, [128, 1, 1]],
[-1, 1, nn.Upsample, [None, 2, 'nearest']],
[[-1, 4], 1, Concat, [1]], #h3_3
[-1, 1, Bottleneck, [256, False]],
[-1, 2, Bottleneck, [256, False]],
[[27, 22, 15], 1, Detect, [76, anchors]], #h3_6
]
it is important to note here that, in order to improve or even eliminate the problem of label rewriting to some extent, the feature size means for increasing the output here is another way to avoid this phenomenon. For example, the original output feature map has a size of 13 × 13, all objects in the image are allocated to 13 × 13 grids, and objects with smaller distances are more easily classified into the same grid, but if the size of the output feature map is doubled and the area is increased by 4 times by changing the value of [ -1,6] in h3_3 to [ -1,4] according to the technical means of the above embodiment, the objects in the image are classified into 26 × 26 grids, and the phenomenon that a plurality of objects are classified into one grid is effectively reduced; in addition to the step of increasing the size of the input image mentioned above, the size of the output feature map is increased proportionally, so that the size of the output feature map is further increased by the corresponding multiple (the same ratio is equal to the multiple increased by the input image), and obviously, the situation that a plurality of objects are divided into one grid is greatly improved, and therefore, when no more objects are divided into one grid, the problem of label rewriting can be eliminated and improved.
It can be seen that in view of the design concept of the present invention, in order to solve the first and second problems, the present invention ingeniously proposes a scheme of increasing the size of an input image and increasing the size of an output feature map by designing a network twice, but fundamentally, each anchor can be responsible for multiple targets, even if the sizes of the targets are close to each other, this idea is opposite to the most basic idea of yolo, for this purpose, an ingenious design is made, each anchor is still responsible for the prediction of one target, but the third branch (because the third branch mainly predicts a small target such as a license plate character) allocates three anchors with similar sizes to each grid, and the targets with similar sizes are dispersedly allocated to the three anchors, but not all the anchors, so that the label rewriting phenomenon can be effectively avoided.
Step S33: performing post-processing rewrite
Since the yolov3 model described above outputs all detected objects, but these objects are unordered and the license plate numbers are ordered. Therefore, it is also necessary to perform post-processing on the output target, such as setting each license plate character to be detected and output
Each detected vehicle is
Each of
There are four position coordinates:
，
，
，
are respectively a character
The x coordinate and the y coordinate of the vertex at the upper left corner of the rectangle and the x coordinate and the y coordinate of the fixed point at the lower right corner.
Each one of which is
All have four position coordinates
，
，
，
Are respectively a character
The x coordinate and the y coordinate of the vertex at the upper left corner of the rectangle and the x coordinate and the y coordinate of the fixed point at the lower right corner, if the coordinates are the same
≧ 
≧ 
And is
≧ 
≧ 
Then conclude the wordSymbol
Belonging to vehicles
One of the license plate characters of (1) then for all belonging vehicles
Is a character of
According to which
The sizes of the vehicles are sequenced to obtain the vehicles
A complete and ordered set of license plate numbers.
As shown in fig. 3, an image includes two vehicles, car1 and car2, Q1 is a minimum bounding rectangle of car1, Q2 is a minimum bounding rectangle of car2, each vehicle has 7 license plate characters, Pi represents a minimum bounding rectangle of each character, such as P1, P2.. P7, and each image is input to obtain an array of all objects to be detected (vehicles and license plate characters) in the image, including:
（class，confidence，left，top，right，bottom）；
class is the type of the object, whether it is a car or a character, confidence is confidence, and left, top, right, bottom represent four values of the position of the object. In the above figure (
，
) I.e. the coordinates of the upper left vertex of the minimum bounding rectangle Q1 of vehicle car1, (in the upper graph)
，
) I.e. the coordinates of the vertex of the lower right corner of the minimum bounding rectangle Q1 of vehicle car1, (in the upper graph)
，
) I.e., the coordinates of the top left vertex of the minimum bounding rectangle P1 for the character [ kyo ], and so on, with such coordinates for each vehicle and each character. But these arrays of model outputs (class, confidence, left, top, right, bottom) are unordered, but the license plate numbers themselves are ordered,
if there are 2 vehicles in an image, the trade marks are [ Jing AF0236 ] and [ Su E730V7 ], the model only outputs the information of 2 groups of vehicles and the information of 14 groups of characters:
such as:
（car1，0.83，23，86，435，538），
（car2，confidence，left，top，right，bottom），
such as:
（F，0.92，124，254，131，270）
（6，confidence，left，top，right，bottom）
(Beijing, confidence, left, top, right, bottom)
（7，confidence，left，top，right，bottom）
（0，confidence，left，top，right，bottom）
（A，confidence，left，top，right，bottom）；
Information integration is carried out on the information according to the scheme, so that license plate characters belonging to each vehicle can be obtained, and the characters are ensured to be orderly arranged from left to right.
Therefore, the invention skillfully judges which vehicle target each character target is in order to solve the third problem on the design concept, and then sequences the character targets in the same vehicle target according to the position result of the upper left corner of the character target, thereby solving the problem of sequencing the disordered license plate numbers of each vehicle, and obtaining the ordered license plate numbers belonging to a certain vehicle, thereby realizing the purpose of solving the license plate recognition problem by adopting a single deep learning model and a process.
(II)
In addition, in another preferred embodiment, in order to improve the recognition effect of the model after the secondary design, the license plate recognition method provided by the invention further includes: step S4 training and optimizing the model, specifically, training the post-secondary-design model using the data set collected and labeled in the above embodiment i. In a preferred embodiment, the loss function of the model during the optimization training comprises:
where the first term is the center coordinate error of the box: when the jth anchor box of the ith grid is responsible for a certain real target, the bounding box generated by the anchor box should be compared with the box of the real target, and the center coordinate error is calculated.
Wherein the second term is the width-to-height coordinate error of the box: here, w and h are not set aside, so as to increase the weight of width and height; when the jth anchor box of the ith grid is responsible for a certain real target, the bounding box generated by the anchor box should be compared with the box of the real target, and the width and height errors are calculated.
Both of which are provided with
And the weight of the frame information (including the coordinates of the central point and the width and height) is increased, so that the targets with different sizes can be better identified.
Is the width and height of the real frame.
Wherein the third term and the fourth term are confidence errors: the confidence error is expressed by using cross entropy, and is divided into two terms of an object and an object-free term, because of the cross entropy, coefficients in the two terms are negative, and the fifth term is the same reason.
Where the fifth term is the classification error: when the jth anchor box of the ith grid is responsible for a certain real target, the bounding box generated by the anchor box is used for calculating the classification loss function.
By designing parameters required by training of the existing models of different combinations, such as learning laws, input image sizes, batch _ size, epoch and the like, testing each trained model on a test set, comprehensively comparing test results, if the test results are not satisfactory, reversely deducing a place needing optimization according to the performance of the test set, then repeating the steps, and if the test results are satisfactory, selecting the model with the best effect as a final model.
(III)
On the other hand, the invention also provides a license plate recognition system, wherein, the license plate is recognized by adopting a pure deep learning method in the scheme of the embodiment of the invention, and the final license plate can be obtained from the image or video stream only by using one model from end to end.
For example, in a preferred embodiment, the license plate recognition system includes: the main controller is connected with the main controller to transmit the vehicle image data to the main controller, wherein any one of the license plate identification methods recorded in the first embodiment and the second embodiment is stored in the main controller, so that the main controller can identify the license plate number in the vehicle image data. In the example, the master controller can be made of a Haisi hi3519 core board, and can be conveniently placed in a machine body of a common camera due to small volume, so that a license plate recognition system with a real-time license plate number recognition function can be obtained.
In another preferred embodiment, a license plate recognition system of an online mode can be further constructed, wherein the system includes: the server is connected with the camera to transmit the vehicle image data to the server, wherein any one of the license plate recognition methods described in the first to second embodiments is stored in the server, so that the server can recognize the license plate number in the vehicle image data. In the example, the vehicle image data transmitted by the cameras in various places are identified by the server, so that a large number of images can be rapidly identified in batches, and online real-time identification can be performed, and the identification efficiency is improved.
(IV)
In another aspect of the present invention, a readable storage medium is further provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the license plate recognition method according to any one of the first to second embodiments.
In summary, the license plate recognition method and the system thereof provided by the invention overcome the defects of the prior art, solve the license plate recognition problem through one process by adopting a single deep learning model, and have better recognition speed and accuracy compared with the traditional image processing method or the traditional method combining image processing and deep learning; meanwhile, the identification accuracy of the scheme can be infinitely close to 100 percent along with the increase of the data sets, and the method is not limited by the precision and the speed of the traditional processing method; in addition, the scheme can meet the requirements of detecting and positioning different types of vehicles, license plates with different colors and numbers of different license plates through one model, and special processing is not required to be carried out on special license plate colors, so that the method has remarkable progress compared with the prior art.
The preferred embodiments of the invention disclosed above are intended to be illustrative only. The preferred embodiments are not intended to be exhaustive or to limit the invention to the precise embodiments disclosed. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, to thereby enable others skilled in the art to best utilize the invention. The invention is limited only by the claims and the full scope and equivalents thereof, and any modification, equivalent replacement, or improvement made within the spirit and principle of the invention should be included in the protection scope of the invention.
It will be appreciated by those skilled in the art that, in addition to implementing the system, apparatus and various modules thereof provided by the present invention in the form of pure computer readable program code, the same procedures may be implemented entirely by logically programming method steps such that the system, apparatus and various modules thereof provided by the present invention are implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system, the device and the modules thereof provided by the present invention can be considered as a hardware component, and the modules included in the system, the device and the modules thereof for implementing various programs can also be considered as structures in the hardware component; modules for performing various functions may also be considered to be both software programs for performing the methods and structures within hardware components.
In addition, all or part of the steps of the method according to the above embodiments may be implemented by a program instructing related hardware, where the program is stored in a storage medium and includes several instructions to enable a single chip, a chip, or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In addition, any combination of various different implementation manners of the embodiments of the present invention is also possible, and the embodiments of the present invention should be considered as disclosed in the embodiments of the present invention as long as the combination does not depart from the spirit of the embodiments of the present invention.
Claims (9)
1. A license plate recognition method is characterized by comprising the following steps:
s1 preparing a data set: collecting vehicle image data, and labeling each character in the vehicle and the license plate image data thereof according to a first rule;
s2, selecting a basic model and carrying out secondary design, wherein the secondary design step comprises the following steps: the increasing the size of the input image data of the model includes: processing the area near the license plate number by a data enhancement processing means to increase the resolution of the local area near the license plate number; performing cluster analysis on the minimum circumscribed rectangle of the object labeled in the step S1 to obtain anchors of the data set and setting branches in the headers of the model, which are responsible for predicting license plate characters, to increase the size of the output characteristic diagram of the model; and carrying out post-processing and rewriting on the model, and sequencing the disordered recognition result output by the model according to a second rule.
2. The license plate recognition method of claim 1, wherein the first rule comprises: the vehicle image data is sorted according to a preset proportion into: the method comprises a training set, a verification set and a test set, wherein each character in image data of vehicles and license plates thereof in the training set and the verification set is labeled.
3. The license plate recognition method of claim 1, wherein the labeling step comprises: labeling a vehicle type in the image data and labeling each character in a license plate of a corresponding vehicle, and the labels include: the type of each character in the vehicle and the license plate thereof, and the coordinates of the fixed point at the upper left corner and the fixed point at the lower right corner of the minimum circumscribed rectangle of each character in the vehicle and the corresponding license plate thereof.
4. The license plate recognition method of claim 1, wherein the secondary design step further comprises: setting a detection head of the model, wherein the steps comprise: making the first branch in the detection head compensate, modifying convolution step sizes in h1_2 and h1_4 and making the convolution step sizes responsible for detection of the vehicle types; the second branch in the detection head is responsible for detecting the vehicle type and the license plate characters; let the third branch in the inspection head be responsible for the inspection of license plate characters, where h1_2 means the layer 2 structure of the first inspection head of yolov3, and h1_4 means the layer 4 structure of the first inspection head of yolov 3.
5. The license plate recognition method of claim 1, wherein the second rule comprises: judging which vehicle target is at each license plate character target position; and sorting the character targets in the same vehicle target according to the position result of the upper left corner of the character targets.
6. The license plate recognition method of claim 1, further comprising a step of S3 model training and optimization, the step comprising: training the secondarily designed model in the step S2 by preset hyper-parameter combinations; testing each trained model on a test set; comparing the test results to optimize; and repeating the steps until the model reaches the standard.
7. A license plate recognition method is characterized by comprising the following steps:
s1 preparing a data set: collecting vehicle image data, and labeling each character in the vehicle and the license plate image data thereof according to a first rule; wherein the labeling step comprises: labeling a vehicle type in the image data and labeling each character in a license plate of a corresponding vehicle, and the labels include: the type of each character in the vehicle and the license plate thereof, and the coordinates of the fixed point at the upper left corner and the fixed point at the lower right corner of the minimum circumscribed rectangle of each character in the vehicle and the corresponding license plate thereof,
s2, selecting a basic model and carrying out secondary design, wherein the secondary design step comprises the following steps: the increasing the size of the input image data of the model includes: processing the area near the license plate number by a data enhancement processing means to increase the resolution of the local area near the license plate number; performing clustering analysis on the minimum circumscribed rectangular frame of the object labeled in the step S1 to obtain anchors of the data set, setting branches in the headers of the model, which are responsible for predicting license plate characters, to increase the size of the output characteristic diagram of the model, and outputting a disordered recognition result;
s3 post-processing the model to rewrite: sequencing the disordered results output in the step S2, and setting each license plate character to be detected and output as
Each vehicle of which the kind is detected is
Each of which
Four position coordinates are set:
，
，
，
are respectively a character
The x coordinate and the y coordinate of the vertex of the upper left corner of the rectangle and the x coordinate and the y coordinate of the fixed point of the lower right corner are positioned; each one of which is
Setting four position coordinates
，
，
，
Are respectively a character
The x coordinate and the y coordinate of the vertex at the upper left corner of the rectangle and the x coordinate and the y coordinate of the fixed point at the lower right corner, if the coordinates are the same
≧ 
≧
And is
≧ 
≧
Then the character is judged
Belonging to vehicles
One of the license plate characters of (1) then for all belonging vehicles
Is a character of
According to which
Is sorted by size.
8. A license plate recognition system characterized by comprising: the main controller, the camera is connected with main controller to the transmission vehicle image data to main controller, wherein have the license plate of any claim 1 to 7 in the storage in the main controller to the license plate number in the vehicle image data of execution discernment of main controller.
9. A readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the license plate recognition method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111288251.0A CN113723408B (en) | 2021-11-02 | 2021-11-02 | License plate recognition method and system and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111288251.0A CN113723408B (en) | 2021-11-02 | 2021-11-02 | License plate recognition method and system and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113723408A CN113723408A (en) | 2021-11-30 |
| CN113723408B true CN113723408B (en) | 2022-02-25 |
Family
ID=78686440
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111288251.0A Active CN113723408B (en) | 2021-11-02 | 2021-11-02 | License plate recognition method and system and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113723408B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116310390B (en) * | 2023-05-17 | 2023-08-18 | 上海仙工智能科技有限公司 | Visual detection method and system for hollow target and warehouse management system |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105160292A (en) * | 2015-07-07 | 2015-12-16 | 深圳市捷顺科技实业股份有限公司 | Vehicle identification recognition method and system |
| CN107729801A (en) * | 2017-07-11 | 2018-02-23 | 银江股份有限公司 | A kind of vehicle color identifying system based on multitask depth convolutional neural networks |
| CN107730904A (en) * | 2017-06-13 | 2018-02-23 | 银江股份有限公司 | Multitask vehicle driving in reverse vision detection system based on depth convolutional neural networks |
| CN108171136A (en) * | 2017-12-21 | 2018-06-15 | 浙江银江研究院有限公司 | A kind of multitask bayonet vehicle is to scheme to search the system and method for figure |
| CN108241822A (en) * | 2016-12-23 | 2018-07-03 | 杭州海康威视数字技术股份有限公司 | A kind of recognition methods of type of vehicle and device |
| CN109165643A (en) * | 2018-08-21 | 2019-01-08 | 浙江工业大学 | A kind of licence plate recognition method based on deep learning |
| CN110222593A (en) * | 2019-05-18 | 2019-09-10 | 四川弘和通讯有限公司 | A kind of vehicle real-time detection method based on small-scale neural network |
| CN111079744A (en) * | 2019-12-06 | 2020-04-28 | 鲁东大学 | Intelligent vehicle license plate identification method and device suitable for complex illumination environment |
| CN111523415A (en) * | 2020-04-13 | 2020-08-11 | 北京巨视科技有限公司 | Image-based two-passenger one-dangerous vehicle detection method and device |
| CN111680556A (en) * | 2020-04-29 | 2020-09-18 | 平安国际智慧城市科技股份有限公司 | Method, device and equipment for identifying vehicle type at traffic gate and storage medium |
| CN112308061A (en) * | 2020-11-20 | 2021-02-02 | 珠海大横琴科技发展有限公司 | License plate character sorting method, recognition method and device |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104217217B (en) * | 2014-09-02 | 2017-06-30 | 武汉睿智视讯科技有限公司 | A kind of vehicle mark object detecting method and system based on two layers of classified |
| CN106570477B (en) * | 2016-10-28 | 2019-07-16 | 中科视语(北京)科技有限公司 | Vehicle cab recognition model building method and model recognizing method based on deep learning |
| CN112966631A (en) * | 2021-03-19 | 2021-06-15 | 浪潮云信息技术股份公司 | License plate detection and identification system and method under unlimited security scene |
| CN113205026B (en) * | 2021-04-26 | 2022-08-16 | 武汉大学 | Improved vehicle type recognition method based on fast RCNN deep learning network |
-
2021
- 2021-11-02 CN CN202111288251.0A patent/CN113723408B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105160292A (en) * | 2015-07-07 | 2015-12-16 | 深圳市捷顺科技实业股份有限公司 | Vehicle identification recognition method and system |
| CN108241822A (en) * | 2016-12-23 | 2018-07-03 | 杭州海康威视数字技术股份有限公司 | A kind of recognition methods of type of vehicle and device |
| CN107730904A (en) * | 2017-06-13 | 2018-02-23 | 银江股份有限公司 | Multitask vehicle driving in reverse vision detection system based on depth convolutional neural networks |
| CN107729801A (en) * | 2017-07-11 | 2018-02-23 | 银江股份有限公司 | A kind of vehicle color identifying system based on multitask depth convolutional neural networks |
| CN108171136A (en) * | 2017-12-21 | 2018-06-15 | 浙江银江研究院有限公司 | A kind of multitask bayonet vehicle is to scheme to search the system and method for figure |
| CN109165643A (en) * | 2018-08-21 | 2019-01-08 | 浙江工业大学 | A kind of licence plate recognition method based on deep learning |
| CN110222593A (en) * | 2019-05-18 | 2019-09-10 | 四川弘和通讯有限公司 | A kind of vehicle real-time detection method based on small-scale neural network |
| CN111079744A (en) * | 2019-12-06 | 2020-04-28 | 鲁东大学 | Intelligent vehicle license plate identification method and device suitable for complex illumination environment |
| CN111523415A (en) * | 2020-04-13 | 2020-08-11 | 北京巨视科技有限公司 | Image-based two-passenger one-dangerous vehicle detection method and device |
| CN111680556A (en) * | 2020-04-29 | 2020-09-18 | 平安国际智慧城市科技股份有限公司 | Method, device and equipment for identifying vehicle type at traffic gate and storage medium |
| CN112308061A (en) * | 2020-11-20 | 2021-02-02 | 珠海大横琴科技发展有限公司 | License plate character sorting method, recognition method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113723408A (en) | 2021-11-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2017215622A1 (en) | Object segmentation method and apparatus and computing device | |
| CN111784685A (en) | Power transmission line defect image identification method based on cloud edge cooperative detection | |
| US20210192227A1 (en) | Method and apparatus for detecting parking space usage condition, electronic device, and storage medium | |
| CN107358596A (en) | A kind of car damage identification method based on image, device, electronic equipment and system | |
| CN112016605B (en) | Target detection method based on corner alignment and boundary matching of bounding box | |
| CN106557778A (en) | Generic object detection method and device, data processing equipment and terminal device | |
| CN110264444B (en) | Damage detection method and device based on weak segmentation | |
| CN109993138A (en) | A kind of car plate detection and recognition methods and device | |
| CN111797829A (en) | License plate detection method and device, electronic equipment and storage medium | |
| CN111767927A (en) | Lightweight license plate recognition method and system based on full convolution network | |
| CN110334709B (en) | License plate detection method based on end-to-end multi-task deep learning | |
| CN113688652A (en) | Method and device for processing abnormal driving behaviors | |
| CN111738036B (en) | Image processing method, device, equipment and storage medium | |
| CN104615986A (en) | Method for utilizing multiple detectors to conduct pedestrian detection on video images of scene change | |
| CN113076804B (en) | Target detection method, device and system based on YOLOv4 improved algorithm | |
| EP2447884A1 (en) | Method for detecting and recognising an object in an image, and an apparatus and a computer program therefor | |
| CN113723408B (en) | License plate recognition method and system and readable storage medium | |
| CN112819748A (en) | Training method and device for strip steel surface defect recognition model | |
| CN115984537A (en) | Image processing method and device and related equipment | |
| CN114519819B (en) | Remote sensing image target detection method based on global context awareness | |
| CN111914845A (en) | Character layering method and device in license plate and electronic equipment | |
| CN115019133A (en) | Method and system for detecting weak target in image based on self-training and label anti-noise | |
| CN114820922A (en) | Automatic driving effective static scene construction method and system based on complexity | |
| CN112184679A (en) | YOLOv 3-based wine bottle flaw automatic detection method | |
| CN112580624A (en) | Method and device for detecting multidirectional text area based on boundary prediction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |