CN112580629A - License plate character recognition method based on deep learning and related device - Google Patents
License plate character recognition method based on deep learning and related device Download PDFInfo
- Publication number
- CN112580629A CN112580629A CN202011541596.8A CN202011541596A CN112580629A CN 112580629 A CN112580629 A CN 112580629A CN 202011541596 A CN202011541596 A CN 202011541596A CN 112580629 A CN112580629 A CN 112580629A
- Authority
- CN
- China
- Prior art keywords
- license plate
- image
- characters
- character
- neural network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 76
- 238000013135 deep learning Methods 0.000 title claims abstract description 46
- 238000003062 neural network model Methods 0.000 claims abstract description 56
- 238000010586 diagram Methods 0.000 claims abstract description 41
- 238000012545 processing Methods 0.000 claims abstract description 28
- 230000011218 segmentation Effects 0.000 claims abstract description 11
- 238000000605 extraction Methods 0.000 claims abstract description 8
- 239000000463 material Substances 0.000 claims description 33
- 238000012937 correction Methods 0.000 claims description 27
- 238000007781 pre-processing Methods 0.000 claims description 20
- 238000012549 training Methods 0.000 claims description 8
- 238000002372 labelling Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 description 5
- 238000001514 detection method Methods 0.000 description 4
- 238000011176 pooling Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000007796 conventional method Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 238000003709 image segmentation Methods 0.000 description 2
- 238000009434 installation Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000009827 uniform distribution Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
- G06V20/63—Scene text, e.g. street names
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/24—Aligning, centring, orientation detection or correction of the image
- G06V10/243—Aligning, centring, orientation detection or correction of the image by compensating for image skew or non-uniform image deformations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/28—Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/14—Image acquisition
- G06V30/148—Segmentation of character regions
- G06V30/153—Segmentation of character regions using recognition of characters or words
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
- G06V20/625—License plates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
- Character Input (AREA)
- Character Discrimination (AREA)
Abstract
The application discloses a license plate character recognition method, a license plate character recognition system, a license plate character recognition device and a computer readable storage medium based on deep learning, which are used for improving the success rate of license plate character segmentation. The method comprises the following steps: acquiring a license plate image of the vehicle, wherein the license plate image comprises a license plate area of the vehicle; inputting the license plate image into a pre-configured neural network model, wherein the neural network model is used for extracting character features in the license plate image and generating a thermodynamic diagram according to the character features, and the thermodynamic diagram is used for highlighting the positions of characters in the license plate image; acquiring the thermodynamic diagram output by the neural network model; processing the thermodynamic diagram by an MSER region feature extraction method, and segmenting a character region containing the characters to obtain target license plate characters; and identifying the segmented target license plate characters, and obtaining an identification result.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to a method, a system, an apparatus, and a computer-readable storage medium for recognizing license plate characters based on deep learning.
Background
With the continuous development of information technology, license plate recognition equipment has been widely applied to areas such as parking lots, urban roads, highways and the like for automatic capturing and recognizing of license plates of vehicles. In the past years, the license plate recognition technology is rapidly developed, and the recognition rate of the license plate is improved, so that the pure license plate recognition and the unattended scheme become feasible.
In the scheme provided by the prior art, license plate recognition generally comprises license plate detection, namely, searching a license plate region in an image, license plate character segmentation, namely, segmenting a single character in the detected license plate region, and license plate character recognition, namely, recognizing the single license plate character. However, in practical applications, due to the influence of the installation angle of the device or illumination, the method may cause the license plate characters to be easily segmented incorrectly. Therefore, the method has low recognition rate of the license plate characters.
Disclosure of Invention
In order to solve the technical problem, the application provides a license plate character recognition method based on deep learning, which is used for improving the recognition rate of license plate characters.
The application provides a license plate character recognition method based on deep learning in a first aspect, and the method comprises the following steps:
acquiring a license plate image of the vehicle, wherein the license plate image comprises a license plate area of the vehicle;
inputting the license plate image into a pre-configured neural network model, wherein the neural network model is used for extracting character features in the license plate image and generating a thermodynamic diagram according to the character features, and the thermodynamic diagram is used for highlighting the positions of characters in the license plate image;
acquiring the thermodynamic diagram output by the neural network model;
processing the thermodynamic diagram by an MSER region feature extraction method, and segmenting a character region containing the characters to obtain target license plate characters;
and identifying the segmented target license plate characters, and obtaining an identification result.
Optionally, before the step of recognizing the segmented target license plate character and obtaining a recognition result, the method further includes:
and scaling the target license plate characters to a preset size and carrying out histogram equalization on the scaled target license plate characters.
Optionally, before the inputting the license plate image into the preconfigured neural network model, the method further includes:
and performing first preprocessing on the license plate image, wherein the first preprocessing at least comprises one of horizontal correction, vertical correction or perspective correction.
Optionally, the neural network model is obtained by training according to the following method:
acquiring a plurality of material images, wherein each material image comprises a region of a license plate of a vehicle;
carrying out binarization processing on each material image to obtain a corresponding sample image;
marking each pixel point in each sample image, wherein a first label is marked for the pixel point of a character area in the license plate, and a second label is marked for the pixel point outside the character area, so as to obtain a corresponding label image;
inputting the sample image and the corresponding label image into an initialized neural network model;
and calculating the characteristic loss difference between the material image and the label image, and dynamically adjusting the initialized neural network model according to the characteristic loss difference until the converged neural network model is obtained.
Optionally, before the binarizing processing is performed on each of the material images, the method further includes:
and adjusting the size of the material image to enable the aspect ratio of the material image to be 3: 1.
Optionally, before labeling each pixel point in each sample image, the method further includes:
judging whether the characters of the license plate in the sample image are complete:
if not, filling the character incomplete part and/or removing the character redundant part.
Optionally, before the binarizing processing is performed on each of the material images, the method further includes: and performing second preprocessing on the sample image, wherein the second preprocessing at least comprises one of horizontal correction, vertical correction or perspective correction.
The second aspect of the present application provides a license plate character recognition system based on deep learning, the system including:
the first acquisition unit is used for acquiring a license plate image of the vehicle, wherein the license plate image comprises a license plate area of the vehicle;
the input unit is used for inputting the license plate image into a pre-configured neural network model;
the second acquisition unit is used for acquiring a thermodynamic diagram output by the neural network model, and the thermodynamic diagram is used for representing the positions of characters in the license plate image;
the segmentation unit is used for processing the thermodynamic diagram by an MSER region feature extraction method and segmenting a character region containing the characters to obtain target license plate characters;
and the recognition unit is used for recognizing the segmented target license plate characters and obtaining a recognition result.
The third aspect of the present application provides a license plate character recognition device based on deep learning, the device including:
the device comprises a processor, a memory, an input and output unit and a bus;
the processor is connected with the memory, the input and output unit and the bus;
the memory holds a program that the processor calls to perform the method of any of the first aspect and the first aspect.
A fourth aspect of the present application provides a computer readable storage medium having a program stored thereon, which when executed on a computer performs the method of any one of the first aspect and the first aspect.
According to the technical scheme, the method has the following advantages:
according to the license plate character recognition method based on deep learning, a terminal can obtain a license plate image of a vehicle, the license plate image is input into a pre-configured neural network model, the neural network model can extract character features in the license plate image and output thermodynamic diagrams used for highlighting the characters, then target license plate characters are segmented through binarization processing, accuracy of segmentation of the characters in the license plate can be improved, and finally the segmented target license plate characters are recognized.
Drawings
In order to more clearly illustrate the technical solutions in the present application, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is a schematic flowchart of an embodiment of a license plate character recognition method based on deep learning according to the present application;
fig. 2 is a schematic flowchart of another embodiment of a license plate character recognition method based on deep learning according to the present application;
fig. 3 is a schematic flowchart of another embodiment of a license plate character recognition method based on deep learning according to the present application;
fig. 4 is a schematic flowchart of another embodiment of a license plate character recognition method based on deep learning according to the present application;
fig. 5 is a schematic structural diagram of an embodiment of a license plate character recognition system based on deep learning according to the present application;
fig. 6 is a schematic structural diagram of an embodiment of a license plate character recognition device based on deep learning according to the present application.
Detailed Description
In the scheme provided by the prior art, license plate recognition generally comprises license plate detection, namely, searching a license plate region in an image, license plate character segmentation, namely, segmenting a single character in the detected license plate region, and license plate character recognition, namely, recognizing the single license plate character. However, in practical applications, due to the influence of the installation angle of the device or illumination, the method may cause the license plate characters to be easily segmented incorrectly. Therefore, the method has low recognition rate of the license plate characters.
Based on the above, the application provides a license plate character recognition method based on deep learning, which is used for improving the success rate of license plate character segmentation and further improving the recognition rate of a license plate.
It should be noted that the license plate character recognition method based on deep learning provided by the present application may be applied to a terminal, a system, or a server, for example, the terminal may be an embedded device, a smart phone or a computer, a tablet computer, a smart television, a smart watch, a portable computer terminal, or a fixed terminal such as a desktop computer. For convenience of explanation, the terminal is taken as an execution subject for illustration in the present application.
Referring to fig. 1, fig. 1 is a schematic flowchart illustrating an embodiment of a license plate character recognition method based on deep learning according to the present application, where the license plate character recognition method based on deep learning includes:
101. acquiring a license plate image of a vehicle, wherein the license plate image comprises a license plate area of the vehicle;
the terminal acquires a license plate image containing a license plate area. The license plate image may be a color image. The license plate image containing the license plate region mentioned or discussed in the application refers to the region containing the license plate searched in the image. The search for the region containing the license plate may use a conventional method or a detection method based on deep learning, which is not limited herein.
102. Inputting a license plate image into a pre-configured neural network model, wherein the neural network model is used for extracting character features in the license plate image and generating a thermodynamic diagram according to the character features, and the thermodynamic diagram is used for highlighting the positions of characters in the license plate image;
in the method provided by the application, the terminal inputs the license plate graph into a preconfigured neural network model, the neural network model can be used for extracting character features in the license plate image and generating a thermodynamic diagram according to the character features, and the thermodynamic diagram refers to a mode of highlighting in a highlight or highlight mode to represent the character features, especially the positions of characters. The character features are expressed by means of thermodynamic diagrams, so that the characters have obvious boundaries and distinction in the image, and the subsequent character segmentation is facilitated.
The neural network model used in the present application may be an improvement over the U-Net model. The traditional U-Net is a semantic segmentation network proposed for medical image segmentation. The structure contains a systolic path for capturing semantics and a symmetric extended path for fine positioning. The network can train an end-to-end network by using a small amount of data and has good performance. The traditional U-Net is directed at medical image segmentation and has high precision requirement. The invention only needs to highlight characters on the license plate area, inhibit background and remove interference on the license plate, thereby simplifying U-Net in design. Through simplification, the network has high speed and can be applied to terminal equipment. The conventional U-net uses input images with uniform width and height, but the license plate is rectangular and has an aspect ratio of about 3 to 1, so that the aspect ratio of the input images is modified in the present application and an aspect ratio of 3 to 1 is used.
103. Acquiring a thermodynamic diagram output by a neural network model;
in the method provided by the application, after the character features are extracted by the neural network model, a thermodynamic diagram is output, and the terminal acquires the thermodynamic diagram and carries out corresponding processing.
104. Processing the thermodynamic diagram by an MSER region feature extraction method, and segmenting a character region containing characters to obtain target license plate characters;
specifically, the characters may be extracted by an MSER (maximum stable extreme regions) method, which is an algorithm that can select an appropriate threshold value for an image to obtain connected components, and detect stationarity of the connected components to obtain a final stationary region. The present application may use the MSER to extract regions of the thermodynamic diagram that are displayed as license plate characters.
The binarization processing of the image is to set the gray value of a point on the image to be 0 or 255, that is, to make the whole image show obvious black and white effect. That is, a gray scale image with 256 brightness levels is selected by a proper threshold value to obtain a binary image which can still reflect the whole and local features of the image.
105. And identifying the segmented target license plate characters, and obtaining an identification result.
And the terminal identifies the segmented target license plate characters.
According to the license plate character recognition method based on deep learning, a terminal can obtain a license plate image of a vehicle, the license plate image is input into a pre-configured neural network model, the neural network model can extract character features in the license plate image and output thermodynamic diagrams used for highlighting the characters, then target license plate characters are segmented through binarization processing, accuracy of segmentation of the characters in the license plate can be improved, finally the segmented target license plate characters are recognized, the target license plate characters refer to images of the segmented characters, the characters in the images can be obtained by recognizing the images through the terminal, recognition results are obtained, and the recognition results refer to license plate numbers of the vehicle.
In practical applications, before inputting a license plate image into a neural network model, the license plate image may be preprocessed to improve accuracy of processing the license plate image and improve a recognition rate of characters of the license plate, which will be described in detail below with reference to the accompanying drawings.
Referring to fig. 2, fig. 2 is a schematic flowchart illustrating an embodiment of a license plate character recognition method based on deep learning provided in the present application, where the license plate character recognition method based on deep learning includes:
201. acquiring a license plate image of a vehicle, wherein the license plate image comprises a license plate area of the vehicle;
the terminal acquires a license plate image containing a license plate area. The license plate image may be a color image. The license plate image containing the license plate region mentioned or discussed in the application refers to the region containing the license plate searched in the image. The search for the region containing the license plate may use a conventional method or a detection method based on deep learning, which is not limited herein.
202. Performing first preprocessing on the license plate image, wherein the first preprocessing at least comprises one of horizontal correction, vertical correction or perspective correction;
in practical application, under the influence of a shooting angle, a license plate area in a license plate image may not be a complete front view, and has a certain bias, in order to improve the accuracy of processing the license plate image by a neural network model, a first preprocessing operation can be performed on the license plate image, the first preprocessing operation can be one of horizontal correction, vertical correction or perspective correction, and finally the angle of the license plate image is reasonable, so that the success rate of finally recognizing characters is improved.
203. Inputting a license plate image into a pre-configured neural network model, wherein the neural network model is used for extracting character features in the license plate image and generating a thermodynamic diagram according to the character features, and the thermodynamic diagram is used for highlighting the positions of characters in the license plate image;
204. acquiring a thermodynamic diagram output by a neural network model;
205. processing the thermodynamic diagram by an MSER region feature extraction method, and segmenting a character region containing characters to obtain target license plate characters;
206. Scaling the target license plate characters to a preset size and carrying out histogram equalization on the scaled target license plate characters;
and the terminal zooms the target license plate characters to enable the target license plate characters to conform to a preset size, such as 32 pixels by 32 pixels, so as to regulate the size of each target license plate character. And then, the Histogram of the zoomed target license plate character is equalized, namely, the Histogram Equalization is carried out on the target license plate character, the Histogram Equalization (Histogram Equalization) is a method for enhancing the Image Contrast (Image Contrast), and the main idea is to change the Histogram distribution of a pair of images into approximately uniform distribution so as to enhance the Image Contrast. The characters of the target license plate are more prominent, so that the recognition success rate is increased.
207. And identifying the segmented target license plate characters, and obtaining an identification result.
And the terminal identifies the segmented target license plate characters.
In practical applications, before implementing the method provided by the present application, a neural network model should be constructed and trained, and a specific model training method will be described in detail below with reference to the accompanying drawings.
Referring to fig. 3, fig. 3 is a schematic flowchart illustrating an embodiment of a license plate character recognition method based on deep learning provided in the present application, where the license plate character recognition method based on deep learning includes:
the neural network model mentioned or discussed in the method provided by the application is obtained by training through the following method:
301. acquiring a plurality of material images, wherein each material image comprises a region of a license plate of a vehicle;
the terminal acquires a plurality of material images, each material image comprises an area of a license plate of a vehicle, the image can be colorful, the material image can be adjusted to be 192x 64 pixels in size, and the color image comprises three channels of RGB (red, green and blue), so that 3 feature maps (feature maps) are provided, namely a red feature map, a green feature map and a blue feature map.
302. Carrying out binarization processing on each material image to obtain a corresponding sample image;
the terminal carries out binarization processing on each material image, clear character areas and non-character areas such as background areas or license plate borders can be obtained after binarization processing, sample images are obtained, the sample images can be gray level images, and each material image corresponds to one sample image.
303. Marking each pixel point in each sample image, wherein a first label is marked for the pixel point in the character area in the license plate, and a second label is marked for the pixel point outside the character area to obtain a corresponding label image;
in the process of model training, in order to continuously and dynamically update the model, the character region in the image needs to be continuously identified, so that each pixel point in the sample image can be labeled, the pixel in the character region in the sample image is labeled as a first label, for example, 1, the pixel outside the character region is labeled as a second label, for example, 0, the label can be at the pixel level, that is, each pixel point is labeled with one label, and finally, a label image carrying the label is obtained, and each sample image has one corresponding label image.
304. Inputting the sample image and the corresponding label image into an initialized neural network model;
the terminal can input a certain number of sample images and corresponding label images into the initial neural network model according to actual needs, and the initial neural network model is an untrained network model with initialization parameters and can be an initialization model constructed in advance.
305. And calculating the characteristic loss difference between the material image and the label image, and dynamically adjusting the initialized neural network model according to the characteristic loss difference until a converged neural network model is obtained.
After the label image and the sample image are input into the initial neural network model, the model can be trained, in the training process, the neural network model can detect characters and a background on a graph, compare the characters and the background with the label image, calculate loss (loss) of the characters and the background, dynamically adjust parameters of the neural network model, and finally obtain the converged neural network model.
The method of the present application will be further illustrated by way of example below:
1. the input image is 192 × 64 pixels in size, and contains three channels of RGB, i.e., 3 feature maps. After the 1 st convolutional layer (Conv1), the size of the convolutional kernel is 3x 3, stride is 1, and pad is 1, resulting in 8 feature maps of 192x 64. Note that fig. 1 represents 3 feature maps using FM ═ 3, and so on.
2. Through the 1 st Pooling layer (Pooling1, window size 2x 2), 8 feature maps of 96x 32 are output.
3. After passing through the 2 nd convolutional layer (Conv2), the convolutional kernel size is 3x 3, stride is 1, and pad is 1, resulting in 16 feature maps of 96x 32.
4. Through the 2 nd Pooling layer (Pooling2, window size 2x 2), 16 feature maps of 48x 16 are output.
5. After passing through the 3 rd convolutional layer (Conv3), the size of the convolutional kernel is 3x 3, stride is 1, and pad is 1, resulting in 32 feature maps of 48x 16.
6. Through the 3 rd Pooling layer (Pooling3, window size 2x 2), 32 feature maps of 24x 8 are output.
7. After the 4 th convolutional layer (Conv4), the convolutional kernel size is 3x 3, stride 1, and pad 1, resulting in 64 feature maps of 24x 8.
8. Through the 4 th Pooling layer (Pooling4, window size 2x 2), 64 feature maps of 12x 4 are output.
9. After the 5 th convolutional layer (Conv5), the size of the convolutional kernel is 3x 3, stride is 1, and pad is 1, resulting in 128 feature maps of 12x 4.
10. Through the 1 st UpSampling layer (UpSampling1, window size 2x 2), 128 feature maps of 24x 8 are output.
11. After passing through the 6 th convolutional layer (Conv6), the convolutional kernel size is 3x 3, stride is 1, and pad is 1, resulting in 64 feature maps of 24x 8. The 64 feature maps of 24 × 8 obtained from this layer are merged with the 64 feature maps of 24 × 8 obtained from the 4 th convolutional layer (Conv4) to obtain 128 feature maps of 24 × 8.
12. After passing through the 7 th convolutional layer (Conv7), the convolutional kernel size is 3x 3, stride is 1, and pad is 1, resulting in 64 feature maps of 24x 8.
13. Passing through the 2 nd UpSampling layer (UpSampling2, window size 2x 2), 64 feature maps of 48x 16 are output.
14. After passing through the 8 th convolutional layer (Conv8), the convolutional kernel size is 3x 3, stride is 1, and pad is 1, resulting in 32 feature maps of 48x 16. The 32 feature maps of 48 × 16 obtained from this layer are merged with the 32 feature maps of 48 × 16 obtained from the 3 rd convolutional layer (Conv3) to obtain 64 feature maps of 48 × 16.
15. After the 9 th convolutional layer (Conv9), the convolutional kernel size is 3x 3, stride is 1, and pad is 1, resulting in 32 feature maps of 48x 16.
16. Passing through the 3 rd UpSampling layer (UpSampling3, window size 2x 2), 32 feature maps of 96x 32 are output.
17. After the 10 th convolutional layer (Conv10), the size of the convolutional kernel is 3x 3, stride is 1, and pad is 1, resulting in 16 feature maps of 96x 32. The 16 feature maps of 96 × 32 obtained from this layer are merged with the 16 feature maps of 96 × 32 obtained from the 2 nd convolutional layer (Conv2) to obtain 32 feature maps of 96 × 32.
18. After the 11 th convolutional layer (Conv11), the size of the convolutional kernel is 3x 3, stride is 1, and pad is 1, resulting in 16 feature maps of 96x 32.
19. Passing through the 4 th UpSampling layer (UpSampling4, window size 2x 2), 16 feature maps of 192x 64 are output.
20. After the 12 th convolutional layer (Conv12), the size of the convolutional kernel is 3x 3, stride is 1, and pad is 1, resulting in 8 feature maps of 192x 64. The 8 feature maps of 192 × 64 are merged with the 8 feature maps of 192 × 64 obtained by passing through the 1 st convolutional layer (Conv1) to obtain 16 feature maps of 192 × 64.
21. After passing through the 13 th convolutional layer (Conv13), the convolutional kernel size is 3x 3, stride is 1, and pad is 1, resulting in 8 feature maps of 192x 64.
22. After passing through the Output convolution layer (Output Conv), the size of the convolution kernel is 3x 3, stride is 1, and pad is 1, so that 2 feature maps of 192x 64 are obtained. These two are the thermodynamic diagrams output by the U-net, one showing the location of the characters and one showing the location of the background.
Referring to fig. 4, fig. 4 is a schematic flowchart illustrating an embodiment of a license plate character recognition method based on deep learning provided in the present application, where the license plate character recognition method based on deep learning includes:
401. acquiring a plurality of material images, wherein each material image comprises a region of a license plate of a vehicle;
402. adjusting the size of the material image to enable the length-width ratio of the material image to be 3: 1;
the image size input in the stage of detecting and identifying the license plate may be an aspect ratio of 3:1, for example, 192 pixels × 64 pixels, or the image size input in the stage of training the model may be an aspect ratio of 3:1, for example, 192 pixels × 64 pixels.
403. Performing second preprocessing on the sample image, wherein the second preprocessing at least comprises one of horizontal correction, vertical correction or perspective correction;
the second preprocessing is carried out on the sample image, in practical application, the sample image is influenced by a shooting angle, a license plate area in the license plate image is possibly not a complete front view, certain bias exists, in order to improve the accuracy of the neural network model for processing the license plate image, the second preprocessing operation can be carried out on the license plate image, the second preprocessing can be one of horizontal correction, vertical correction or perspective correction, and finally the angle of the license plate image is reasonable, so that the accuracy of training is improved.
404. Carrying out binarization processing on each material image to obtain a corresponding sample image;
the terminal carries out binarization processing on each material image, clear character areas and non-character areas such as background areas or license plate borders can be obtained after binarization processing, sample images are obtained, the sample images can be gray level images, and each material image corresponds to one sample image.
405. Judging whether the characters of the license plate in the sample image are complete: if not, go to step 406, if yes, go to step 407;
the terminal judges whether the license plate characters in the sample image are complete or not,
406. filling in the character incomplete part and/or removing the character redundant part.
After the image after binarization is processed, the complete sample image only contains characters and a background, so that an incorrect image after binarization needs to be processed, the terminal judges whether the characters are complete, incomplete parts need to be supplemented or redundant parts need to be removed if the characters are incomplete, for example, a frame is wiped off, and excessive parts need to be removed if the characters adhere to partial backgrounds after binarization. After binarization, if a part of the character is lost, the character needs to be supplemented.
407. Marking each pixel point in each sample image, wherein a first label is marked for the pixel point in the character area in the license plate, and a second label is marked for the pixel point outside the character area to obtain a corresponding label image;
408. inputting the sample image and the corresponding label image into an initialized neural network model;
409. and calculating the characteristic loss difference between the material image and the label image, and dynamically adjusting the initialized neural network model according to the characteristic loss difference until a converged neural network model is obtained.
The license plate character recognition method based on deep learning provided in the present application is explained in detail above, and the license plate character recognition system based on deep learning, the license plate character recognition device based on deep learning, and the storage medium in the present application are explained below with reference to the accompanying drawings.
Referring to fig. 5, fig. 5 is a schematic structural diagram of an embodiment of a license plate character recognition system based on deep learning provided in the present application, where the license plate character recognition system based on deep learning includes:
the first acquiring unit 501 is configured to acquire a license plate image of a vehicle, where the license plate image includes a license plate region of the vehicle;
an input unit 502, configured to input a license plate image into a preconfigured neural network model;
a second obtaining unit 503, configured to obtain a thermodynamic diagram output by the neural network model, where the thermodynamic diagram is used to represent positions of characters in the license plate image;
the segmentation unit 504 is configured to process the thermodynamic diagram by using a MSER region feature extraction method, and segment a character region including characters to obtain target license plate characters;
and the recognition unit 505 is configured to recognize the segmented target license plate characters and obtain a recognition result.
Optionally, the system further comprises:
and an equalizing unit 506, configured to scale the target license plate character to a preset size and perform histogram equalization on the scaled target license plate character.
Optionally, the system further comprises:
the first preprocessing unit is used for performing first preprocessing on the license plate image, and the first preprocessing at least comprises one of horizontal correction, vertical correction or perspective correction.
The application also provides a license plate character recognition method and device based on deep learning, and the license plate character recognition method and device based on deep learning comprise the following steps:
a processor 601, a memory 602, an input-output unit 603, a bus 604;
the processor 601 is connected with the memory 602, the input/output unit 603 and the bus 604;
the memory 602 stores a program, and the processor 601 calls the program to execute any of the above license plate character recognition methods based on deep learning.
The present application also relates to a computer-readable storage medium having a program stored thereon, wherein the program, when executed on a computer, causes the computer to perform any of the above methods for recognizing characters on a license plate based on deep learning.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and the like.
Claims (10)
1. A license plate character recognition method based on deep learning is characterized in that the method applies license plate recognition of vehicles, and the method comprises the following steps:
acquiring a license plate image of the vehicle, wherein the license plate image comprises a license plate area of the vehicle;
inputting the license plate image into a pre-configured neural network model, wherein the neural network model is used for extracting character features in the license plate image and generating a thermodynamic diagram according to the character features, and the thermodynamic diagram is used for highlighting the positions of characters in the license plate image;
acquiring the thermodynamic diagram output by the neural network model;
processing the thermodynamic diagram by an MSER region feature extraction method, and segmenting a character region containing the characters to obtain target license plate characters;
and identifying the segmented target license plate characters, and obtaining an identification result.
2. The method for recognizing license plate characters based on deep learning of claim 1, wherein before the recognizing the segmented target license plate characters and obtaining the recognition result, the method further comprises:
and scaling the target license plate characters to a preset size and carrying out histogram equalization on the scaled target license plate characters.
3. The deep learning-based license plate character recognition method of claim 1, wherein before the inputting the license plate image into the preconfigured neural network model, the method further comprises:
and performing first preprocessing on the license plate image, wherein the first preprocessing at least comprises one of horizontal correction, vertical correction or perspective correction.
4. The deep learning-based license plate character recognition method of any one of claims 1 to 3, wherein the neural network model is obtained by training according to the following method:
acquiring a plurality of material images, wherein each material image comprises a region of a license plate of a vehicle;
carrying out binarization processing on each material image to obtain a corresponding sample image;
marking each pixel point in each sample image, wherein a first label is marked for the pixel point of a character area in the license plate, and a second label is marked for the pixel point outside the character area, so as to obtain a corresponding label image;
inputting the sample image and the corresponding label image into an initialized neural network model;
and calculating the characteristic loss difference between the material image and the label image, and dynamically adjusting the initialized neural network model according to the characteristic loss difference until the converged neural network model is obtained.
5. The method for recognizing characters of license plate based on deep learning as claimed in claim 4, wherein before the binarizing processing on each material image, the method further comprises:
and adjusting the size of the material image to enable the aspect ratio of the material image to be 3: 1.
6. The method for recognizing characters on license plates based on deep learning of claim 4, wherein before labeling each pixel point in each sample image, the method further comprises:
judging whether the characters of the license plate in the sample image are complete:
if not, filling the character incomplete part and/or removing the character redundant part.
7. The method for recognizing characters of license plate based on deep learning as claimed in claim 4, wherein before the binarizing processing on each material image, the method further comprises: and performing second preprocessing on the sample image, wherein the second preprocessing at least comprises one of horizontal correction, vertical correction or perspective correction.
8. A license plate character recognition system based on deep learning, the system comprising:
the first acquisition unit is used for acquiring a license plate image of the vehicle, wherein the license plate image comprises a license plate area of the vehicle;
the input unit is used for inputting the license plate image into a pre-configured neural network model;
the second acquisition unit is used for acquiring a thermodynamic diagram output by the neural network model, and the thermodynamic diagram is used for representing the positions of characters in the license plate image;
the segmentation unit is used for processing the thermodynamic diagram by an MSER region feature extraction method and segmenting a character region containing the characters to obtain target license plate characters;
and the recognition unit is used for recognizing the segmented target license plate characters and obtaining a recognition result.
9. A license plate character recognition device based on deep learning, characterized in that the device comprises:
the device comprises a processor, a memory, an input and output unit and a bus;
the processor is connected with the memory, the input and output unit and the bus;
the memory holds a program that the processor calls to perform the method of any one of claims 1 to 7.
10. A computer-readable storage medium having a program stored thereon, the program, when executed on a computer, performing the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011541596.8A CN112580629A (en) | 2020-12-23 | 2020-12-23 | License plate character recognition method based on deep learning and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011541596.8A CN112580629A (en) | 2020-12-23 | 2020-12-23 | License plate character recognition method based on deep learning and related device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112580629A true CN112580629A (en) | 2021-03-30 |
Family
ID=75139132
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011541596.8A Pending CN112580629A (en) | 2020-12-23 | 2020-12-23 | License plate character recognition method based on deep learning and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112580629A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113221867A (en) * | 2021-05-11 | 2021-08-06 | 北京邮电大学 | Deep learning-based PCB image character detection method |
| CN115116047A (en) * | 2022-08-29 | 2022-09-27 | 松立控股集团股份有限公司 | License plate character region thermodynamic diagram-based license plate detection method |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106650731A (en) * | 2016-12-23 | 2017-05-10 | 中山大学 | Robust license plate and logo recognition method |
| CN108985137A (en) * | 2017-06-02 | 2018-12-11 | 杭州海康威视数字技术股份有限公司 | A kind of licence plate recognition method, apparatus and system |
| CN110378308A (en) * | 2019-07-25 | 2019-10-25 | 电子科技大学 | The improved harbour SAR image offshore Ship Detection based on Faster R-CNN |
| CN111027544A (en) * | 2019-11-29 | 2020-04-17 | 武汉虹信技术服务有限责任公司 | MSER license plate positioning method and system based on visual saliency detection |
| CN111507337A (en) * | 2020-04-10 | 2020-08-07 | 河海大学 | License plate recognition method based on hybrid neural network |
| CN111723710A (en) * | 2020-06-10 | 2020-09-29 | 河海大学常州校区 | License plate recognition method based on neural network |
| CN111753590A (en) * | 2019-03-28 | 2020-10-09 | 杭州海康威视数字技术股份有限公司 | Behavior identification method and device and electronic equipment |
| CN111798480A (en) * | 2020-07-23 | 2020-10-20 | 北京思图场景数据科技服务有限公司 | Character detection method and device based on single character and character connection relation prediction |
| CN111881914A (en) * | 2020-06-23 | 2020-11-03 | 安徽清新互联信息科技有限公司 | License plate character segmentation method and system based on self-learning threshold |
| CN112036231A (en) * | 2020-07-10 | 2020-12-04 | 武汉大学 | Vehicle-mounted video-based lane line and road surface indication mark detection and identification method |
-
2020
- 2020-12-23 CN CN202011541596.8A patent/CN112580629A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106650731A (en) * | 2016-12-23 | 2017-05-10 | 中山大学 | Robust license plate and logo recognition method |
| CN108985137A (en) * | 2017-06-02 | 2018-12-11 | 杭州海康威视数字技术股份有限公司 | A kind of licence plate recognition method, apparatus and system |
| CN111753590A (en) * | 2019-03-28 | 2020-10-09 | 杭州海康威视数字技术股份有限公司 | Behavior identification method and device and electronic equipment |
| CN110378308A (en) * | 2019-07-25 | 2019-10-25 | 电子科技大学 | The improved harbour SAR image offshore Ship Detection based on Faster R-CNN |
| CN111027544A (en) * | 2019-11-29 | 2020-04-17 | 武汉虹信技术服务有限责任公司 | MSER license plate positioning method and system based on visual saliency detection |
| CN111507337A (en) * | 2020-04-10 | 2020-08-07 | 河海大学 | License plate recognition method based on hybrid neural network |
| CN111723710A (en) * | 2020-06-10 | 2020-09-29 | 河海大学常州校区 | License plate recognition method based on neural network |
| CN111881914A (en) * | 2020-06-23 | 2020-11-03 | 安徽清新互联信息科技有限公司 | License plate character segmentation method and system based on self-learning threshold |
| CN112036231A (en) * | 2020-07-10 | 2020-12-04 | 武汉大学 | Vehicle-mounted video-based lane line and road surface indication mark detection and identification method |
| CN111798480A (en) * | 2020-07-23 | 2020-10-20 | 北京思图场景数据科技服务有限公司 | Character detection method and device based on single character and character connection relation prediction |
Non-Patent Citations (2)
| Title |
|---|
| ZHENG ZHANG，等: "Multi-Oriented Text Detection with Fully Convolutional Networks", 《2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION》, 31 December 2016 (2016-12-31), pages 4159 - 4167 * |
| 张拯: "基于文字条的自然场景文字检测算法研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》, no. 1, 15 January 2018 (2018-01-15), pages 138 - 1518 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113221867A (en) * | 2021-05-11 | 2021-08-06 | 北京邮电大学 | Deep learning-based PCB image character detection method |
| CN115116047A (en) * | 2022-08-29 | 2022-09-27 | 松立控股集团股份有限公司 | License plate character region thermodynamic diagram-based license plate detection method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112132156B (en) | Image saliency target detection method and system based on multi-depth feature fusion | |
| CN107609549B (en) | Text detection method for certificate image in natural scene | |
| Gllavata et al. | A robust algorithm for text detection in images | |
| CN109711407B (en) | License plate recognition method and related device | |
| CN111680690B (en) | Character recognition method and device | |
| CN111191611A (en) | Deep learning-based traffic sign label identification method | |
| CN112580629A (en) | License plate character recognition method based on deep learning and related device | |
| CN111539980B (en) | Multi-target tracking method based on visible light | |
| CN113591831A (en) | Font identification method and system based on deep learning and storage medium | |
| CN113065404B (en) | Method and system for detecting train ticket content based on equal-width character segments | |
| CN108877030B (en) | Image processing method, device, terminal and computer readable storage medium | |
| CN113989604A (en) | Tire DOT information identification method based on end-to-end deep learning | |
| CN116798041A (en) | Image recognition method and device and electronic equipment | |
| Zhang et al. | A novel approach for binarization of overlay text | |
| CN110991440A (en) | Pixel-driven mobile phone operation interface text detection method | |
| CN111881897B (en) | Parking lot ground Chinese sign recognition method and system and storage medium thereof | |
| CN114882469B (en) | Traffic sign detection method and system based on DL-SSD model | |
| CN110633666A (en) | Gesture track recognition method based on finger color patches | |
| CN111860173B (en) | Remote sensing image ground feature element extraction method and system based on weak supervision | |
| CN111931689B (en) | Method for extracting video satellite data identification features on line | |
| CN114283087A (en) | Image denoising method and related equipment | |
| CN114529570A (en) | Image segmentation method, image identification method, user certificate subsidizing method and system | |
| Khin et al. | License Plate Identification from Myanmar Vehicle Images under Different Environmental Conditions. | |
| CN111105394A (en) | Method and device for detecting characteristic information of luminous ball | |
| CN111507324B (en) | Card frame recognition method, device, equipment and computer storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |