CN112906699B - Detection and identification method for license plate amplified number - Google Patents
Detection and identification method for license plate amplified number Download PDFInfo
- Publication number
- CN112906699B CN112906699B CN202011532814.1A CN202011532814A CN112906699B CN 112906699 B CN112906699 B CN 112906699B CN 202011532814 A CN202011532814 A CN 202011532814A CN 112906699 B CN112906699 B CN 112906699B
- Authority
- CN
- China
- Prior art keywords
- license plate
- image
- layer
- network
- sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000001514 detection method Methods 0.000 title claims abstract description 34
- 238000012549 training Methods 0.000 claims abstract description 53
- 230000008569 process Effects 0.000 claims abstract description 19
- 238000000605 extraction Methods 0.000 claims abstract description 9
- 238000004364 calculation method Methods 0.000 claims abstract description 5
- 230000006870 function Effects 0.000 claims description 17
- 238000013518 transcription Methods 0.000 claims description 15
- 230000035897 transcription Effects 0.000 claims description 15
- 238000009826 distribution Methods 0.000 claims description 14
- 238000012545 processing Methods 0.000 claims description 14
- 125000004122 cyclic group Chemical group 0.000 claims description 9
- 238000005070 sampling Methods 0.000 claims description 9
- 230000009466 transformation Effects 0.000 claims description 8
- 238000005520 cutting process Methods 0.000 claims description 7
- 238000004458 analytical method Methods 0.000 claims description 4
- 230000002950 deficient Effects 0.000 claims description 3
- 230000010354 integration Effects 0.000 claims description 3
- 230000007246 mechanism Effects 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 238000006243 chemical reaction Methods 0.000 claims description 2
- 238000003860 storage Methods 0.000 claims description 2
- 230000001105 regulatory effect Effects 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 10
- 238000005516 engineering process Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 11
- 238000013527 convolutional neural network Methods 0.000 description 9
- 102100032202 Cornulin Human genes 0.000 description 6
- 101000920981 Homo sapiens Cornulin Proteins 0.000 description 6
- 230000001629 suppression Effects 0.000 description 5
- 101100247599 Hordeum vulgare RCAB gene Proteins 0.000 description 3
- 238000002372 labelling Methods 0.000 description 3
- 238000007781 pre-processing Methods 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 230000003321 amplification Effects 0.000 description 2
- 230000002779 inactivation Effects 0.000 description 2
- 238000003064 k means clustering Methods 0.000 description 2
- 238000003199 nucleic acid amplification method Methods 0.000 description 2
- 238000005507 spraying Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/14—Image acquisition
- G06V30/148—Segmentation of character regions
- G06V30/153—Segmentation of character regions using recognition of characters or words
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
- G06V20/63—Scene text, e.g. street names
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
- G06V20/625—License plates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Evolutionary Biology (AREA)
- Character Discrimination (AREA)
- Image Analysis (AREA)
Abstract
The invention belongs to the field of intelligent traffic, and discloses a detection and identification method for license plate amplified numbers, which comprises the following steps: detecting and positioning the area where the license plate amplified number is located to obtain a sample image of the original license plate amplified number; the recognition network based on the deep convolution recognizes the license plate amplified number characters, expands the original license plate amplified number sample image in the training stage to obtain a training sample set, then constructs the recognition network and performs feature extraction on the actual license plate amplified number image to obtain a final text recognition result. According to the invention, the weight calculation is carried out on the loss generated by the text recognition part and the super-resolution image reconstruction part in the training process of the recognition network, the feature expression capacity of the recognition network on the low-quality image is improved, the optimized weight parameter is obtained, the difficulty in creating a training sample set is reduced, the recognition efficiency of license plate characters is effectively improved, and the problem that the effect is poor due to the influence of the size, the style and the interval inconsistency of characters in the detection of the amplified number of the license plate is solved.
Description
Technical Field
The invention belongs to the field of intelligent traffic, and particularly relates to a detection and identification method for license plate amplified numbers.
Background
The license plate recognition technology is a key and core module of an intelligent traffic system, and the detection and recognition technology of license plate amplified numbers can technically improve the existing license plate recognition technology in the following three aspects:
1. The license plate related to the existing license plate recognition technology is mainly a standard motor vehicle license plate, but the number of the license plate is less, namely the amplified license plate is recognized, and the thirteenth rule of the national road traffic safety law of the people's republic of China is: the rear parts of the bodies or carriages of heavy-duty and medium-duty trucks and trailers thereof, tractors and trailers thereof should be sprayed with amplified marks, and the characters should be right and clear. Therefore, the detection and recognition method of the license plate amplified number can supplement the existing license plate recognition technology and improve the functional integrity of the license plate recognition technology.
2. Due to the influences of working environment, driving road conditions and hanging positions of the license plates of the vehicles, the license plate detection and recognition effects of the license plate recognition technology on standard license plate detection and recognition effects of the license plate detection technology on the license plate of the passenger-cargo vehicle tail shot images can be caused by the fact that the passenger-cargo vehicle tail shot images acquired by the camera possibly have the conditions of missing, shielding, fouling, blurring, overexposure and the like. Compared with a standard license plate, the size of the license plate amplified number is larger, the position is more obvious, and the license plate amplified number can be clearly captured even if the standard license plate is more fuzzy.
3. The existing license plate recognition technology is directly applied to recognition of license plate amplified numbers and is poor in effect. The main reason is that: the spraying position of the license plate amplified number is generally at the rear part of a carriage of a passenger-cargo transportation automobile, the background of the license plate amplified number is not uniform, and more interference exists; font size, font style and character spacing of license plate amplified number spraying are inconsistent.
Disclosure of Invention
In order to solve the technical problems in the prior art, the invention provides a detection and identification method for license plate amplified numbers, which effectively improves the identification efficiency of license plate characters while reducing the creation difficulty of a training sample set in a mode of improving the sample marking efficiency, carries out weighted calculation analysis on the loss of a text identification part and a super-resolution image reconstruction part in the training process, and improves the characteristic extraction effect of a network model on low-quality images, thereby effectively solving the technical problem that the character detection of license plate amplified numbers is affected by inconsistent character sizes, styles, intervals and the like and has poor effect.
The invention is realized by adopting the following technical scheme: a detection and identification method for license plate amplified numbers comprises the following steps:
s1, detecting and positioning an area where a license plate amplified number is located to obtain a sample image of an original license plate amplified number;
S2, identifying license plate amplified number characters by an identification network based on deep convolution;
The step S2 comprises the following steps:
S21, in a training stage of identifying a network model, expanding a sample image of an original license plate amplified size to obtain a training sample set;
S22, based on the expanded training sample set, constructing a recognition network, and carrying out feature extraction on an actual license plate amplified number image by using the constructed recognition network to obtain a final text recognition result; and carrying out weighted calculation analysis on the loss generated by the text recognition part and the super-resolution image reconstruction part in the training process of the recognition network, and improving the feature expression capability of the recognition network on the low-quality image to obtain the optimized weight parameters of the recognition network.
Compared with the prior art, the invention has the following advantages and beneficial effects:
1. The license plate is the most distinctive characteristic of the vehicle, and compared with a standard motor vehicle license plate, the license plate amplified number has no fixed background, the length-width ratio is greatly changed, the fonts and the sizes of characters are diversified, the character spacing is inconsistent, and the current license plate recognition technology is difficult to be directly applied to the recognition of the license plate amplified number.
Aiming at the characteristics of license plate amplified numbers, the invention designs a license plate amplified number detection and identification algorithm based on a deep convolution network. The algorithm performs weighted calculation analysis on losses generated by a text recognition part and a super-resolution image reconstruction part in the training process by increasing sample image diversity in the network model training process, improves the characteristic extraction effect of the network model on low-quality images, can well learn the characteristics of license plate amplified numbers, and has the characteristics of strong robustness and high applicability.
2. The existing main flow license plate recognition technology mainly comprises the following steps: ① The three processes of license plate detection positioning, ② license plate character segmentation positioning and ③ license plate character classification recognition respectively need to construct three models of license plate detection, character segmentation and character recognition, which consumes time and resources, and when a training sample set is established in a model training stage, single character position labeling needs to be carried out on a complete license plate sample, which consumes too much manpower.
In the license plate number amplification recognition process, the flow of a license plate recognition technology is simplified, an end-to-end license plate number amplification recognition algorithm is provided to replace the existing license plate character segmentation positioning algorithm and character classification recognition algorithm, and the recognition efficiency is improved; the diversity of sample images is enriched through data preprocessing, data enhancement and other modes, and the robustness of an algorithm is improved; by reducing the labeling flow, improving the labeling efficiency, the creating difficulty of the training sample set is reduced.
Drawings
Fig. 1 is a schematic diagram of a license plate information detection and identification flow in an embodiment of the invention;
FIG. 2 is a diagram showing an example of a license plate enlarged number position marking in a vehicle tail shot image according to an embodiment of the present invention;
FIG. 3 is a diagram of a license plate enlarged detection network in an embodiment of the invention;
FIG. 4 is a schematic diagram of a training sample image expansion process in an embodiment of the present invention;
FIG. 5 is an exemplary view of a positive sample image of a license plate magnification number, wherein three sub-views (a), (b) and (c) illustrate one form of the positive sample image, respectively;
FIG. 6 is a schematic diagram of a license plate number-enlarging recognition network in a training phase according to an embodiment of the present invention;
fig. 7 is a schematic diagram of a CNN layer structure of a license plate number-enlarging recognition network according to an embodiment of the present invention;
FIG. 8 is a schematic diagram of an RNN layer structure of a license plate number-enlarging recognition network according to an embodiment of the present invention;
FIG. 9 is a schematic diagram of an SR layer structure of a license plate amplified number recognition network in the embodiment of the invention;
FIG. 10 is a schematic diagram of the RG module structure of the SR layer;
FIG. 11 is a schematic diagram of the RCAB sub-module structure in the RG module of the SR layer;
fig. 12 is a schematic structural diagram of a license plate number-amplified recognition network in an inference stage according to an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to examples and drawings, but embodiments of the present invention are not limited thereto.
Examples
The embodiment provides a detection and identification method for license plate amplified numbers based on a deep neural network, wherein the main flow is shown in fig. 1, the method mainly comprises the steps of detection and identification of the license plate amplified numbers, and the position label information of the license plate amplified numbers is shown in fig. 2; the detailed steps are as follows:
s1, detecting and positioning the area where the license plate amplified number is located, and obtaining a sample image of the original license plate amplified number.
The detection and positioning of license plate amplified numbers are carried out on the basis of a detection network (such as a convolutional neural network) with deep convolution, and a lightweight network architecture MobileNet-SSD is taken as an example for carrying out detailed explanation:
s11, firstly, calculating generation parameters of default frames of each layer by using a k-means clustering algorithm (k-means clustering algorithm, k-means) of a convolutional neural network (YOLOv) according to label sample data distribution in training samples. Since the typical aspect ratio of the license plate amplified number sample image is relatively large, the input image size of the detection network is set to w×h, (1.5×w < h < 2*w) so as to eliminate the influence on the detection effect.
S12, using various data enhancement methods in the training process to increase the diversity of sample images and improve the detection performance of the detection network, including horizontal overturn, clipping, zooming in and out and the like.
And S13, extracting the characteristics of the sample image by using a main convolution network (MobileNet), and constructing a 6-layer characteristic pyramid network to perform position regression and category classification.
S14, processing the output of the multi-layer characteristic pyramid network through a non-maximum value suppression unit to obtain a final detection positioning result of the area where the license plate amplified number is located.
The structure of the detection network is shown in fig. 3, and includes a main convolution network MobileNet, a Non-maximum suppression unit (Non-Maximum Suppression, NMS) and a multi-layer feature pyramid network, where the main convolution network MobileNet is connected to the input end of the multi-layer feature pyramid network, and the output end of each layer feature pyramid network is connected to the Non-maximum suppression unit, and the Non-maximum suppression unit outputs the final detection positioning result.
Step S2, recognizing characters of license plate amplified numbers
The method comprises the following steps of carrying out recognition of license plate amplified number characters based on a deep convolutional recognition network (such as a convolutional neural network CRNN):
s21, expanding the training sample image to obtain a training sample set.
In the training stage of the recognition network model, the CRNN convolutional neural network uses an end-to-end (end-to-end) training mode, a large number of input sample images are required for network optimization training, the invention firstly marks the sample image of the amplified size of the original license plate, and then expands the marked sample image of the amplified size of the original license plate, and the expansion flow is shown in fig. 4 and mainly comprises the following steps:
S211, cutting the sample image to generate area images with different sizes, wherein the area images obtained after cutting specifically comprise the following types as shown in (a) - (c) of fig. 5:
① Original license plate enlarged sample (7-8 characters): such sample images are regional images of the original license plate magnification number, as shown in fig. 5 (a);
② Defective license plate enlargement number samples (5-7 characters): the sample image is an area image obtained by cutting the original license plate province area, as shown in a (b) chart of fig. 5;
③ Boundary-expanded samples: the sample image is an area image obtained after random boundary expansion is carried out on the two license plate amplified number area images. The extended formula is specifically as follows:
Wherein l, r, u and b are respectively the expansion sizes of the license plate amplified number region image at the left, right, upper and lower boundaries, w, h is the width and height of the original license plate amplified number region image, and random is a random function.
④ Negative sample: such samples are false detection samples of the detection network, namely non-license plate amplified number areas.
S212, image normalization processing and color transformation: before model training, the convolutional neural network CRNN needs to normalize the four types of area images obtained after the sample cutting in the step S211, normalizes the size to W x 32, and normalizes the size to W as the width of the normalized image, and then performs color conversion; the method mainly comprises the following steps:
① The random width stretching is carried out on the image while the height h is kept unchanged, and the recognition capability of the convolutional neural network CRNN on narrower characters is improved; the formula of the random width stretching transformation is:
w*＝w*(random(0.4*w,0.8*w)+1)
Where w * is the image width after the stretching transformation, w is the original image width, and random is a random function.
② Judging whether the aspect ratio W */h of the width-stretched image is equal to the normalized size, namely W/32:
1) Scaling the image to W x 32 if W */h = W/32;
2) If W */h < W/32, scaling the image to W ***32,w**＝w* x (32/h) and then expanding the left and right boundaries of the image, the formula is as follows:
Wherein l and r are the extension sizes of the left and right boundaries, respectively, and random is a random function. In this embodiment, the convolutional neural network CRNN has no requirement on the width of the image, and therefore, in the size normalization process, the normalized size is w×32, but in the width stretching transformation and the left-right boundary expansion process, the width maximum value is set to 280.
3) If W */h > W/32, scaling the image to W.multidot.h **,h**＝h*(W/w* first, then expanding the upper and lower boundaries of the image, the formula is as follows:
wherein u and b are the extension sizes of the upper and lower boundaries, respectively, and random is a random function.
③ And performing random color space transformation to further increase the diversity of samples and generate sample images which are finally input into the identification network.
S213, generating a sample label: and storing each license plate character of the license plate number in an array, and generating a sample label of the license plate number according to the index value corresponding to the license plate character in the storage array.
The convolutional neural network CRNN needs to set a space (blank) tag, which is generally set to the first bit ("0") or the last bit ("n-1") of the tag list, where n is the tag list length, i.e., the number of characters), and the sample tag length is 8, and if the sample tag length is less than 8 bits, the tag value is complemented with "0".
For example, setting the blank tag value to "0", then the tag value of the positive sample image with the license plate number "87569" is "9 8 67 1000 0", that is, the tag value of the license plate character is the index value corresponding to the license plate character in the tag list plus 1, and the tag value is the index value corresponding to the character in the tag list plus 1 no matter whether the license plate character is a number, a letter or a Chinese character; for the negative sample image, its label value is "0 00000 0 0".
S22, based on the expanded sample image, an identification network is constructed, and the constructed identification network is used for extracting features of an actual license plate amplified number image.
In this embodiment, a feature extraction network is constructed as an identification network, specifically a deep convolution network including a convolution layer (CNN), a feature super-resolution branch network (SR layer), a circulation layer (RNN), a transcription layer (CTC), and a loss function layer, where the convolution layer is connected to the SR layer and the circulation layer, the transcription layer is connected to the circulation layer, the transcription layer and the feature super-resolution branch network are connected to the loss function layer, respectively, the input image size is w×32, W is an image width, and 32 is an image height.
In the invention, the SR layer and the RNN layer share the characteristic sequence of the image, and an additional characteristic extraction network is not needed, so that the SR layer has fewer network layers, the SR layer has simpler structure than the existing super-resolution network, occupies less video memory of a display card in the training process, and takes shorter training time.
S221, extracting a characteristic sequence from the input image through a convolution layer (CNN).
Taking a dense convolution network (DenseNet) as an example, when a feature extraction network is constructed, 3 DenseNet blocks are used for connecting CNN layers in series, the depth of each DenseNet block is d, the feature map growth rate is r, each two DenseNet blocks are connected by using a convolution layer with the kernel size of k and a random inactivation layer (dropout), the proportion of the random inactivation layer dropout is set to be ratio, and finally, a pooling layer with the kernel size of m is connected, and a feature map with the dimension of N, C, H and W is output, wherein the feature map is in batch processing size, feature map channel number, feature map height and feature map width.
And S222, in the training stage, improving the characteristic expression capacity of the CNN layer through a characteristic super-resolution branch network (SR layer), and reconstructing and outputting a super-resolution image.
The aim of the feature super-resolution branch network is to obtain high resolution image features using low resolution images. Because of the influence of the hardware condition, the working environment and the driving road condition, a large number of license plate amplified number images with low quality are often collected by a camera, and the recognition result is influenced. Therefore, the feature super-resolution branch network is added in the training process to improve the feature expression capability of the CNN layer, namely, the feature sequence obtained by the CNN layer is input into the SR layer to reconstruct the super-resolution image, so that the low-resolution features are restored into the corresponding super-resolution image.
Because the license plate amplified number identification data set does not distinguish high-resolution images and low-resolution images, in the training process, the invention uses two image expansion modes of Gaussian blur processing and 4 times up and down sampling to perform online expansion pretreatment on an original image to generate low-resolution images so as to enrich the diversity of sample images in the training data set; the generated low-resolution image is input into a characteristic super-resolution branch network SR layer to be reconstructed into a super-resolution image after the characteristic sequence is extracted by the convolution layer. In this embodiment, the image processed by the "gaussian blur processing" and the "4-fold up-down sampling" is expressed as:
Wherein I blur is a processed low resolution image, f d-u and f gau represent "4 times up-down sampling" and "gaussian blur processing", O is an original image, p 1 and p 2 are two random parameters, and α is a threshold.
The SR layer is mainly realized by 2 super-resolution basic units based on a residual error network structure (Resnet) and an up-sampling unit (UpSample), wherein the super-resolution basic units are residual error channel attention blocks RG, RCAB is a sub-module of the residual error channel attention blocks RG, and two RCAB sub-blocks form a residual error attention module RG.
The SR layer uses the feature sequence F CNN output by the CNN layer to reconstruct super resolution, firstly, the deeper features are output by two RG layers, namely:
FRG＝HRG(HRG(FCNN))
Wherein, F RG is the feature processed by two layers of RG modules, and H RG is the operation corresponding to the RG modules; the F RG features are then processed using an upsampling layer UpSample, a convolution operation to obtain a super-resolution reconstructed image O of the same size as the input image.
Wherein, F UP is the feature processed by the module of the upsampling layer UpSample, H UP is the operation corresponding to the UpSample module, and H Conv is the operation corresponding to the convolution module. And finally, taking the original high-resolution image in the training sample set as a real sample label, calculating the loss of the reconstructed super-resolution image by using the super-resolution loss function of S225, and judging and evaluating the reconstruction effect of the super-resolution image according to the magnitude of the loss value.
S223, predicting tag value distribution, namely true value distribution, of the feature sequence acquired from the convolution layer (CNN) through the loop layer (RNN).
The cyclic layer RNN comprises two bidirectional long-short-term memory networks (BiLSTM), features extracted by the cyclic layer CNN are transformed by the cyclic layer to obtain features of T, N and M dimensions, the features are input into the cyclic layer RNN, wherein T is the time sequence length of the cyclic layer RNN, N is the batch processing size, M is the input feature length, then a label distribution result of T, N dimensions is obtained through the full-connection layer, and N is the length of a label list (character class number); the loop layer RNN may be denoted as y=r w (x), where x is the input, w is a parameter of the RNN layer, and y is the output.
S224, converting the label value distribution acquired from the circulation layer RNN into a final recognition result through operations such as de-duplication integration and the like through a transcription layer (CTC).
A blank mechanism is introduced into a transcription layer CTC in order to obtain a final predicted text sequence through operations such as de-duplication integration. Taking "-" symbol as an example, the transcription layer CTC looks at "continuous repeated characters without blank interval" as the same character, firstly deletes "continuous repeated characters without blank interval" for the character sequence, and then deletes all "-" characters from the path, so as to obtain the final predicted text sequence.
For an input x given by the cyclic layer RNN, the probability that the transcription layer outputs the correct license plate is:
Wherein pi epsilon B -1 (L) represents the path of the correct license plate L after B transformation (i.e. after the processing of the cyclic layer RNN), and L is the predicted output sequence (i.e. predicted license plate number); for any path pi there is:
Where L' is all paths. In the training process, the training target of the transcriptome CTC is essentially realized by gradient The parameters w of the loop layer RNN are adjusted so that the probability p (i|x) of the correct license plate is maximized for an input sample pi e B -1 (i).
S225, calculating and identifying the total loss of the network through the loss function.
In the training process, the loss function simultaneously comprises the loss of the text recognition part and the loss of the super-resolution branch network part, so that the feature sequence extracted by the CNN layer simultaneously comprises the information of the recognition part and the super-resolution branch part, the feature expression capacity of the recognition network on the low-quality image is improved, and the feature extraction effect of the recognition network on the low-quality image is improved.
That is, in the present invention, the total loss of the recognition network is the sum of the text recognition loss L rec generated by the transcription layer CTC and the super-resolution image loss L sr generated by the super-resolution branch network, and one super-parameter λ is used to adjust the weight of the super-resolution image loss L sr, that is, the weighted sum; the loss function can be described as:
Wherein, O is the original image, O i,j is the pixel value of the original image at the (I, j) position, I i,j is the pixel value of the super-resolution image output by the SR layer of the characteristic super-resolution branch network at the (I, j) position, x is input, S is the training sample set, and z is the sample real label. The total loss of the identification network is reduced through training, and the optimized weight parameters of the identification network are obtained.
The five phases of steps S221-S225 described above constitute a training phase for the identification network, see in particular fig. 6-11.
S226, reasoning output stage
The training is used to obtain an identification network model for reasoning output, and referring specifically to fig. 12, the main process includes: the actual license plate amplified number image is directly input into a CNN layer without being subjected to Gaussian blur processing or image preprocessing such as up-down sampling and the like, so that a corresponding characteristic sequence is obtained; the feature sequence output by the CNN layer is directly input into the RNN layer, so that probability distribution of all character categories of each time step is obtained; inputting the probability distribution of the character types output by the RNN layer into the CTC layer, wherein the CTC layer takes the character with the highest probability distribution in all the character types of each time step as the output character of the time step, then splices the output characters of all the time steps to obtain a sequence path which is used as the maximum probability path, and finally obtains the final text recognition result by using a blank mechanism in the CTC layer.
That is, in the training phase, the SR layer continuously updates the network weight through iterative training to minimize the loss function and obtain the optimized weight parameter. In the reasoning stage, the CNN layer inputs an actually acquired license plate amplified image, and image preprocessing such as Gaussian blur processing or up-down sampling is not performed; and the SR layer is not used in the recognition network any more, and the trained weight parameters are directly used, because the output result of the SR layer is a super-resolution image, the SR layer is discarded in the reasoning stage without any influence on the character recognition result.
The above examples are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above examples, and any other changes, modifications, substitutions, combinations, and simplifications that do not depart from the spirit and principle of the present invention should be made in the equivalent manner, and the embodiments are included in the protection scope of the present invention.
Claims (6)
1. The detection and identification method for the license plate amplified number is characterized by comprising the following steps of:
s1, detecting and positioning an area where a license plate amplified number is located to obtain a sample image of an original license plate amplified number;
S2, identifying license plate amplified number characters by an identification network based on deep convolution;
The step S2 comprises the following steps:
S21, in a training stage of identifying a network model, expanding a sample image of an original license plate amplified size to obtain a training sample set;
S22, based on the expanded training sample set, constructing a recognition network, and carrying out feature extraction on an actual license plate amplified number image by using the constructed recognition network to obtain a final text recognition result; the method comprises the steps of carrying out weighted calculation analysis on losses generated by a text recognition part and a super-resolution image reconstruction part in the training process of the recognition network, improving the characteristic expression capability of the recognition network on low-quality images, and obtaining optimized weight parameters of the recognition network;
The identification network constructed in the step S22 is a deep convolution network comprising a convolution layer, a characteristic super-resolution branch network, a circulation layer, a transcription layer and a loss function layer, wherein the convolution layer is respectively connected with the characteristic super-resolution branch network and the circulation layer, the transcription layer is connected with the circulation layer, and the transcription layer and the characteristic super-resolution branch network are respectively connected with the loss function layer;
Step S22 includes:
S221, extracting a characteristic sequence from an input image through a convolution layer;
S222, in a training stage, expanding an original image to generate a low-resolution image, extracting a feature sequence from the generated low-resolution image through a convolution layer, and inputting the feature sequence into a feature super-resolution branch network to reconstruct into a super-resolution image; meanwhile, the feature sequence extracted by the convolution layer of the original high-resolution image in the training sample set is used as a real sample label, and the loss of the reconstructed super-resolution image is calculated;
S223, predicting tag value distribution of the feature sequence obtained from the convolution layer through the loop layer;
s224, converting the label value distribution obtained from the circulating layer into a final recognition result through a de-duplication integration operation through a transcription layer;
S225, calculating the total loss of the recognition network through a loss function, and carrying out weighted summation on text recognition loss L rec generated by a transcription layer and super-resolution image loss L sr generated by a super-resolution branch network to serve as the total loss of the recognition network, wherein the weight of the super-resolution image loss L sr is regulated through a super parameter lambda; the total loss of the identification network is reduced through training, and the optimized weight parameter of the identification network is obtained;
S226, performing reasoning output by using the recognition network model obtained after training, and directly inputting an actual license plate amplified number image into a convolution layer to obtain a corresponding feature sequence; inputting the feature sequence into a circulating layer to obtain probability distribution of all character categories of each time step; inputting the probability distribution of the character categories output by the circulating layer into the transcribing layer, wherein the transcribing layer takes the characters with the maximum probability distribution in all the character categories of each time step as the output characters of the time step, then splices the output characters of all the time steps to obtain a sequence path which is used as the maximum probability path, and finally obtains a final text recognition result by using a blank mechanism in the transcribing layer;
In step S222, the original image is expanded by gaussian blur processing and multiple up-down sampling to generate a low resolution image, let I blur be the processed low resolution image, f d-u and f gau represent multiple up-down sampling and gaussian blur processing, respectively, and O be the original image, then the image after multiple up-down sampling and gaussian blur processing is represented as:
Where p 1 and p 2 are two random parameters, α is a threshold;
The loss function of step S225 is described as:
Wherein, O is the original image, O i,j is the pixel value of the original image at the (I, j) position, I i,j is the pixel value of the super-resolution image output by the characteristic super-resolution branch network at the (I, j) position, x is the input, S is the training sample set, and z is the sample real label.
2. The method for detecting and identifying a license plate amplified number according to claim 1, wherein step S21 comprises:
s211, cutting the sample image to generate area images with different sizes;
S212, carrying out normalization processing on the multi-class area images obtained after the sample cutting in the step S211 before training the identification network model, and then carrying out color conversion; normalizing the size to W.times.32, wherein W is the normalized image width;
S213, storing each license plate character of the license plate number in an array, and generating a sample label of the license plate number according to the index value corresponding to the license plate character in the storage array.
3. The method for detecting and identifying the amplified number of the license plate according to claim 2, wherein the area image obtained after clipping in step S211 includes the following categories:
The original license plate amplified number sample is an area image of the original license plate amplified number;
the defective license plate amplified number sample is a region image obtained by cutting after discarding the original license plate province region for short;
The sample after boundary expansion is an area image obtained by carrying out random boundary expansion on an original license plate amplified number sample and a defective license plate amplified number sample;
The negative sample is a false detection sample of the detection network, namely a non-license plate amplified number area.
4. The method for detecting and identifying a license plate amplified number according to claim 2, wherein step S212 comprises:
(1) Randomly stretching the image by keeping the height h unchanged;
(2) Judging whether the aspect ratio W */h of the image after the width stretching is equal to W/32; if yes, scaling the image to W.times.32; if W */h < W/32, scaling the image to W ***32,w**＝w* x (32/h) and then expanding the left and right boundaries of the image:
wherein l and r are the expansion sizes of the left and right boundaries respectively, and random is a random function; if W */h > W/32, scaling the image to W.multidot.h **,h**＝h*(W/w*) and then expanding the upper and lower boundaries of the image:
Wherein u and b are the expansion sizes of the upper and lower boundaries respectively, and random is a random function;
(3) A random color space transformation is performed to generate a sample image that is ultimately input into the recognition network.
5. The method for detecting and identifying the amplified number of the license plate according to claim 1, wherein in the step S223, the circulation layer includes two-way long-short-term memory networks, features extracted by the convolution layer are transformed by the circulation layer to obtain features in dimension T, N, M, and N, respectively, and the features are input to the circulation layer, wherein T is a time sequence length of the circulation layer, N is a batch size, M is an input feature length, and then a tag distribution result in dimension T, N is a length of a tag list is obtained through the full connection layer; the cyclic layer is denoted y=r w (x), where x is the input, w is a parameter of the cyclic layer, and y is the output.
6. The method according to claim 5, wherein in step S224, for the input x given by the loop layer, the probability that the transcription layer outputs the correct license plate is:
Wherein pi epsilon B -1 (L) represents the path of the correct license plate L as the result after transformation of the circulating layer, and L is the predicted output sequence; for any path pi there is:
Wherein L' is all paths; during the training process, the training target of the transcription layer passes through the gradient The parameters w of the loop layer are adjusted so that the probability p (l|x) of the correct license plate is maximized for an input sample pi e B -1 (l).
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011532814.1A CN112906699B (en) | 2020-12-23 | 2020-12-23 | Detection and identification method for license plate amplified number |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011532814.1A CN112906699B (en) | 2020-12-23 | 2020-12-23 | Detection and identification method for license plate amplified number |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112906699A CN112906699A (en) | 2021-06-04 |
| CN112906699B true CN112906699B (en) | 2024-06-14 |
Family
ID=76111578
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011532814.1A Active CN112906699B (en) | 2020-12-23 | 2020-12-23 | Detection and identification method for license plate amplified number |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112906699B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113538347B (en) * | 2021-06-29 | 2023-10-27 | 中国电子科技集团公司电子科学研究院 | Image detection method and system based on efficient bidirectional path aggregation attention network |
| CN114118199A (en) * | 2021-09-01 | 2022-03-01 | 济宁安泰矿山设备制造有限公司 | Image classification method and system for fault diagnosis of intelligent pump cavity endoscope |
| CN114639090A (en) * | 2021-10-28 | 2022-06-17 | 中国公路工程咨询集团有限公司 | Robust Chinese license plate recognition method under uncontrolled environment |
| CN114332844B (en) * | 2022-03-16 | 2022-07-01 | 武汉楚精灵医疗科技有限公司 | Intelligent classification application method, device, equipment and storage medium of medical image |
| CN115035526A (en) * | 2022-06-21 | 2022-09-09 | 亿嘉和科技股份有限公司 | Deep learning-based automatic LED character positioning and identifying method |
| CN114882492B (en) * | 2022-07-11 | 2022-11-22 | 浙江大华技术股份有限公司 | License plate recognition method, device, terminal and computer readable storage medium |
| CN115063786A (en) * | 2022-08-18 | 2022-09-16 | 松立控股集团股份有限公司 | High-order distant view fuzzy license plate detection method |
Family Cites Families (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7399220B2 (en) * | 2002-08-02 | 2008-07-15 | Kriesel Marshall S | Apparatus and methods for the volumetric and dimensional measurement of livestock |
| CN104809443B (en) * | 2015-05-05 | 2018-12-28 | 上海交通大学 | Detection method of license plate and system based on convolutional neural networks |
| CN106446895A (en) * | 2016-10-28 | 2017-02-22 | 安徽四创电子股份有限公司 | License plate recognition method based on deep convolutional neural network |
| CN107704857B (en) * | 2017-09-25 | 2020-07-24 | 北京邮电大学 | End-to-end lightweight license plate recognition method and device |
| CN108764228A (en) * | 2018-05-28 | 2018-11-06 | 嘉兴善索智能科技有限公司 | Word object detection method in a kind of image |
| US11544167B2 (en) * | 2019-03-23 | 2023-01-03 | Uatc, Llc | Systems and methods for generating synthetic sensor data via machine learning |
| CN110020651B (en) * | 2019-04-19 | 2022-07-08 | 福州大学 | License plate detection and positioning method based on deep learning network |
| RU2706891C1 (en) * | 2019-06-06 | 2019-11-21 | Самсунг Электроникс Ко., Лтд. | Method of generating a common loss function for training a convolutional neural network for converting an image into an image with drawn parts and a system for converting an image into an image with drawn parts |
| GB201912428D0 (en) * | 2019-08-29 | 2019-10-16 | Sita Information Networking Computing Uk Ltd | Luggage identification via image recognition |
| CN110633755A (en) * | 2019-09-19 | 2019-12-31 | 北京市商汤科技开发有限公司 | Network training method, image processing method and device and electronic equipment |
| CN111461134B (en) * | 2020-05-18 | 2023-04-07 | 南京大学 | Low-resolution license plate recognition method based on generation countermeasure network |
| CN111626295B (en) * | 2020-07-27 | 2021-02-09 | 杭州雄迈集成电路技术股份有限公司 | Training method and device for license plate detection model |
-
2020
- 2020-12-23 CN CN202011532814.1A patent/CN112906699B/en active Active
Non-Patent Citations (2)
| Title |
|---|
| Gait changes after weight loss on adolescent with severe obesity after sleeve gastrectomy;Summa, S;SURGERY FOR OBESITY AND RELATED DISEASES;第15卷(第3期);374-381 * |
| 基于深度学习的光照不均匀文本图像的识别系统;何鎏一;杨国为;;计算机应用与软件(第06期);196-223 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112906699A (en) | 2021-06-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112906699B (en) | Detection and identification method for license plate amplified number | |
| CN112308860B (en) | Earth observation image semantic segmentation method based on self-supervision learning | |
| CN110390251B (en) | Image and character semantic segmentation method based on multi-neural-network model fusion processing | |
| CN111914797B (en) | Traffic sign identification method based on multi-scale lightweight convolutional neural network | |
| CN112070158B (en) | Facial flaw detection method based on convolutional neural network and bilateral filtering | |
| CN110163069B (en) | Lane line detection method for driving assistance | |
| CN110059768A (en) | The semantic segmentation method and system of the merging point and provincial characteristics that understand for streetscape | |
| CN113191969A (en) | Unsupervised image rain removing method based on attention confrontation generation network | |
| CN112990065B (en) | Vehicle classification detection method based on optimized YOLOv5 model | |
| CN110533068B (en) | Image object identification method based on classification convolutional neural network | |
| Dorbe et al. | FCN and LSTM based computer vision system for recognition of vehicle type, license plate number, and registration country | |
| CN111127354A (en) | Single-image rain removing method based on multi-scale dictionary learning | |
| CN113239865B (en) | Deep learning-based lane line detection method | |
| CN114998815B (en) | Traffic vehicle identification tracking method and system based on video analysis | |
| CN111507337A (en) | License plate recognition method based on hybrid neural network | |
| CN114299286A (en) | Road scene semantic segmentation method based on category grouping in abnormal weather | |
| CN112633429A (en) | Method for recognizing handwriting choice questions of students | |
| CN112465700A (en) | Image splicing positioning device and method based on depth clustering | |
| CN114743126A (en) | Lane line sign segmentation method based on graph attention machine mechanism network | |
| CN112990371B (en) | Unsupervised night image classification method based on feature amplification | |
| CN112132746B (en) | Small-scale pedestrian target rapid super-resolution method for intelligent roadside equipment | |
| CN111401360B (en) | Method and system for optimizing license plate detection model, license plate detection method and system | |
| CN105654070A (en) | Low-resolution face recognition method | |
| CN110188752B (en) | Deblurring recognition system and deblurring recognition method for blurred license plate under monitoring video | |
| CN116883650A (en) | Image-level weak supervision semantic segmentation method based on attention and local stitching |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |