CN112906699A - Method for detecting and identifying enlarged number of license plate - Google Patents
Method for detecting and identifying enlarged number of license plate Download PDFInfo
- Publication number
- CN112906699A CN112906699A CN202011532814.1A CN202011532814A CN112906699A CN 112906699 A CN112906699 A CN 112906699A CN 202011532814 A CN202011532814 A CN 202011532814A CN 112906699 A CN112906699 A CN 112906699A
- Authority
- CN
- China
- Prior art keywords
- license plate
- image
- layer
- network
- sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000012549 training Methods 0.000 claims abstract description 52
- 230000008569 process Effects 0.000 claims abstract description 23
- 238000001514 detection method Methods 0.000 claims abstract description 22
- 238000004364 calculation method Methods 0.000 claims abstract description 5
- 238000013518 transcription Methods 0.000 claims description 18
- 230000035897 transcription Effects 0.000 claims description 18
- 230000006870 function Effects 0.000 claims description 17
- 238000012545 processing Methods 0.000 claims description 16
- 238000000605 extraction Methods 0.000 claims description 11
- 238000005070 sampling Methods 0.000 claims description 9
- 125000004122 cyclic group Chemical group 0.000 claims description 7
- 230000009466 transformation Effects 0.000 claims description 6
- 238000004458 analytical method Methods 0.000 claims description 4
- 230000002950 deficient Effects 0.000 claims description 3
- 230000010354 integration Effects 0.000 claims description 3
- 230000007246 mechanism Effects 0.000 claims description 3
- 230000002457 bidirectional effect Effects 0.000 claims description 2
- 238000006243 chemical reaction Methods 0.000 claims description 2
- 230000015654 memory Effects 0.000 claims description 2
- 230000008030 elimination Effects 0.000 claims 1
- 238000003379 elimination reaction Methods 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 9
- 238000010586 diagram Methods 0.000 description 12
- 238000005516 engineering process Methods 0.000 description 11
- 230000003321 amplification Effects 0.000 description 9
- 238000013527 convolutional neural network Methods 0.000 description 9
- 238000003199 nucleic acid amplification method Methods 0.000 description 9
- 102100032202 Cornulin Human genes 0.000 description 6
- 101000920981 Homo sapiens Cornulin Proteins 0.000 description 6
- 238000002372 labelling Methods 0.000 description 4
- 238000007781 pre-processing Methods 0.000 description 4
- 230000001629 suppression Effects 0.000 description 4
- 101100247599 Hordeum vulgare RCAB gene Proteins 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 230000033228 biological regulation Effects 0.000 description 2
- 230000002779 inactivation Effects 0.000 description 2
- 238000003064 k means clustering Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 238000005507 spraying Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 230000007787 long-term memory Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000006403 short-term memory Effects 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 239000000725 suspension Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/14—Image acquisition
- G06V30/148—Segmentation of character regions
- G06V30/153—Segmentation of character regions using recognition of characters or words
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
- G06V20/63—Scene text, e.g. street names
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
- G06V20/625—License plates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
Abstract
The invention belongs to the field of intelligent transportation, and discloses a method for detecting and identifying an enlarged number of a license plate, which comprises the following steps: detecting and positioning the area where the license plate enlarged number is located to obtain a sample image of the original license plate enlarged number; the method comprises the steps of identifying license plate enlarged number characters by a deep convolution-based identification network, expanding a sample image of an original license plate enlarged number in a training stage to obtain a training sample set, then constructing the identification network and extracting features of an actual license plate enlarged number image to obtain a final text identification result. The method and the device perform weighted calculation on the loss generated by the text recognition part and the super-resolution image reconstruction part in the training process of the recognition network, improve the feature expression capability of the recognition network on low-quality images, obtain optimized weight parameters, reduce the difficulty in creating a training sample set, effectively improve the recognition efficiency of license plate characters, and solve the problem that the detection of characters of the license plate enlarged number is poor in effect due to the influence of the inconsistency of the size, the style and the spacing of the characters.
Description
Technical Field
The invention belongs to the field of intelligent transportation, and particularly relates to a method for detecting and identifying an enlarged license plate number.
Background
The license plate recognition technology is a key and core module of an intelligent traffic system, and the detection and recognition technology of the enlarged license plate number can technically promote the existing license plate recognition technology in the following three aspects:
1. the number plate related to the existing number plate recognition technology is mainly a standard motor vehicle number plate, but relates to little or little number of the number plate amplified numbers (namely the amplified number plate numbers), and the thirteenth regulation of the national road traffic safety law enforcement regulations: the heavy and medium-sized truck and the trailer thereof, the tractor and the trailer thereof, the body or the rear part of the carriage of the tractor and the trailer thereof should be sprayed with enlarged marks, and characters should be aligned and kept clear. Therefore, the method for detecting and identifying the enlarged license plate number can supplement the existing license plate identification technology and improve the functional integrity of the license plate identification technology.
2. Due to the influences of the working environment, the driving road condition and the suspension position of the number plate of the vehicle, the situations of the number plate missing, blocking, fouling, blurring, overexposure and the like may exist in the images of the passenger and freight vehicles captured by the camera, and the detection and identification effects of the number plate identification technology on the standard number plate in the images of the passenger and freight vehicles captured by the tail are further influenced. Compared with a standard license plate, the license plate enlarged number is larger in size and more obvious in position, and the license plate enlarged number can be clearly captured even if the standard license plate is fuzzy.
3. The existing license plate recognition technology is directly applied to recognition of the enlarged license plate number, and the effect is poor. The main reasons are that: the spraying position of the license plate enlarged number is generally the rear part of a passenger and freight vehicle carriage, the background of the license plate enlarged number is not uniform, and more interference exists; the font size, font style and character spacing of the license plate enlarged number spray coating are inconsistent.
Disclosure of Invention
In order to solve the technical problems in the prior art, the invention provides a detection and recognition method for a license plate enlarged number, which effectively improves the recognition efficiency of license plate characters while reducing the difficulty in creating a training sample set by improving the sample labeling efficiency, performs weighted calculation and analysis on the losses of a text recognition part and a super-resolution image reconstruction part in the training process, and improves the feature extraction effect of a network model on low-quality images, thereby effectively solving the technical problem that the detection of the characters of the license plate enlarged number is not good in effect due to the influences of inconsistent character sizes, patterns and intervals and the like.
The invention is realized by adopting the following technical scheme: a method for detecting and identifying an enlarged number of a license plate comprises the following steps:
s1, detecting and positioning the area where the license plate enlarged number is located to obtain a sample image of the original license plate enlarged number;
s2, recognizing the license plate enlarged number characters based on a deep convolution recognition network;
step S2 includes:
s21, in the training stage of the recognition network model, expanding the sample image of the original license plate enlarged number to obtain a training sample set;
s22, constructing a recognition network based on the expanded training sample set, and performing feature extraction on the actual license plate enlarged number image by using the constructed recognition network to obtain a final text recognition result; and performing weighted calculation analysis on losses generated by the text recognition part and the super-resolution image reconstruction part of the recognition network in the training process, so as to improve the feature expression capability of the recognition network on low-quality images and obtain optimized weight parameters of the recognition network.
Compared with the prior art, the invention has the following advantages and beneficial effects:
1. the license plate is the most distinctive feature of a vehicle, compared with a standard motor vehicle license plate, the license plate enlarged number has no fixed background, the length-width ratio is greatly changed, the fonts and the sizes of characters are diversified, and the character intervals are inconsistent, so that the existing license plate recognition technology is difficult to be directly applied to recognition of the license plate enlarged number.
Aiming at the characteristics of the license plate enlarged number, the invention designs a license plate enlarged number detection and recognition algorithm based on a deep convolutional network. According to the algorithm, in the network model training process, through a mode of increasing sample image diversity, weighted calculation analysis is carried out on losses generated by a text recognition part and a super-resolution image reconstruction part in the training process, the characteristic extraction effect of the network model on low-quality images is improved, the characteristics of the license plate enlarged number can be well learned, and the algorithm has the characteristics of strong robustness and high applicability.
2. The existing mainstream license plate recognition technology mainly comprises the following processes: the three processes of license plate detection and positioning, license plate character segmentation and positioning and license plate character classification and recognition are carried out, three models of license plate detection, character segmentation and character recognition need to be constructed respectively, time and resources are consumed, and when a training sample set is established in a model training stage, single-character position labeling needs to be carried out on a complete license plate sample, so that too much labor is consumed.
In the license plate amplified number recognition process, the invention simplifies the flow of the license plate recognition technology, provides an end-to-end license plate amplified number recognition algorithm to replace the existing license plate character segmentation positioning algorithm and the existing character classification recognition algorithm, and improves the recognition efficiency; the diversity of sample images is enriched through data preprocessing, data enhancement and other modes, and the robustness of the algorithm is improved; by reducing the labeling process and improving the labeling efficiency, the difficulty in creating the training sample set is reduced.
Drawings
FIG. 1 is a schematic diagram of a license plate information detection and identification process in an embodiment of the present invention;
FIG. 2 is an illustration of an enlarged license plate position in a captured image of a vehicle according to an embodiment of the present invention;
FIG. 3 is a diagram of a license plate number amplification detection network structure in an embodiment of the present invention;
FIG. 4 is a schematic diagram illustrating a training sample image expansion process according to an embodiment of the present invention;
FIG. 5 is an exemplary diagram of a positive sample image of a license plate with an enlarged number, wherein three sub-images (a), (b), and (c) respectively illustrate one form of the positive sample image;
FIG. 6 is a schematic structural diagram of a license plate amplification number recognition network in a training phase in the embodiment of the invention;
FIG. 7 is a schematic diagram of a CNN layer structure of a license plate number amplification identification network in an embodiment of the present invention;
FIG. 8 is a schematic diagram of an RNN layer structure of a license plate number-enlarging recognition network according to an embodiment of the present invention;
FIG. 9 is a schematic diagram of an SR layer structure of a license plate amplified number recognition network in the embodiment of the present invention;
FIG. 10 is a schematic diagram of the RG module structure of the SR layer;
FIG. 11 is a schematic diagram of the RCAB sub-module in the RG module of the SR layer;
FIG. 12 is a schematic structural diagram of a license plate amplification number recognition network in an inference stage in the embodiment of the invention.
Detailed Description
The present invention will be described in further detail with reference to examples and drawings, but the present invention is not limited thereto.
Examples
The embodiment provides a method for detecting and identifying an enlarged license plate number based on a deep neural network, which mainly comprises the steps of detecting and identifying the enlarged license plate number as shown in figure 1, wherein the position label information of the enlarged license plate number is shown in figure 2; the detailed steps are as follows:
and S1, detecting and positioning the area where the license plate enlarged number is located, and obtaining a sample image of the original license plate enlarged number.
In the step, a detection network (such as a convolutional neural network) based on deep convolution is used for detecting and positioning the enlarged license plate number, and a lightweight network architecture MobileNet-SSD is taken as an example for detailed description:
s11, firstly, according to the label sample data distribution in the training sample, calculating the generation parameters of each layer of default box by using a k-means clustering algorithm (k-means clustering, k-means) of a convolutional neural network (YOLOv 3). Because the license plate amplification number sample image generally has a large width-to-height ratio, the size of the input image of the detection network is set to be w x h, (1.5 w < h < 2 w) so as to eliminate the influence on the detection effect.
And S12, using various data enhancement methods in the training process to increase the diversity of the sample images and improve the detection performance of the detection network, including horizontal turning, cutting, zooming in and zooming out and the like.
And S13, extracting the features of the sample image by using a backbone convolutional network (MobileNet), and constructing a feature pyramid network with 6 layers for position regression and class classification.
And S14, processing the output of the multilayer characteristic pyramid network through the non-maximum suppression unit to obtain a final detection positioning result of the area where the license plate amplification number is located.
The structure of the detection network is shown in fig. 3, and includes a backbone convolutional network MobileNet, a Non-Maximum Suppression unit (NMS), and a multilayer feature pyramid network, where the backbone convolutional network MobileNet is connected to an input end of the multilayer feature pyramid network, an output end of each layer of feature pyramid network is connected to the Non-Maximum Suppression unit, and the Non-Maximum Suppression unit outputs a final detection positioning result.
Step S2, recognizing characters of the enlarged number of the license plate
The method comprises the following steps of identifying license plate enlarged number characters by a deep convolution-based identification network (such as a convolution neural network (CRNN)), specifically comprising the following steps:
and S21, expanding the training sample image to obtain a training sample set.
In the training stage of identifying the network model, the CRNN convolutional neural network uses an end-to-end (end-to-end) training mode, and needs a large number of input sample images to perform network optimization training, the invention firstly labels the sample images of the original license plate enlarged number, and then expands the labeled sample images of the original license plate enlarged number, and the expansion process is shown in fig. 4 and mainly comprises the following steps:
s211, cropping the sample image to generate area images with different sizes, as shown in (a) - (c) of fig. 5, the area images obtained after cropping specifically include the following categories:
the original license plate enlarged sample (7-8 characters): such a sample image is an enlarged-size area image of the original license plate, as shown in fig. 5 (a);
② defective license plate enlarged sample (5-7 characters): the sample image is a region image obtained by cutting after discarding the original license plate province region for short, as shown in the diagram (b) of fig. 5;
③ sample after boundary expansion: the sample image is an area image obtained by carrying out random boundary expansion on the two types of license plate amplified number area images. The extended formula is specifically as follows:
wherein l, r, u and b are the expansion sizes of the license plate magnified region image at the left, right, upper and lower boundaries respectively, w and h are the width and height of the original license plate magnified region image, and random is a random function.
Fourthly, loading samples: the samples are false detection samples of a detection network, namely non-license plate amplified number areas.
S212, image normalization processing and color transformation: before model training, the convolutional neural network CRNN needs to normalize the four types of region images obtained after the sample is cut in step S211, normalize the size to W × 32, where W is the normalized image width, and then perform color transformation; the method mainly comprises the following steps:
keeping the height h unchanged, and stretching the random width of the image to improve the recognition capability of a convolutional neural network (CRNN) on narrower characters; the formula for the random width stretch transform is:
w*＝w*(random(0.4*w,0.8*w)+1)
wherein, w*And w is the original image width, and random is a random function.
② judging the aspect ratio w of the image after width stretching*Whether/h is equal to the normalized size, i.e., W/32:
1) if w is*W/32, scaling the image to W32;
2) if w is*W/32 is smaller than h, the image is scaled to W***32,w**＝w*(32/h), then expanding the left and right image boundaries, the formula is as follows:
wherein l and r are the expansion sizes of the left and right boundaries respectively, and random is a random function. In this embodiment, the convolutional neural network CRNN has no requirement on the width of the image, and therefore, in the size normalization process, the normalized size is W × 32, but the maximum width value is set to 280 in the width stretching transformation and left and right boundary expansion processes.
3) If w is*W/32, scaling the image to W x h**,h**＝h*(W/w*) Then, the upper and lower boundaries of the image are expanded, and the formula is as follows:
wherein u and b are the extension sizes of the upper and lower boundaries respectively, and random is a random function.
And thirdly, random color space transformation is carried out, the diversity of the samples is further increased, and the sample images which are finally input into the identification network are generated.
S213, generating a sample label: and storing each license plate character of the license plate number in an array, and then generating a sample label of the license plate number according to the index value of the license plate character corresponding to the array.
The convolutional neural network CRNN needs to set a space (blank) tag, which is generally set as the first bit ("0") or the last bit ("n-1") of the tag list, where n is the length of the tag list, i.e., the number of character classes), the length of the sample tag is 8, and less than 8 bits are complemented with "0" after the tag value.
For example, if the label value of blank is set to be "0", the label value of the positive sample image with the license plate number of "87569" is "986710000", that is, the label value of the license plate character is obtained by adding 1 to the corresponding index value of the license plate character in the label list, and the label value of the license plate character is obtained by adding 1 to the corresponding index value of the character in the label list no matter whether the license plate character is a number, a letter or a Chinese character; for a negative sample image, its label value is "00000000".
S22, constructing a recognition network based on the expanded sample image, and performing feature extraction on the actual license plate enlarged number image by using the constructed recognition network.
In this embodiment, a feature extraction network is constructed as an identification network, and specifically, the feature extraction network is a deep convolutional network including a convolutional layer (CNN), a feature super-resolution branch network (SR layer), a cyclic layer (RNN), a transcription layer (CTC), and a loss function layer, where the convolutional layer is connected to the SR layer and the cyclic layer, the transcription layer is connected to the cyclic layer, the transcription layer and the feature super-resolution branch network are connected to the loss function layer, the size of an input image is W × 32, W is an image width, and 32 is an image height.
In the invention, the SR layer and the RNN layer share the characteristic sequence of the image and do not need an additional characteristic extraction network, so the number of network layers of the SR layer is less, the SR layer has a simpler structure than the existing super-resolution network, the occupied video memory of a video card is less in the training process, and the training time is shorter.
S221, a feature sequence is extracted from the input image by the convolutional layer (CNN).
Taking a dense convolutional network (DenseNet) as an example, when constructing a feature extraction network, connecting CNN layers in series by using 3 DenseNet blocks, wherein the depth of each DenseNet block is d, the feature map growth rate is r, connecting convolution layers with the kernel size of k × k and random inactivation layers (dropouts) between every two DenseNet blocks, setting the proportion of the random inactivation layers dropouts as ratio, finally connecting a pooling layer with the kernel size of m × N, and outputting a feature map with dimensions of N × C × H × W, wherein N, C, H and W are the batch processing size, the feature map channel number, the feature map height and the feature map width respectively.
S222, in the training stage, the feature expression capability of the CNN layer is improved through a feature super-resolution branch network (SR layer), and a super-resolution image is reconstructed and output.
The purpose of the feature super-resolution branch network is to obtain high-resolution image features using low-resolution images. Due to the influence of hardware conditions, working environments and driving road conditions, the camera can often acquire a large amount of low-quality license plate amplified number images, and the recognition result is influenced. Therefore, in the training process, the characteristic super-resolution branch network is added to improve the characteristic expression capability of the CNN layer, namely, the characteristic sequence obtained by the CNN layer is input into the SR layer to reconstruct the super-resolution image, so that the low-resolution characteristics are restored into the corresponding super-resolution image.
Because the license plate amplification number identification data set does not distinguish high-resolution images and low-resolution images, in the training process, the invention uses two image expansion modes of Gaussian blur processing and 4-time up-down sampling to perform online expansion preprocessing on the original image to generate the low-resolution images so as to enrich the diversity of sample images in the training data set; and after the generated low-resolution image is subjected to feature sequence extraction by the convolutional layer, the low-resolution image is input to a SR layer of a feature super-resolution branch network and is reconstructed into a super-resolution image. In this embodiment, the image after the "gaussian blur processing" and the "4-fold up-down sampling" processing is represented as:
wherein, IblurFor processed low resolution images, fd-uAnd fgauRespectively representing 4 times of up-down sampling and Gaussian blur processing, O is an original image, p1And p2Are two random parameters and alpha is a threshold.
The SR layer is mainly implemented by 2 super-resolution base units based on a residual network structure (Resnet) and an upsampling unit (UpSample), where the super-resolution base unit is a residual channel attention block RG, the RCAB is a sub-module of the residual channel attention block RG, and two RCAB sub-blocks constitute a residual attention module RG.
SR layer uses characteristic sequence F output by CNN layerCNNPerforming super-resolution reconstruction, and outputting deeper features through two RG layers, namely:
FRG＝HRG(HRG(FCNN))
wherein, FRGFeatures processed by two layers of RG modules, HRGCorresponding operation for the RG module; then using the upsampling layer UpSample, convolution operation pair FRGAnd processing the characteristics to obtain a super-resolution reconstructed image O with the same size as the input image.
Wherein, FUPFor the feature processed by the UpSample module at the up-sampling layer, HUPFor the corresponding operation of UpSample Module, HConvCorresponding operations for the convolution module. And finally, the original high-resolution image in the training sample set is used as a real sample label, the loss of the reconstructed super-resolution image is calculated by using the super-resolution loss function of S225, and the reconstruction effect of the super-resolution image is judged and evaluated according to the loss value.
S223, the tag value distribution, i.e., the true value distribution of the feature sequence obtained from the convolutional layer (CNN) is predicted by the cycle layer (RNN).
The circulation layer RNN comprises two bidirectional long and short term memory networks (BilSTM), the features extracted by the convolution layer CNN are converted by the circulation layer to obtain T x N x M dimensional features, the T is the time sequence length of the circulation layer RNN, N is the batch processing size, M is the input feature length, then T x N x N dimensional label distribution results are obtained by the full connection layer, and N is the length of a label list (character category number); the loop layer RNN may be expressed as y ═ Rw(x) Where x is the input, w is the RNN layer parameter, and y is the output.
S224, the tag value distribution obtained from the cycle layer RNN is converted into the final recognition result by the transcription layer (CTC) through operations such as de-duplication integration.
A blank mechanism is introduced into CTC in a transcription layer, and the purpose is to obtain a final predicted text sequence through operations such as de-duplication integration. Taking the "-" symbol representing blank as an example, the CTC in the transcription layer considers that "continuous repeated characters without blank intervals" are the same character, deletes "continuous repeated characters without blank intervals" for the character sequence, and then deletes all "-" characters from the path to obtain the final predicted text sequence.
For the input x given by the cycle layer RNN, the probability that the transcription layer outputs the correct license plate is as follows:
wherein, pi ∈ B-1(l) Representing all paths of which the result is correct license plate L after B conversion (namely after cycle layer RNN processing), and L is a prediction output sequence (namely a predicted license plate number); for any path π is:
where L' is all paths. In the training process, the training target of the CTC of the transcription layer is essentially through the gradient
Adjusting the parameter w of the loop layer RNN such that for an input sample, pi ∈ B-1(l) The probability p (l | x) of the correct license plate is the greatest.
And S225, calculating and identifying the total loss of the network through a loss function.
In the training process, the loss function simultaneously contains the loss of the text recognition part and the loss of the super-resolution branch network part, so that the feature sequence extracted by the CNN layer simultaneously contains the information of the recognition part and the super-resolution branch network part, the feature expression capability of the recognition network on low-quality images is improved, and the feature extraction effect of the recognition network on the low-quality images is improved.
That is, in the present invention, the total loss of recognition network is the text recognition loss L generated by the transcription layer CTCrecAnd super-resolution image loss L generated by super-resolution branch networksrSumming and using a hyper-parameter lambda to the super-resolution image loss LsrThe weights of (a) are adjusted, i.e. weighted summation; the loss function can be described as:
wherein O is the original image, Oi,jIs the pixel value of the original image at the (I, j) position, Ii,jThe pixel value of a super-resolution image output by a SR layer of the characteristic super-resolution branch network at the (i, j) position is represented by x, S is a training sample set and z is a sample real label. And reducing the total loss of the recognition network through training to obtain the optimized weight parameter of the recognition network.
The five stages of steps S221 to S225 form a training stage of the recognition network, and refer to fig. 6 to 11 in detail.
S226, reasoning output stage
Using the trained recognition network model to perform inference output, specifically referring to fig. 12, the main process includes: directly inputting an actual license plate amplification number image into a CNN layer without image preprocessing such as Gaussian blur processing or up-down sampling to obtain a corresponding characteristic sequence; directly inputting the characteristic sequence output by the CNN layer into the RNN layer to obtain the probability distribution of all character types of each time step; inputting the character type probability distribution output by the RNN layer into a CTC layer, taking the characters with the maximum probability distribution in all the character types of each time step as the output characters of the time step by the CTC layer, splicing the output characters with all the time steps to obtain a sequence path as the maximum probability path, and finally obtaining the final text recognition result by using a blank mechanism in the CTC layer.
That is, in the training stage, the SR layer continuously updates the network weights through iterative training to minimize the loss function, so as to obtain the optimized weight parameters. In the inference stage, the input of the CNN layer is an actually acquired license plate amplification number image, and image preprocessing such as Gaussian blur processing or up-down sampling is not performed; and the recognition network does not use the SR layer any more, and directly uses the trained weight parameters, and because the output result of the SR layer is a super-resolution image and is useless for reasoning of the recognition network, the SR layer is discarded in the reasoning stage, and the character recognition result cannot be influenced.
The above embodiments are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be construed as equivalents thereof, and all such changes, modifications, substitutions, combinations, and simplifications are intended to be included in the scope of the present invention.
Claims (10)
1. A method for detecting and identifying an enlarged number of a license plate is characterized by comprising the following steps:
s1, detecting and positioning the area where the license plate enlarged number is located to obtain a sample image of the original license plate enlarged number;
s2, recognizing the license plate enlarged number characters based on a deep convolution recognition network;
step S2 includes:
s21, in the training stage of the recognition network model, expanding the sample image of the original license plate enlarged number to obtain a training sample set;
s22, constructing a recognition network based on the expanded training sample set, and performing feature extraction on the actual license plate enlarged number image by using the constructed recognition network to obtain a final text recognition result; and performing weighted calculation analysis on losses generated by the text recognition part and the super-resolution image reconstruction part of the recognition network in the training process, so as to improve the feature expression capability of the recognition network on low-quality images and obtain optimized weight parameters of the recognition network.
2. The method for detecting and identifying the enlarged license plate number of claim 1, wherein the step S21 includes:
s211, cutting the sample image to generate area images with different sizes;
s212, before training of the recognition network model, normalizing the multi-class area image obtained after the sample is cut in the step S211, and then performing color transformation; normalizing the size to W32, W being the normalized image width;
s213, storing each license plate character of the license plate number in an array, and then generating a sample label of the license plate number according to the corresponding index value of the license plate character in the stored array.
3. The method for detecting and identifying the enlarged license plate number according to claim 2, wherein the area image obtained after clipping in step S211 includes the following categories:
the original license plate amplified number sample is an area image of the original license plate amplified number;
the defective license plate enlarged number sample is an area image obtained by cutting after discarding the original license plate province, namely the area;
the sample after the boundary expansion is a regional image obtained by carrying out random boundary expansion on the original license plate amplified number sample and the defective license plate amplified number sample;
and the negative sample is a false detection sample of the detection network, namely a non-license plate amplified number area.
4. The method for detecting and identifying the enlarged license plate number of claim 2, wherein step S212 comprises:
(1) keeping the height h unchanged, and stretching the image in random width;
(2) determining the aspect ratio w of the width-stretched image*Whether/h is equal to W/32; if yes, scaling the image to W32; if w is*W/32 is smaller than h, the image is scaled to W***32,w**＝w*(32/h), then expand the left and right image borders:
wherein l and r are the expansion sizes of the left and right boundaries respectively, and random is a random function; if w is*W/32, scaling the image to W x h**,h**＝h*(W/w*) Then is aligned withAnd (3) expanding the upper and lower boundaries of the image:
wherein u and b are respectively the expansion sizes of the upper and lower boundaries, and random is a random function;
(3) and carrying out random color space transformation to generate a sample image which is finally input into the identification network.
5. The method for detecting and identifying an enlarged license plate number of claim 1, wherein the identification network constructed in step S22 is a deep convolutional network comprising a convolutional layer, a characteristic super-resolution tributary network, a cyclic layer, a transcription layer, and a loss function layer, wherein the convolutional layer is connected to the characteristic super-resolution tributary network and the cyclic layer, the transcription layer is connected to the cyclic layer, and the transcription layer and the characteristic super-resolution tributary network are connected to the loss function layer.
6. The method for detecting and identifying the enlarged license plate number of claim 5, wherein the step S22 includes:
s221, extracting a characteristic sequence from the input image through the convolution layer;
s222, in the training stage, the original image is expanded to generate a low-resolution image, and the generated low-resolution image is input into a characteristic super-resolution branch network to be reconstructed into a super-resolution image after a characteristic sequence is extracted from a convolutional layer; meanwhile, a characteristic sequence extracted from the convolutional layer of the original high-resolution image in the training sample set is used as a real sample label, and the loss of the reconstructed super-resolution image is calculated;
s223, predicting the label value distribution of the characteristic sequence acquired from the convolutional layer through the loop layer;
s224, converting the label value distribution obtained from the circulation layer into a final recognition result through a duplication elimination integration operation by the transcription layer;
s225, calculating and identifying the total loss of the network through a loss function, and transcribing the layersResulting text recognition loss LrecAnd super-resolution image loss L generated by super-resolution branch networksrWeighted summation as the total loss of the identified network, where the super-resolution image loses LsrThe weight of (a) is adjusted by a hyperparameter λ; the total loss of the recognition network is reduced through training, and the optimized weight parameter of the recognition network is obtained;
s226, reasoning and outputting by using the recognition network model obtained after training, and directly inputting the actual license plate enlarged number image into the convolutional layer to obtain a corresponding characteristic sequence; inputting the characteristic sequence into a circulation layer to obtain the probability distribution of all character types of each time step; inputting the character type probability distribution output by the circulation layer into the transcription layer, taking the character with the maximum probability distribution in all the character types of each time step as the output character of the time step by the transcription layer, splicing the output characters with all the time steps to obtain a sequence path as the maximum probability path, and finally obtaining the final text recognition result by using a blank mechanism in the transcription layer.
7. The method for detecting and identifying enlarged license plate number of claim 5, wherein step S222 is performed by expanding the original image through Gaussian blur processing and multiple up-down sampling to generate a low-resolution image, and setting IblurFor processed low resolution images, fd-uAnd fgauRespectively representing multiple up-down sampling and Gaussian blur processing, wherein O is an original image, and the image after multiple up-down sampling and Gaussian blur processing is represented as follows:
wherein p is1And p2Are two random parameters and alpha is a threshold.
8. The method for detecting and identifying the enlarged license plate number of claim 7, wherein the loss function of step S225 is described as:
wherein O is the original image, Oi,jIs the pixel value of the original image at the (I, j) position, Ii,jAnd (3) outputting the pixel value of the super-resolution image at the (i, j) position by the characteristic super-resolution branch network, wherein x is input, S is a training sample set, and z is a sample real label.
9. The method for detecting and identifying an enlarged license plate according to claim 6, wherein the loop layer in step S223 includes two bidirectional long-short term memory networks, the features extracted from the convolution layer are transformed by the loop layer to obtain T × N × M dimensional features, and the T × N × M dimensional features are input into the loop layer, where T is the length of the time sequence of the loop layer, N is the size of batch processing, M is the length of the input features, and then T × N dimensional label distribution results are obtained through the fully connected layer, and N is the length of the label list; the cyclic layer is denoted by y ═ Rw(x) Where x is the input, w is the parameter of the loop layer, and y is the output.
10. The method for detecting and identifying the enlarged license plate number according to claim 9, wherein in step S224, for the input x given by the loop layer, the probability that the transcription layer outputs the correct license plate is:
wherein, pi ∈ B-1(l) Representing all paths of which the result is the correct license plate L after the conversion of the circulation layer, wherein L is a prediction output sequence; for any path π is:
wherein L' is all paths; in the training process, the transcription layerBy gradient of training target
Adjusting the parameter w of the loop layer so that for the input sample, pi ∈ B-1(l) The probability p (l | x) of the correct license plate is the greatest.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011532814.1A CN112906699A (en) | 2020-12-23 | 2020-12-23 | Method for detecting and identifying enlarged number of license plate |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011532814.1A CN112906699A (en) | 2020-12-23 | 2020-12-23 | Method for detecting and identifying enlarged number of license plate |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112906699A true CN112906699A (en) | 2021-06-04 |
Family
ID=76111578
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011532814.1A Pending CN112906699A (en) | 2020-12-23 | 2020-12-23 | Method for detecting and identifying enlarged number of license plate |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112906699A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113538347A (en) * | 2021-06-29 | 2021-10-22 | 中国电子科技集团公司电子科学研究院 | Image detection method and system based on efficient bidirectional path aggregation attention network |
| CN114332844A (en) * | 2022-03-16 | 2022-04-12 | 武汉楚精灵医疗科技有限公司 | Intelligent classification application method, device, equipment and storage medium of medical image |
| CN114882492A (en) * | 2022-07-11 | 2022-08-09 | 浙江大华技术股份有限公司 | License plate recognition method, device, terminal and computer readable storage medium |
| CN115063786A (en) * | 2022-08-18 | 2022-09-16 | 松立控股集团股份有限公司 | High-order distant view fuzzy license plate detection method |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050257748A1 (en) * | 2002-08-02 | 2005-11-24 | Kriesel Marshall S | Apparatus and methods for the volumetric and dimensional measurement of livestock |
| CN104809443A (en) * | 2015-05-05 | 2015-07-29 | 上海交通大学 | Convolutional neural network-based license plate detection method and system |
| CN106446895A (en) * | 2016-10-28 | 2017-02-22 | 安徽四创电子股份有限公司 | License plate recognition method based on deep convolutional neural network |
| CN108764228A (en) * | 2018-05-28 | 2018-11-06 | 嘉兴善索智能科技有限公司 | Word object detection method in a kind of image |
| US20190095730A1 (en) * | 2017-09-25 | 2019-03-28 | Beijing University Of Posts And Telecommunications | End-To-End Lightweight Method And Apparatus For License Plate Recognition |
| CN110020651A (en) * | 2019-04-19 | 2019-07-16 | 福州大学 | Car plate detection localization method based on deep learning network |
| GB201912428D0 (en) * | 2019-08-29 | 2019-10-16 | Sita Information Networking Computing Uk Ltd | Luggage identification via image recognition |
| RU2706891C1 (en) * | 2019-06-06 | 2019-11-21 | Самсунг Электроникс Ко., Лтд. | Method of generating a common loss function for training a convolutional neural network for converting an image into an image with drawn parts and a system for converting an image into an image with drawn parts |
| CN110633755A (en) * | 2019-09-19 | 2019-12-31 | 北京市商汤科技开发有限公司 | Network training method, image processing method and device and electronic equipment |
| CN111461134A (en) * | 2020-05-18 | 2020-07-28 | 南京大学 | Low-resolution license plate recognition method based on generation countermeasure network |
| CN111626295A (en) * | 2020-07-27 | 2020-09-04 | 杭州雄迈集成电路技术股份有限公司 | Training method and device for license plate detection model |
| US20200301799A1 (en) * | 2019-03-23 | 2020-09-24 | Uatc, Llc | Systems and Methods for Generating Synthetic Sensor Data via Machine Learning |
-
2020
- 2020-12-23 CN CN202011532814.1A patent/CN112906699A/en active Pending
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050257748A1 (en) * | 2002-08-02 | 2005-11-24 | Kriesel Marshall S | Apparatus and methods for the volumetric and dimensional measurement of livestock |
| CN104809443A (en) * | 2015-05-05 | 2015-07-29 | 上海交通大学 | Convolutional neural network-based license plate detection method and system |
| CN106446895A (en) * | 2016-10-28 | 2017-02-22 | 安徽四创电子股份有限公司 | License plate recognition method based on deep convolutional neural network |
| US20190095730A1 (en) * | 2017-09-25 | 2019-03-28 | Beijing University Of Posts And Telecommunications | End-To-End Lightweight Method And Apparatus For License Plate Recognition |
| CN108764228A (en) * | 2018-05-28 | 2018-11-06 | 嘉兴善索智能科技有限公司 | Word object detection method in a kind of image |
| US20200301799A1 (en) * | 2019-03-23 | 2020-09-24 | Uatc, Llc | Systems and Methods for Generating Synthetic Sensor Data via Machine Learning |
| CN110020651A (en) * | 2019-04-19 | 2019-07-16 | 福州大学 | Car plate detection localization method based on deep learning network |
| RU2706891C1 (en) * | 2019-06-06 | 2019-11-21 | Самсунг Электроникс Ко., Лтд. | Method of generating a common loss function for training a convolutional neural network for converting an image into an image with drawn parts and a system for converting an image into an image with drawn parts |
| GB201912428D0 (en) * | 2019-08-29 | 2019-10-16 | Sita Information Networking Computing Uk Ltd | Luggage identification via image recognition |
| CN110633755A (en) * | 2019-09-19 | 2019-12-31 | 北京市商汤科技开发有限公司 | Network training method, image processing method and device and electronic equipment |
| CN111461134A (en) * | 2020-05-18 | 2020-07-28 | 南京大学 | Low-resolution license plate recognition method based on generation countermeasure network |
| CN111626295A (en) * | 2020-07-27 | 2020-09-04 | 杭州雄迈集成电路技术股份有限公司 | Training method and device for license plate detection model |
Non-Patent Citations (2)
| Title |
|---|
| SUMMA, S: "Gait changes after weight loss on adolescent with severe obesity after sleeve gastrectomy", SURGERY FOR OBESITY AND RELATED DISEASES, vol. 15, no. 3, pages 374 - 381, XP085685502, DOI: 10.1016/j.soard.2019.01.007 * |
| 何鎏一;杨国为;: "基于深度学习的光照不均匀文本图像的识别系统", 计算机应用与软件, no. 06, pages 196 - 223 * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113538347A (en) * | 2021-06-29 | 2021-10-22 | 中国电子科技集团公司电子科学研究院 | Image detection method and system based on efficient bidirectional path aggregation attention network |
| CN113538347B (en) * | 2021-06-29 | 2023-10-27 | 中国电子科技集团公司电子科学研究院 | Image detection method and system based on efficient bidirectional path aggregation attention network |
| CN114332844A (en) * | 2022-03-16 | 2022-04-12 | 武汉楚精灵医疗科技有限公司 | Intelligent classification application method, device, equipment and storage medium of medical image |
| CN114882492A (en) * | 2022-07-11 | 2022-08-09 | 浙江大华技术股份有限公司 | License plate recognition method, device, terminal and computer readable storage medium |
| CN114882492B (en) * | 2022-07-11 | 2022-11-22 | 浙江大华技术股份有限公司 | License plate recognition method, device, terminal and computer readable storage medium |
| CN115063786A (en) * | 2022-08-18 | 2022-09-16 | 松立控股集团股份有限公司 | High-order distant view fuzzy license plate detection method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112906699A (en) | Method for detecting and identifying enlarged number of license plate | |
| CN110390251B (en) | Image and character semantic segmentation method based on multi-neural-network model fusion processing | |
| CN109840521B (en) | Integrated license plate recognition method based on deep learning | |
| CN109815956B (en) | License plate character recognition method based on self-adaptive position segmentation | |
| CN111914797B (en) | Traffic sign identification method based on multi-scale lightweight convolutional neural network | |
| CN105868700A (en) | Vehicle type recognition and tracking method and system based on monitoring video | |
| Kaur et al. | Automatic License Plate Recognition System for Vehicles Using a CNN. | |
| CN101140625A (en) | Multiple distinguishabilitys retrogress character self-adapting recognition system and method | |
| CN112990065B (en) | Vehicle classification detection method based on optimized YOLOv5 model | |
| Dorbe et al. | FCN and LSTM based computer vision system for recognition of vehicle type, license plate number, and registration country | |
| CN111008632A (en) | License plate character segmentation method based on deep learning | |
| CN114998815B (en) | Traffic vehicle identification tracking method and system based on video analysis | |
| CN111507337A (en) | License plate recognition method based on hybrid neural network | |
| Latha et al. | Image understanding: semantic segmentation of graphics and text using faster-RCNN | |
| CN116721378A (en) | Anti-collision method based on image recognition | |
| CN114743126A (en) | Lane line sign segmentation method based on graph attention machine mechanism network | |
| Li et al. | Traffic context aware data augmentation for rare object detection in autonomous driving | |
| CN110751150A (en) | FPGA-based binary neural network license plate recognition method and system | |
| CN113239865B (en) | Deep learning-based lane line detection method | |
| Ahmed et al. | A deep learning based bangladeshi vehicle classification using fine-tuned multi-class vehicle image network (mvinet) model | |
| CN111738255A (en) | Guideboard text detection and recognition algorithm based on deep learning | |
| CN114550134A (en) | Deep learning-based traffic sign detection and identification method | |
| CN113901913A (en) | Convolution network for ancient book document image binaryzation | |
| Kumar et al. | Designing of Licensed Number Plate Recognition system using hybrid technique from neural network & template matching | |
| CN115424250A (en) | License plate recognition method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |