CN113408574A - License plate classification method, license plate classification device and computer readable storage medium - Google Patents
License plate classification method, license plate classification device and computer readable storage medium Download PDFInfo
- Publication number
- CN113408574A CN113408574A CN202110513559.4A CN202110513559A CN113408574A CN 113408574 A CN113408574 A CN 113408574A CN 202110513559 A CN202110513559 A CN 202110513559A CN 113408574 A CN113408574 A CN 113408574A
- Authority
- CN
- China
- Prior art keywords
- license plate
- classification
- feature
- image
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000000605 extraction Methods 0.000 claims abstract description 11
- 239000013598 vector Substances 0.000 claims description 46
- 238000007493 shaping process Methods 0.000 claims description 25
- 230000004927 fusion Effects 0.000 claims description 21
- 238000004590 computer program Methods 0.000 claims description 11
- 238000001514 detection method Methods 0.000 claims description 3
- 238000013527 convolutional neural network Methods 0.000 description 20
- 238000010586 diagram Methods 0.000 description 11
- 238000013135 deep learning Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 239000000284 extract Substances 0.000 description 4
- 238000011176 pooling Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000011946 reduction process Methods 0.000 description 2
- 108091026890 Coding region Proteins 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008450 motivation Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
- G06V20/625—License plates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Abstract
The application discloses a license plate classification method, a license plate classification device and a computer readable storage medium, wherein the method comprises the following steps: carrying out license plate recognition processing on the first license plate image to obtain a license plate recognition result; coding the license plate recognition result to obtain a first license plate characteristic; performing feature extraction processing on the first license plate image to obtain a second license plate feature; and processing the first license plate characteristic and the second license plate characteristic by using a classification network to obtain a first license plate classification result. By means of the method, the license plate classification accuracy can be improved.
Description
Technical Field
The present application relates to the field of image processing technologies, and in particular, to a license plate classification method, a license plate classification device, and a computer-readable storage medium.
Background
The overseas license plate classification technology refers to the country to which a license plate is input and output, and the current license plate classification technology is mainly divided into two categories: compared with the traditional image processing method, the license plate classification method based on the deep learning has higher accuracy, is suitable for various types of license plate classification and has wider application range. However, since the overseas license plate country classification task needs to acquire license plate data of each country in overseas, complete data is difficult to acquire, and the requirement on the generalization capability of the classification model is high; in addition, some countries (particularly countries with close geographical positions) have high license plate similarity and are difficult to distinguish, and some countries have various license plates, namely the problem of small inter-class difference and large intra-class difference exists, so that the classification accuracy is low.
Disclosure of Invention
The application provides a license plate classification method, a license plate classification device and a computer readable storage medium, which can improve the accuracy of license plate classification.
In order to solve the technical problem, the technical scheme adopted by the application is as follows: a license plate classification method is provided, and the method comprises the following steps: carrying out license plate recognition processing on the first license plate image to obtain a license plate recognition result; coding the license plate recognition result to obtain a first license plate characteristic; performing feature extraction processing on the first license plate image to obtain a second license plate feature; and processing the first license plate characteristic and the second license plate characteristic by using a classification network to obtain a first license plate classification result.
In order to solve the above technical problem, another technical solution adopted by the present application is: the license plate classification device comprises a memory and a processor which are connected with each other, wherein the memory is used for storing a computer program, and the computer program is used for realizing the license plate classification method in the technical scheme when being executed by the processor.
In order to solve the above technical problem, another technical solution adopted by the present application is: there is provided a computer-readable storage medium for storing a computer program, which, when executed by a processor, is configured to implement the license plate classification method of the above-mentioned technical solution.
Through the scheme, the beneficial effects of the application are that: firstly, acquiring a first license plate image, then carrying out license plate recognition processing on the first license plate image to generate a license plate recognition result, and coding the license plate recognition result to generate a first license plate characteristic; simultaneously extracting the appearance characteristics of the first license plate image to generate second license plate characteristics; finally, processing the first license plate characteristic and the second license plate characteristic by adopting a classification network to obtain a license plate classification result; the classification network provided by the application can give consideration to both the study of the appearance style and the internal format of the license plate, each character in the license plate is obtained by identifying the license plate, the internal rule of the license plate character is extracted, the license plate character can be converted into the useful characteristic for country classification, the study difficulty of the license plate content by the license plate country classification network is reduced, and the accuracy of the license plate country classification is improved; in addition, the rules of the character format of the license plate are integrated into a classification network, rather than being classified and corrected in an isolated manner, so that the license plate with complicated and complicated license plate pattern rules can be distinguished accurately.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts. Wherein:
FIG. 1(a) is a schematic of the license plate of country A;
FIG. 1(B) is a schematic diagram of a license plate of country B;
FIG. 1(C) is a schematic diagram of the license plate of country C;
fig. 2 is a schematic flowchart of an embodiment of a license plate classification method provided in the present application;
FIG. 3 is a schematic flowchart of another embodiment of a license plate classification method provided by the present application;
FIG. 4 is a schematic structural diagram of a license plate classification provided herein;
FIG. 5 is a schematic structural diagram of a Transfomer model provided herein;
FIG. 6 is a schematic diagram of another embodiment of a license plate classification provided herein;
fig. 7 is a schematic structural diagram of an embodiment of a license plate classification device provided in the present application;
FIG. 8 is a schematic structural diagram of an embodiment of a computer-readable storage medium provided in the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
To facilitate the explanation of the principle and motivation of the scheme provided by the present application, fig. 1(a) -1(C) illustrate three license plates with high similarity in different countries, fig. 1(a) illustrates a license plate of country a, fig. 1(B) illustrates a license plate of country B, and fig. 1(C) illustrates a license plate of country C, which have very high similarity in appearance with a license plate of country B, and cannot be accurately classified by using a common classification method, but the two are different in the arrangement order of english letters and numbers. The number plate of country C is different from the number plate of country A in appearance, but the arrangement of letters and numbers is the same as that of the number plate of country A. To accurately distinguish the license plates of the three countries, the appearance of the license plate and the internal character format of the license plate need to be combined at the same time; however, for overseas license plate country classification tasks with the number of classes as many as dozens or even hundreds, the format of some national license plates is complicated, and the difficulty in counting rules only through manpower is high; therefore, a qualified overseas license plate classification network needs to learn not only the overall appearance of the license plate, but also the content and arrangement rule of the license plate. In practical application, due to the limited data volume and the large task difficulty, a general classification network hardly gives consideration to key features, and a general Convolutional Neural Network (CNN) classification network usually only well learns the overall appearance of a license plate and hardly learns the content of the license plate.
In order to solve the problems, the application provides a license plate country classification method based on the feature fusion of CNN and Transformer, which uses a double-branch network: the system comprises a Transformer branch and a CNN network branch, wherein the Transformer branch is used for extracting the internal relation between license plate characters, the CNN network branch is used for extracting the key characteristics of the appearance of the license plate, the characteristics in the two aspects are fused to form more robust characteristics, and then the more robust characteristics are sent to a classification network, so that a more accurate classification result is obtained, and the scheme adopted by the application is elaborated in detail below.
Referring to fig. 2, fig. 2 is a schematic flowchart illustrating an embodiment of a license plate classification method provided in the present application, where the method includes:
step 11: and carrying out license plate recognition processing on the first license plate image to obtain a license plate recognition result.
The first license plate image can be obtained through image acquisition equipment or searched in an image database, and the image acquisition equipment can be a camera or equipment with the camera. The image acquisition device can be arranged on the vehicle or independent of the vehicle.
After the first license plate image is obtained, the license plate recognition processing can be carried out on the first license plate image by adopting the existing license plate recognition algorithm to generate a license plate recognition result; for example, taking fig. 1(a) as an example, the license plate of country a is identified, and the obtained license plate identification result is { a, B, D, 5, 0, 7 }.
Furthermore, the acquired scene image (including the license plate) can be directly used as the first license plate image, the image of the area where the license plate is located in the scene image can also be used as the first license plate image, and only the image of the area where the license plate is located is used as the first license plate image, so that the subsequent processing on the first license plate image is simpler and more effective.
Step 12: and coding the license plate recognition result to obtain a first license plate characteristic.
After the license plate recognition result corresponding to the first license plate image is obtained, the license plate recognition result can be encoded by adopting an encoding method, and the corresponding encoded feature (namely the first license plate feature) is obtained.
Step 13: and performing feature extraction processing on the first license plate image to obtain a second license plate feature.
For the acquired first license plate image, a feature extraction method (for example, CNN) may be adopted to directly perform feature extraction processing on the first license plate image, and generate a corresponding feature (i.e., a second license plate feature).
Step 14: and processing the first license plate characteristic and the second license plate characteristic by using a classification network to obtain a first license plate classification result.
After the first license plate feature and the second license plate feature are obtained, the first license plate feature and the second license plate feature can be input into a classification network trained in advance, and the classification network can perform fusion or classification and other processing on the first license plate feature and the second license plate feature to obtain a first license plate classification result; for example, taking fig. 1(a) as an example, after the processing of steps 11 to 14 is performed on the image shown in fig. 1(a), the first license plate classification result with country a of the license plate can be obtained.
The scheme provided by the embodiment mainly relates to the technical field of deep learning, in particular to technologies such as deep learning, license plate classification and natural language processing, the obtained first license plate image is directly subjected to license plate recognition processing to obtain character content of a license plate, the character content is subjected to coding processing to obtain corresponding text features, and then the text features and appearance features obtained by feature extraction of the first license plate image are fused and classified to obtain a license plate classification result; because the rules of the license plate character format are integrated into the classification network, rather than being separated, classified first and then corrected, the license plate with the complicated and complicated license plate pattern rules can be correctly distinguished; in addition, through improving the input of the classification network, the difficulty in learning license plate contents by the classification network of the license plate country is reduced, a better classification effect is obtained, and the accuracy of license plate classification is improved.
Referring to fig. 3, fig. 3 is a schematic flowchart illustrating another embodiment of a license plate classification method provided in the present application, the method including:
step 21: and acquiring an image to be processed, cutting the image to be processed, and generating a first license plate image.
Firstly, acquiring an image to be processed through image acquisition equipment, and acquiring the position of a license plate in the image to be processed through a license plate detection model; and then, intercepting an image of the position of the license plate in the image to be processed to obtain a first license plate image, wherein the first license plate image comprises the license plate.
Step 22: and recognizing characters in the first license plate image by using a license plate recognition network to obtain a license plate recognition result.
As shown in fig. 4, the first license plate image is input to a license plate recognition network, and the license plate recognition network recognizes characters (including numbers, english letters, or other special characters) in the first license plate image to generate a license plate recognition result.
Step 23: and coding the license plate recognition result by using a preset coding mode to obtain a license plate character vector sequence.
In order to convert characters in a license plate to be effective input of a transform model, this embodiment uses a one-hot encoding module to perform one-hot (one-hot) encoding on a license plate recognition result, that is, a preset encoding mode is one-hot encoding, so as to convert a character string in the license plate recognition result into a unique vector corresponding to the character string, where the encoding rule is as follows:
assuming that the character type of the overseas license plate is M, the maximum character length which may appear on the license plate is N, and the length of the characters in the license plate to be classified is S, a vector with the length of M is allocated to each character in the license plate, and because the length of the vector is the same as the number of the character types, each position in the vector is just related to one character. For a character, setting the value of the corresponding position in the vector to 1 and the values of the remaining positions to 0, a unique vector can be generated to represent the character. For a license plate with S characters, S vectors with the length of M can be generated, then the S vectors are supplemented, all the (N-S) vectors with the values of 0 and the length of M are generated, and finally N vectors with the length of M are obtained, namely the sequence of the license plate character vectors is a vector with the length of N multiplied by M, and the vector is a coding sequence corresponding to a license plate number, the coded sequence of the license plate character vectors is similar to a text sequence, and can be directly sent to a Transformer model, as shown in figure 4.
For example, assuming that the maximum character length N of the license plate is 4, the number M of character types is 3 (taking letters "a", "B", and "C" as an example), and a license plate number "BAA" is given, three vectors with a length of 3 are generated to sequentially represent characters "B", "a", and "a", where the vector corresponding to the character "B" is [0, 1, 0], and the vector corresponding to the character "a" is [1, 0, 0], and since the character length of the license plate is 1 smaller than the maximum character length, an empty vector [0, 0, 0] needs to be supplemented, and finally the encoded license plate character vector sequence corresponding to the license plate "BAA" is: { [0, 1, 0 ]; [1, 0, 0 ]; [1, 0, 0 ]; [0,0,0]}.
Step 24: and processing the license plate character vector sequence by using a Transformer model to obtain a first license plate characteristic.
The Transformer model comprises an encoding module and a decoding module, and as shown in fig. 5, firstly, the encoding module is used for encoding the license plate character vector sequence to obtain an encoded license plate character vector; and then decoding the coded license plate character vector by using a decoding module to obtain a first license plate characteristic.
Further, the encoding module is composed of several encoders connected in series, the decoding module is composed of a corresponding number of decoders, that is, the number of encoders is the same as that of decoders, fig. 5 illustrates that the number of encoders and decoders is 2, the structures of all encoders are the same, each encoder includes a first self-attention layer and a first feedforward network, an input vector passes through the first self-attention layer, data output by the first self-attention layer is transmitted to the first feedforward neural network, then enters a next encoder, the output of the last encoder is transmitted to each decoder, and passes through a second self-attention layer, a second encoding and decoding attention layer and a second feedforward neural network in sequence in the decoders, and an output vector obtained finally is a feature vector containing character rules in the license plate.
Step 25: and performing feature extraction processing on the first license plate image to obtain a second license plate feature.
As shown in fig. 4, the CNN network may be used to process the first license plate image, extract the features in the first license plate image, and generate the second license plate features; specifically, in the branch where the CNN network is located (i.e., the CNN network branch), any advanced backbone network may be used to extract the appearance features of the first license plate image, in the CNN network branch, a feature map of the license plate is obtained at the end of the network, and both need to be adjusted in dimension through a shaping network in order to be fused with a feature vector output by a transform model, and the specific scheme is shown in steps 26 to 27.
Step 26: and performing dimension reduction processing on the first license plate characteristic by using a first shaping network to obtain the dimension-reduced first license plate characteristic.
As shown in fig. 4, the classification network includes a first shaping network connected to the Transformer model, and the first shaping network receives the first license plate feature output by the Transformer model, performs a dimension reduction process on the first license plate feature, and inputs the processed feature into the feature fusion layer.
Step 27: and performing dimension reduction processing on the second license plate characteristic by using a second shaping network to obtain the dimension-reduced second license plate characteristic.
As shown in fig. 4, the classification network further includes a second shaping network connected to the CNN network, where the second shaping network receives a second license plate feature output by the CNN network, performs a dimension reduction process on the second license plate feature, so that a dimension of the first license plate feature after the dimension reduction is equal to a dimension of the second license plate feature after the dimension reduction, and inputs the second license plate feature after the dimension reduction into the feature fusion layer, that is, dimensions of two paths of features input into the feature fusion layer are equal.
It is understood that it is also possible to provide only one of the first shaping network and the second shaping network, and to provide only the first shaping network when the dimension of the first license plate feature is greater than the dimension of the second license plate feature, and to reduce the dimension of the first license plate feature to be equal to the dimension of the second license plate feature using the first shaping network; when the dimension of the first license plate feature is smaller than the dimension of the second license plate feature, only the second shaping network is set, and the dimension of the second license plate feature is reduced to be equal to the dimension of the first license plate feature by the second shaping network.
Step 28: and performing fusion processing on the first license plate characteristic and the second license plate characteristic by using the characteristic fusion layer to obtain a fusion license plate characteristic.
As shown in fig. 4, the classification network further includes a feature fusion layer, where the feature fusion layer receives the features output by the first shaping network and the second shaping network, and fuses the features to generate new features (i.e., fusion license plate features); for example, the dimension of the feature output by the first shaping network is 1 × 512, the dimension of the feature output by the second shaping network is 1 × 512, and the dimension of the feature of the fused license plate is 1 × 512.
Step 29: and classifying the fusion license plate features by using a classification layer to obtain a first license plate classification result.
As shown in fig. 4, the classification layer receives the fusion license plate features output by the feature fusion layer, classifies the fusion license plate features, and generates a corresponding classification result, thereby determining the country of the license plate to be currently classified; in particular, the classification layer may consist of a fully connected layer.
In a specific embodiment, as shown in fig. 6, the first license plate image is the image shown in fig. 1(a), the first shaping network is a first fully-connected layer, the second shaping network is an average pooling layer, and the classification layer is a second fully-connected layer.
And reducing the dimension of the characteristic diagram to 1 x 512 by adopting an average pooling method for the characteristic diagram output by the CNN network, converting the characteristic vector output by the Transformer model to the fixed size of 1 x 512 through a first fully-connected layer, fusing and adding the characteristic vector and the characteristic vector by using a characteristic fusion layer to obtain a new characteristic vector, sending the new characteristic vector to a second fully-connected layer, and finally outputting the country of the license plate.
The classification network provided by the embodiment is a multi-input overseas license plate country classification network fusing CNN and Transformer characteristics, the network can give consideration to learning of license plate appearance styles and license plate internal formats, license plate character contents are encoded into Transformer acceptable input through one-hot encoding, internal rules of license plate characters are extracted through the Transformer, the license plate characters are converted into useful characteristics for country classification, and the accuracy of license plate country classification is improved.
It can be understood that before the classification network is used, the classification network needs to be trained to ensure the accuracy of classification, and specifically, the classification network can be trained through the following steps:
a. obtaining classification training data
a1) And acquiring a classification training image.
A second license plate image may be captured with the camera, the second license plate image being an image including a license plate.
a2) And carrying out license plate recognition processing on the classified training images to obtain a second license plate recognition result.
After the classification training image is obtained, the classification training image can be recognized by adopting a license plate recognition network to generate a corresponding license plate recognition result (recorded as a second license plate recognition result).
a3) And coding the second license plate recognition result to obtain a third license plate characteristic.
And coding the second license plate recognition result by adopting a one-hot coding mode to obtain a license plate character vector sequence, and then processing the license plate character vector sequence by adopting a Transfomer model, thereby extracting text characteristics of the second license plate image and recording the text characteristics as third license plate characteristics.
a4) And performing feature extraction processing on the classified training images to obtain a fourth license plate feature.
And extracting the features in the classification training image by adopting the trained CNN network to generate a fourth license plate feature. It can be understood that if the classification training image further includes contents of other non-license plates, the license plate position is obtained through a license plate detection algorithm, and then an image corresponding to the license plate position is captured and taken as an image input to a license plate recognition network and a CNN network, so that the complexity of calculation is reduced, and the accuracy of recognition and the efficiency of feature extraction are improved.
And repeating the steps a1-a4 to obtain enough classification training data for classifying the countries, wherein the classification training data comprise a plurality of groups of training features, and each group of training features comprises a third license plate feature and a corresponding fourth license plate feature.
b. Training classification networks using classification training data
b1) A set of training features is selected from the classified training data as current training features.
A set of training features can be selected from the classification training data according to a set sequence or randomly as training features currently input to the classification network to train the whole classification network, wherein the training features comprise a third license plate feature and a corresponding fourth license plate feature.
b2) And processing the third license plate feature and the fourth license plate feature in the current training features by using a classification network to obtain a second license plate classification result.
After the third license plate feature and the fourth license plate feature are obtained, the two input features can be processed by adopting a classification network to obtain corresponding classification results; specifically, as shown in fig. 4, the classification network includes a first shaping network, a second shaping network, a feature fusion layer and a classification layer, and the specific functions and functions thereof have been described in the above embodiments and are not described herein again.
b3) And judging that the classification accuracy of the classification network exceeds a preset threshold value.
In order to determine when to terminate the training, the classification accuracy of the current classification network may be counted, and the magnitude relationship between the classification accuracy and the preset threshold may be determined.
b4) And if the classification accuracy of the classification network does not exceed the preset threshold value, adjusting the parameters of the classification network based on the second license plate classification result.
If the classification accuracy of the current classification network does not exceed the preset threshold, the current classification accuracy is not qualified, at this time, the parameters of the classification network can be adjusted based on the second license plate classification result, and then the step of selecting a group of training features from the classification training data as the current training features is returned, namely the step b1 is executed until the classification accuracy of the classification network exceeds the preset threshold.
b5) And if the classification accuracy of the classification network exceeds a preset threshold value, stopping training and outputting the classification network.
If the classification accuracy of the current classification network is detected to exceed the preset threshold, the classification accuracy of the current classification network is indicated to be high, the preset requirement is met, at the moment, the training can be stopped, and the trained classification network model is output.
The embodiment provides a new license plate country classification method based on deep learning, which considers the license plate style characteristics of each country, carries out one-hot coding on the license plate character content, codes each character into a unique characteristic vector, inputs the coded vector into a Transformer branch, directly extracts the text characteristics, sends the original license plate image into a CNN network to extract the appearance characteristics, maps the output of the two branches to the same dimensionality and fuses through an average pooling layer and a full connecting layer, and then classifies through a classification layer consisting of the full connecting layer to obtain the country classification result; the Transformer branch can directly extract the text characteristics of the license plate characters, and better finds the relation between the license plate characters and the country, the method can provide effective prior knowledge about the license plate content for a classification network, and reduces the learning difficulty of the CNN network on the internal format characteristics of the license plate, thereby improving the classification accuracy.
Referring to fig. 7, fig. 7 is a schematic structural diagram of an embodiment of a license plate classification device 70 provided in the present application, where the license plate classification device 70 includes a memory 71 and a processor 72 connected to each other, the memory 71 is used for storing a computer program, and the computer program is used for implementing the license plate classification method in the foregoing embodiment when being executed by the processor 72.
The embodiment provides a foreign license plate country classification scheme based on the fusion of CNN and Transformer characteristics, which fuses the characteristics of CNN and Transfermer, codes license plate characters by using one-hot, each character has a unique vector to represent, and then directly extracts text characteristics by using a Transformer model.
Referring to fig. 8, fig. 8 is a schematic structural diagram of an embodiment of a computer-readable storage medium 80 provided in the present application, where the computer-readable storage medium 80 is used for storing a computer program 81, and the computer program 81 is used for implementing the license plate classification method in the foregoing embodiment when being executed by a processor.
The computer readable storage medium 80 may be a server, a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and various media capable of storing program codes.
In the several embodiments provided in the present application, it should be understood that the disclosed method and apparatus may be implemented in other manners. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of modules or units is merely a logical division, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed.
Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The above description is only an example of the present application and is not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings, or which are directly or indirectly applied to other related technical fields, are intended to be included within the scope of the present application.
Claims (12)
1. A license plate classification method is characterized by comprising the following steps:
carrying out license plate recognition processing on the first license plate image to obtain a license plate recognition result;
coding the license plate recognition result to obtain a first license plate characteristic;
performing feature extraction processing on the first license plate image to obtain a second license plate feature;
and processing the first license plate characteristic and the second license plate characteristic by utilizing a classification network to obtain a first license plate classification result.
2. The method for classifying a license plate according to claim 1, wherein the step of encoding the license plate recognition result to obtain a first license plate feature comprises:
coding the license plate recognition result by using a preset coding mode to obtain a license plate character vector sequence;
and processing the license plate character vector sequence by using a Transformer model to obtain the first license plate characteristic.
3. The method of classifying a license plate according to claim 2,
the preset encoding mode is single-hot encoding, and the license plate character vector sequence is an NxM vector, wherein N is the maximum character length of the license plate, and M is the character type of the license plate.
4. The method for classifying a license plate of claim 2, wherein the Transformer model includes an encoding module and a decoding module, and the step of processing the license plate character vector sequence by using the Transformer model to obtain the first license plate feature includes:
the license plate character vector sequence is encoded by the encoding module to obtain an encoded license plate character vector;
and decoding the coded license plate character vector by using the decoding module to obtain the first license plate characteristic.
5. The method for classifying license plates according to claim 1, wherein the classification network includes a feature fusion layer and a classification layer, and the step of processing the first license plate feature and the second license plate feature by using the classification network to obtain a first license plate classification result includes:
fusing the first license plate feature and the second license plate feature by using the feature fusion layer to obtain a fused license plate feature;
and classifying the fusion license plate features by using the classification layer to obtain a first license plate classification result.
6. The license plate classification method of claim 5, wherein the classification network further comprises a first shaping network and a second shaping network, and before the step of fusing the first license plate feature and the second license plate feature by using the feature fusion layer to obtain the fused license plate feature, the method further comprises:
performing dimension reduction processing on the first license plate feature by using the first shaping network to obtain a dimension-reduced first license plate feature;
performing dimension reduction processing on the second license plate feature by using the second shaping network to obtain a dimension-reduced second license plate feature;
wherein the dimension of the first reduced license plate feature is equal to the dimension of the second reduced license plate feature.
7. The method for classifying a license plate according to claim 1, wherein the step of performing license plate recognition processing on the first license plate image to obtain a license plate recognition result is preceded by:
acquiring an image to be processed;
cutting the image to be processed to generate the first license plate image;
wherein the first license plate image includes a license plate.
8. The license plate classification method of claim 7, wherein the step of cropping the image to be processed to generate the first license plate image comprises:
acquiring the position of the license plate in the image to be processed through a license plate detection model;
and intercepting the image of the position of the license plate in the image to be processed to obtain the first license plate image.
9. The method for classifying a license plate according to claim 1, wherein the step of performing license plate recognition processing on the first license plate image to obtain a license plate recognition result comprises:
and recognizing characters in the first license plate image by using a license plate recognition network to obtain a license plate recognition result.
10. The method for classifying a license plate according to claim 1, wherein the step of performing license plate recognition processing on the first license plate image to obtain a license plate recognition result is preceded by:
acquiring a classification training image;
carrying out license plate recognition processing on the classified training images to obtain a second license plate recognition result;
coding the second license plate recognition result to obtain a third license plate characteristic;
carrying out feature extraction processing on the classified training images to obtain a fourth license plate feature;
repeatedly executing the steps to obtain classified training data, wherein the classified training data comprise a plurality of groups of training features, and the training features comprise the third license plate features and the corresponding fourth license plate features;
selecting a set of the training features from the classified training data as current training features;
processing the third license plate feature and the fourth license plate feature in the current training features by using the classification network to obtain a second license plate classification result;
adjusting parameters of the classification network based on the second card classification result;
and returning to the step of selecting a group of training features from the classification training data as the current training features until the classification accuracy of the classification network exceeds a preset threshold.
11. A license plate classification device comprising a memory and a processor connected to each other, wherein the memory is configured to store a computer program, and the computer program is configured to implement the license plate classification method according to any one of claims 1 to 10 when executed by the processor.
12. A computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, is configured to implement the license plate classification method of any one of claims 1 to 10.
Priority Applications (4)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110513559.4A CN113408574B (en) | 2021-05-11 | 2021-05-11 | License plate classification method, license plate classification device and computer readable storage medium |
| EP21941594.0A EP4323915A1 (en) | 2021-05-11 | 2021-09-14 | License plate classification method, license plate classification apparatus, and computer-readable storage medium |
| PCT/CN2021/118340 WO2022237027A1 (en) | 2021-05-11 | 2021-09-14 | License plate classification method, license plate classification apparatus, and computer-readable storage medium |
| US18/388,938 US20240087343A1 (en) | 2021-05-11 | 2023-11-13 | License plate classification method, license plate classification apparatus, and computer-readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110513559.4A CN113408574B (en) | 2021-05-11 | 2021-05-11 | License plate classification method, license plate classification device and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113408574A true CN113408574A (en) | 2021-09-17 |
| CN113408574B CN113408574B (en) | 2022-06-07 |
Family
ID=77678393
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110513559.4A Active CN113408574B (en) | 2021-05-11 | 2021-05-11 | License plate classification method, license plate classification device and computer readable storage medium |
Country Status (4)
| Country | Link |
|---|---|
| US (1) | US20240087343A1 (en) |
| EP (1) | EP4323915A1 (en) |
| CN (1) | CN113408574B (en) |
| WO (1) | WO2022237027A1 (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114332447A (en) * | 2022-03-14 | 2022-04-12 | 浙江大华技术股份有限公司 | License plate correction method, license plate correction device and computer readable storage medium |
| WO2022237027A1 (en) * | 2021-05-11 | 2022-11-17 | Zhejiang Dahua Technology Co., Ltd. | License plate classification method, license plate classification apparatus, and computer-readable storage medium |
| CN116935373A (en) * | 2023-09-18 | 2023-10-24 | 珠海亿智电子科技有限公司 | Training method, device, equipment and storage medium for multi-type license plate recognition model |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115661807B (en) * | 2022-12-28 | 2023-04-07 | 成都西物信安智能系统有限公司 | Method for acquiring license plate information |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111222513A (en) * | 2019-12-31 | 2020-06-02 | 深圳云天励飞技术有限公司 | License plate number recognition method and device, electronic equipment and storage medium |
| CN112257764A (en) * | 2020-10-15 | 2021-01-22 | 浙江大华技术股份有限公司 | License plate classification method, electronic equipment and storage medium |
| CN112381129A (en) * | 2020-11-10 | 2021-02-19 | 浙江大华技术股份有限公司 | License plate classification method and device, storage medium and electronic equipment |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8644561B2 (en) * | 2012-01-18 | 2014-02-04 | Xerox Corporation | License plate optical character recognition method and system |
| US10963719B1 (en) * | 2019-03-04 | 2021-03-30 | Ccc Information Services Inc. | Optimized vehicle license plate recognition |
| CN113408574B (en) * | 2021-05-11 | 2022-06-07 | 浙江大华技术股份有限公司 | License plate classification method, license plate classification device and computer readable storage medium |
-
2021
- 2021-05-11 CN CN202110513559.4A patent/CN113408574B/en active Active
- 2021-09-14 EP EP21941594.0A patent/EP4323915A1/en active Pending
- 2021-09-14 WO PCT/CN2021/118340 patent/WO2022237027A1/en active Application Filing
-
2023
- 2023-11-13 US US18/388,938 patent/US20240087343A1/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111222513A (en) * | 2019-12-31 | 2020-06-02 | 深圳云天励飞技术有限公司 | License plate number recognition method and device, electronic equipment and storage medium |
| CN112257764A (en) * | 2020-10-15 | 2021-01-22 | 浙江大华技术股份有限公司 | License plate classification method, electronic equipment and storage medium |
| CN112381129A (en) * | 2020-11-10 | 2021-02-19 | 浙江大华技术股份有限公司 | License plate classification method and device, storage medium and electronic equipment |
Non-Patent Citations (1)
| Title |
|---|
| 爱编程真是太好了: "Transformer模型详解", 《CSDN》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022237027A1 (en) * | 2021-05-11 | 2022-11-17 | Zhejiang Dahua Technology Co., Ltd. | License plate classification method, license plate classification apparatus, and computer-readable storage medium |
| CN114332447A (en) * | 2022-03-14 | 2022-04-12 | 浙江大华技术股份有限公司 | License plate correction method, license plate correction device and computer readable storage medium |
| CN114332447B (en) * | 2022-03-14 | 2022-08-09 | 浙江大华技术股份有限公司 | License plate correction method, license plate correction device and computer readable storage medium |
| CN116935373A (en) * | 2023-09-18 | 2023-10-24 | 珠海亿智电子科技有限公司 | Training method, device, equipment and storage medium for multi-type license plate recognition model |
| CN116935373B (en) * | 2023-09-18 | 2024-03-12 | 珠海亿智电子科技有限公司 | Training method, device, equipment and storage medium for multi-type license plate recognition model |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022237027A1 (en) | 2022-11-17 |
| EP4323915A1 (en) | 2024-02-21 |
| US20240087343A1 (en) | 2024-03-14 |
| CN113408574B (en) | 2022-06-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113408574B (en) | License plate classification method, license plate classification device and computer readable storage medium | |
| EP3471023B1 (en) | Offline identity authentication method and apparatus | |
| CN112818951B (en) | Ticket identification method | |
| CN110188829B (en) | Neural network training method, target recognition method and related products | |
| CN115471851A (en) | Burma language image text recognition method and device fused with double attention mechanism | |
| CN113723330A (en) | Method and system for understanding chart document information | |
| CN109766918A (en) | Conspicuousness object detecting method based on the fusion of multi-level contextual information | |
| CN114283350A (en) | Visual model training and video processing method, device, equipment and storage medium | |
| Tang et al. | HRCenterNet: An anchorless approach to Chinese character segmentation in historical documents | |
| CN112686219B (en) | Handwritten text recognition method and computer storage medium | |
| CN113762326A (en) | Data identification method, device and equipment and readable storage medium | |
| CN111242114A (en) | Character recognition method and device | |
| CN112215076B (en) | Deep handwriting identification method and device based on double-tower network | |
| CN115565178A (en) | Font identification method and apparatus | |
| CN116090463A (en) | Business card information extraction system training method and device and storage medium | |
| CN114863442A (en) | Character recognition method, system, device and storage medium based on deep learning | |
| CN114612965A (en) | Face living body detection method, system and medium | |
| Huang et al. | Separating Chinese character from noisy background using GAN | |
| CN113609292A (en) | Known false news intelligent detection method based on graph structure | |
| CN112784838A (en) | Hamming OCR recognition method based on locality sensitive hashing network | |
| CN116311275B (en) | Text recognition method and system based on seq2seq language model | |
| Choubey et al. | Bilateral Partitioning based character recognition for Vehicle License plate | |
| Sukanya et al. | A Tool for Extracting Text from Scanned Documents and Convert it into Editable Format | |
| AlKhateeb et al. | Interactive knowledge discovery for baseline estimation and word segmentation in handwritten Arabic text | |
| CN113111882B (en) | Card identification method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |