CN114155362A - License plate positioning method and device for license plate recognition and storage medium - Google Patents

License plate positioning method and device for license plate recognition and storage medium Download PDF

Info

Publication number
CN114155362A
CN114155362A CN202111443006.2A CN202111443006A CN114155362A CN 114155362 A CN114155362 A CN 114155362A CN 202111443006 A CN202111443006 A CN 202111443006A CN 114155362 A CN114155362 A CN 114155362A
Authority
CN
China
Prior art keywords
license plate
recognized
corner
network
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111443006.2A
Other languages
Chinese (zh)
Inventor
何翔
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chengdu Xinchao Media Group Co Ltd
Original Assignee
Chengdu Xinchao Media Group Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chengdu Xinchao Media Group Co Ltd filed Critical Chengdu Xinchao Media Group Co Ltd
Priority to CN202111443006.2A priority Critical patent/CN114155362A/en
Publication of CN114155362A publication Critical patent/CN114155362A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a license plate positioning method, a license plate positioning device and a storage medium for license plate identification, wherein the method comprises the following steps: acquiring an image to be recognized, wherein the image to be recognized at least comprises a license plate to be recognized; inputting an image to be recognized into a license plate positioning model for positioning detection to obtain 4 license plate corner coordinates corresponding to the license plate to be recognized so as to complete license plate positioning in the image to be recognized according to the 4 license plate corner coordinates, wherein the license plate positioning model comprises a MobileNet V1-0.25 network, an FPN network, an SSH network and a head network; the invention can directly obtain 4 angular points of the license plate, realizes end-to-end license plate positioning, avoids the problems of large error and low precision existing in target detection and image processing technologies, adopts the MobileNet V1-0.25 network as the backbone network, realizes license plate positioning under the condition of almost no loss of precision, reduces the parameters and the calculated quantity of an integral model, provides conditions for terminal deployment of a license plate recognition system, and is suitable for large-scale popularization and application.

Description

License plate positioning method and device for license plate recognition and storage medium
Technical Field
The invention belongs to the technical field of license plate recognition, and particularly relates to a license plate positioning method and device for license plate recognition and a storage medium.
Background
With the development of computer vision, digital image processing technology and intelligent transportation technology, the application of license plate recognition technology in the field of intelligent transportation is more and more extensive, such as scenes of parking lot gates, highway toll stations, traffic violation recognition and the like, and in the license plate recognition technology, license plate positioning and license plate correction are the primary problems and are also important factors influencing license plate recognition accuracy.
The existing license plate positioning technology mainly comprises a traditional image processing algorithm and a target detection algorithm based on deep learning, wherein the license plate recognition technology based on the traditional image processing algorithm is mainly used for positioning and segmenting a license plate by means of image binaryzation, license plate color characteristics, license plate edge characteristics and the like, and then correcting the license plate through license plate character inclination characteristics; target detection algorithms based on deep learning, such as a target detection system based on a Single Neural network proposed by Joseph Redmon and Ali faradai in 2015, SSD (Single Shot multi box Detector), an object detection network, and R-CNN (Convolutional Neural network), which is the first algorithm to successfully apply deep learning to target detection, are developed rapidly in the detection field and are applied to abundant theoretical supports and empirical references for license plate positioning.
However, the two positioning methods have the following disadvantages: because the number plate patterns and colors of China are numerous, patterns such as gradual change of green, blue, black and white exist, and are influenced by factors such as illumination in an actual scene, the traditional image processing algorithm has high difficulty in positioning the number plate, insufficient precision and insufficient robustness; in addition, in an actual scene, an uncertain angle exists between a license plate recognition camera and a vehicle, the license plate positioned by a target detection algorithm has a tilt problem, the post-processing such as tilt correction, interference filtering and the like needs to be carried out by combining an image technology or a deep learning technology, and due to interference influence, the high-precision license plate positioning is difficult to achieve in practice, so that the license plate recognition precision is not high; therefore, a high-precision license plate positioning method is urgently provided.
Disclosure of Invention
The invention aims to provide a license plate positioning method, a license plate positioning device and a storage medium for license plate recognition, and aims to solve the problem of low precision of the existing license plate positioning method.
In order to achieve the purpose, the invention adopts the following technical scheme:
the invention provides a license plate positioning method for license plate recognition, which comprises the following steps:
acquiring an image to be recognized, wherein the image to be recognized at least comprises a license plate to be recognized;
inputting the image to be recognized into a license plate positioning model for positioning detection to obtain 4 license plate corner coordinates corresponding to the license plate to be recognized so as to complete license plate positioning in the image to be recognized according to the 4 license plate corner coordinates, wherein the license plate positioning model comprises a MobileNet V1-0.25 network, an FPN network, an SSH network and a head network;
the MobileNet V1-0.25 network is used as a main feature extraction network of the license plate positioning model and is used for carrying out feature extraction on the image to be recognized under three receptive field conditions to obtain three first feature maps;
the FPN network is used for carrying out up-sampling and feature fusion on the three first feature maps to obtain three second feature maps;
the SSH network comprises three parallel convolutional layers and is used for carrying out convolution processing on the three second feature maps by utilizing the three parallel convolutional layers to obtain three third feature maps;
the head network is used for predicting license plate key points of the three third feature maps so as to output 8-dimensional key point vectors, and 4 license plate corner point coordinates are obtained according to the key point vectors.
Based on the disclosure, the invention uses the MobileNet V1-0.25 network, the FPN network, the SSH network and the head network to construct a license plate positioning model in advance, so as to input the image to be recognized into the model for feature extraction, namely, firstly uses the MobileNet V1-0.25 network to extract the main features, then uses the FPN network to perform feature fusion on the extracted main features to obtain a feature map with richer feature information, then uses three parallel convolution layers in the SSH network to perform multi-scale detection on the feature map output by the FPN network, further enhances the feature extraction to enhance the context information of the extracted features, thereby improving the extracted license plate feature quality and achieving the purpose of improving the license plate positioning precision, and finally uses the head network to perform key point prediction and output 8-dimensional key point vectors, namely, the invention changes the branch output dimension of the key points in the model from 10 dimensions to 8 dimensions, therefore, the invention can directly acquire 4 angular points of the license plate, realizes end-to-end license plate positioning, avoids the problems of large error and low precision of target detection and image processing technologies, and further improves license plate positioning precision.
Through the design, the invention can directly obtain 4 angular points of the license plate, realizes end-to-end license plate positioning, avoids the problems of large error and low precision of target detection and image processing technologies, adopts the MobileNet V1-0.25 network as a backbone network, realizes license plate positioning under the condition of almost no loss of precision, reduces the parameter and the calculated amount of an integral model, provides conditions for terminal deployment of a license plate recognition system, and is suitable for large-scale popularization and application.
In one possible design, the loss function of the license plate location model is:
Figure BDA0003383987140000021
in the above formula, L represents a loss function of the license plate location model, LclsRepresenting the target classification loss function, LboxRepresents the regression loss function of the detection box, LptsRepresenting the regression loss function of the key points, piRepresenting the probability that anchor point i is predicted as a license plate corner,
Figure BDA0003383987140000022
a true tag denoted as a positive anchor at 1,
Figure BDA0003383987140000023
true tag representing a negative anchor at 0, tiRepresenting the predicted coordinates of the prediction box in the anchor box,
Figure BDA0003383987140000024
representing the true coordinates of the predicted frame in the anchor frame, liRepresenting the coordinates of the corner points of the license plate predicted in the anchor point frame,
Figure BDA0003383987140000025
indicating the coordinates of the marked corner points, λ, of the license plate1、λ2、λ3And λ4Represents a weight coefficient, and is a constant;
Lpts1representing a corner point distance constraint function which is used for constraining the distance between two parallel edges of the license plate to be recognized;
Lpts2and representing a corner distance proportion constraint function for constraining the proportion of the length and the width of the license plate to be recognized.
In one possible design, the corner distance constraint function is:
Lpts1(li)＝|||l0l1|-|l2l3|||+|||l0l2|-|l1l3|||
in the above formula, /)0l1Represents the distance between the first license plate corner point and the second license plate corner point, l2l3Represents the distance between the third license plate corner point and the fourth license plate corner point, l0l2Represents the distance between the first license plate corner point and the third license plate corner point, l1l3And representing the distance between the second license plate corner point and the fourth license plate corner point, wherein the upper left corner of the license plate to be identified is the first license plate corner point, the upper right corner is the second license plate corner point, the lower left corner is the third license plate corner point, and the lower right corner is the fourth license plate corner point.
In one possible design, the corner distance proportional constraint function is:
Figure BDA0003383987140000031
in the above-mentioned formula, the first and second groups,
Figure BDA0003383987140000032
the distance between the first license plate marking angular point and the second license plate marking angular point is represented,
Figure BDA0003383987140000033
and representing the distance between the first license plate marking angular point and the third license plate marking angular point.
Based on the disclosure, the invention adds an angular point distance constraint function and an angular point distance proportion constraint function on the basis of the traditional Retinaface network, wherein the angular point distance constraint function is used for constraining the distance of the parallel sides of the license plate to be identified, namely the distance of the two parallel sides is equal, namely the value of the angular point distance constraint function is constrained to 0 or approximate to 0, and the angular point distance proportion constraint function is used for constraining a predicted point by using a marking point, namely the length and width proportion of the license plate obtained according to the marking angular point of the license plate is equal to the length and width proportion of the license plate obtained according to the coordinates of the angular point of the license plate, namely the angular point distance proportion constraint function is constrained to 0 or approximate to 0; therefore, the positioning precision of the corner points of the license plate can be further improved.
In one possible design, the weighting factor λ1、λ2、λ3And λ4The values of (a) are 0.25, 0.5, 0.01 and 0.01, respectively.
Through the design, the invention enables the angular point iteration to be more accurate by increasing the weight of each loss function, and further improves the angular point positioning precision.
In one possible design, the prior anchor of the license plate location model has the size: 16 x8, 32 x 16, 64 x 32, 128 x 64, 256 x 128 and 512 x 256, wherein the a priori anchors are used to characterize the anchor boxes where the license plate is located.
Based on the disclosure, the length and the width of the anchor point frame are improved, so that the anchor point frame is closer to the actual size of the license plate, and the angular point positioning precision can be further improved.
In one possible design, after the image to be recognized is input into a license plate positioning model for positioning detection to obtain 4 license plate corner coordinates corresponding to the license plate to be recognized, the method further comprises:
acquiring license plate standard corner coordinates in the image to be recognized;
performing perspective transformation processing on the image to be recognized by using the license plate standard corner coordinates and the 4 license plate corner coordinates to obtain a corrected image to be recognized, wherein the corrected image to be recognized only contains the license plate to be recognized;
and inputting the corrected image to be recognized into a license plate recognition model to obtain a license plate recognition result.
Based on the above disclosure, the invention combines the license plate corner coordinates output by the license plate positioning model and the license plate standard corner coordinates (i.e. coordinates preset by the user and specifically set according to the size to be converted), performs perspective conversion processing on the image to be recognized, can realize the inclination correction of the license plate, and enables the corrected image to only contain the license plate to be recognized, thereby improving the license plate recognition precision.
In a second aspect, the present invention provides a license plate positioning device for license plate recognition, including:
the device comprises an acquisition unit, a recognition unit and a recognition unit, wherein the acquisition unit is used for acquiring an image to be recognized, and the image to be recognized at least comprises a license plate to be recognized;
and the license plate positioning unit is used for inputting the image to be recognized into a license plate positioning model for positioning detection to obtain 4 license plate corner coordinates corresponding to the license plate to be recognized so as to complete license plate positioning in the image to be recognized according to the 4 license plate corner coordinates, wherein the license plate positioning model comprises a MobileNet V1-0.25 network, a FPN network, an SSH network and a head network.
In a third aspect, the present invention provides another license plate location apparatus for license plate recognition, taking the apparatus as a computer main device as an example, and including a memory, a processor and a transceiver, which are sequentially connected in a communication manner, where the memory is used to store a computer program, the transceiver is used to transmit and receive messages, and the processor is used to read the computer program and execute the license plate location method for license plate recognition as described in any one of the first aspect or the first aspect.
In a fourth aspect, the present invention provides a storage medium having instructions stored thereon, where the instructions, when executed on a computer, perform the license plate location method for license plate recognition as described in the first aspect or any one of the possible designs of the first aspect.
In a fifth aspect, the present invention provides a computer program product comprising instructions which, when run on a computer, cause the computer to perform the license plate location method for license plate recognition as described in the first aspect or any one of the possible designs of the first aspect.
Drawings
FIG. 1 is a schematic structural diagram of a license plate location model provided by the present invention;
FIG. 2 is a schematic flow chart illustrating steps of a license plate location method for license plate recognition according to the present invention;
FIG. 3 is a schematic structural diagram of an SSH network provided by the present invention;
FIG. 4 is a schematic diagram illustrating labeling of 4 license plate corner points on a license plate to be recognized according to the present invention;
FIG. 5 is a schematic diagram of a license plate positioning effect of a license plate to be recognized according to the present invention;
FIG. 6 is a flowchart illustrating the steps of license plate recognition provided by the present invention;
FIG. 7 is a schematic structural diagram of a license plate positioning device for license plate recognition according to the present invention;
fig. 8 is a schematic structural diagram of a computer main device provided in the present invention.
Detailed Description
The invention is further described with reference to the following figures and specific embodiments. It should be noted that the description of the embodiments is provided to help understanding of the present invention, but the present invention is not limited thereto. Specific structural and functional details disclosed herein are merely illustrative of example embodiments of the invention. This invention may, however, be embodied in many alternate forms and should not be construed as limited to the embodiments set forth herein.
It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of example embodiments of the present invention.
It should be understood that, for the term "and/or" as may appear herein, it is merely an associative relationship that describes an associated object, meaning that three relationships may exist, e.g., a and/or B may mean: a exists alone, B exists alone, and A and B exist at the same time; for the term "/and" as may appear herein, which describes another associative object relationship, it means that two relationships may exist, e.g., a/and B, may mean: a exists independently, and A and B exist independently; in addition, for the character "/" that may appear herein, it generally means that the former and latter associated objects are in an "or" relationship.
Examples
Referring to fig. 1, a license plate location model is provided for an application, and the license plate location model is improved on a conventional Retinaface detection network, and includes: the vehicle license plate positioning model comprises a MobileNet V1-0.25 network, an FPN network, an SSH network and a head network, wherein the MobileNet V1-0.25 network is used as a main feature extraction network for carrying out feature extraction on an input image to be identified under three receptive field conditions to obtain three first feature maps (namely extracting feature information under different dimensions to obtain feature maps under different dimensions), and the FPN network is used for carrying out up-sampling and feature fusion on the three first feature maps to obtain three second feature maps (namely obtaining feature maps with more abundant feature information through feature fusion); meanwhile, the SSH network in the model is provided with three parallel convolution layers, namely, the three parallel convolution layers are utilized to respectively carry out convolution processing on the three second feature maps so as to obtain three third feature maps (the essence is that the convolution processing is utilized to carry out multi-scale feature detection, so that the context information of the extracted features is enhanced, the quality of the extracted features is improved, and the angular point positioning precision is further improved); finally, license plate key point prediction is carried out on the three third feature maps by using a head network, so that 8-dimensional key point vectors are output, and 4 license plate corner point coordinates are obtained according to the key point vectors (namely each license plate corner point coordinate is represented by a 2-dimensional vector (namely horizontal and vertical coordinates), and the 8-dimensional vector exactly corresponds to 4 corner points); therefore, through the design, the invention can directly obtain 4 angular points of the license plate, realizes end-to-end license plate positioning, and avoids the problems of large error and low precision of target detection and image processing technologies, and the invention adopts the MobileNet V1-0.25 network as the backbone network, thereby realizing license plate positioning under the condition of almost no loss of precision, reducing the parameters and the calculated amount of an integral model, providing conditions for terminal deployment of a license plate recognition system, and being suitable for large-scale popularization and application.
The following describes a license plate location method provided in an embodiment of the present application with reference to the accompanying drawings, and the method is specifically described by applying the method to the license plate location model structure shown in fig. 1.
As shown in fig. 2, the license plate location method for license plate recognition provided in the first aspect of this embodiment is suitable for license plate location in any scene (for example, parking lot entry and exit, highway toll station entry and exit, and traffic violation capture, etc.), and may include, but is not limited to, the following step S101 and step S102.
S101, obtaining an image to be recognized, wherein the image to be recognized at least comprises a license plate to be recognized.
Step S101 is a process of acquiring an image to be recognized so as to be input into a license plate positioning model in the following process, and four license plate corner coordinates of a license plate are extracted, so that the positioning of the license plate in the image is completed; in specific implementation, the method can be, but is not limited to: the license plate pictures shot when the parking lot enters or exits the gate and/or the license plate pictures shot when the parking lot enters or exits the highway toll station are used as images to be recognized, any picture in a section of license plate video can be used as an image to be recognized (namely, the video is processed frame by frame to obtain a plurality of images), and the images containing the license plate to be recognized can be directly uploaded by personnel in work, namely, the images to be recognized can select different acquisition modes according to specific application scenes, and the method is not particularly limited.
After the image to be recognized is obtained, the image may be recognized by using the model shown in fig. 1, so as to locate the license plate in the image, as shown in the following step S102.
S102, inputting the image to be recognized into a license plate positioning model for positioning detection to obtain 4 license plate corner coordinates corresponding to the license plate to be recognized, so that license plate positioning in the image to be recognized is completed according to the 4 license plate corner coordinates.
As described above, the license plate location model provided in this embodiment is improved on the conventional Retinaface detection network, that is, the MobileNetV1-0.25 network is used as the main feature extraction network of the Retinaface detection network, and the FPN network, the SSH network, and the head network in the conventional Retinaface detection network are retained, so as to extract the coordinates of the corner points of the license plate, and complete the location of the license plate in the image to be recognized.
Referring to fig. 1, the following description specifically explains the license plate location model:
firstly, the MobileNet network is a lightweight deep neural network proposed by Google for embedded devices such as mobile phones, and the core concept of the use of the MobileNet network is deep separable convolution (depthwise separable convolution) with three structures of V1, V2 and V3, while the MobileNet V1-0.25 network simplifies the structure on the basis of MobileNet V1, i.e. compresses the number of channels thereof to 1/4 of the MobileNet V1 network, so that under the condition of slight precision loss, the calculated amount and the parameter amount of the model are greatly reduced, and favorable conditions are provided for the model to be deployed on each embedded device.
In specific implementation, the mobilenetV1-0.25 network performs feature extraction on an image to be recognized under the condition of three different receptive fields, so as to obtain a first feature map with three different channel numbers, namely, the first feature extraction is performed on the image to be recognized, the receptive field is the size of a mapping region of pixel points on the feature map on an original image (namely, the image to be recognized), namely, the larger the receptive field is, the larger the range of the extracted original image is, and the extracted image contains more abundant feature information; in the embodiment, three receptive fields are used for feature extraction, which is equivalent to extracting feature information of three dimensions, so that the comprehensiveness of feature information extraction is ensured.
In this embodiment, sampling may be performed under conditions of 8 times (i.e., performing convolution processing on the image to be recognized by using convolution operation, such as performing convolution operation using a convolution kernel with a size of 2 × 2), 16 times, and 32 times (which may be but is not limited to performing feature sampling by using dw (dewapthise) convolution), so as to obtain first feature maps (i.e., C3, C4, and C5 in fig. 1) with 64 channels, 128 channels, and 256 channels, that is, outputting the three first feature maps to the FPN network for feature fusion, so as to further improve the extracted feature information.
In specific implementation, the FPN (Feature Pyramid network) can greatly improve the detection performance of the object without increasing the calculation amount of the model, that is, the channel number of the first Feature map output by the MobileNetV1-0.25 network is changed first, and then the upsampling and the Feature fusion are performed in sequence to obtain the second Feature map with richer Feature information; optionally, the FPN network may, but not limited to, perform convolution operation on the first feature maps with the number of 64 channels, the number of 128 channels, and the number of 256 channels by using a convolution kernel of 1 × 1, so as to adjust the number of channels of the three to the number of 64 channels, where the convolution of 1 × 1 does not need to consider the relationship between pixels and peripheral pixels, and may perform linear combination on pixels on different channels, and then perform nonlinear operation to complete the function of reducing the dimensions (i.e., while reducing the number of channels, the width and height of the image are not changed, so as to achieve the purpose of reducing the amount of computation).
After the three first feature maps are channel-adjusted, upsampling (i.e., image amplification) and feature fusion may be performed, and in the specific implementation, the upsampling may be performed by using a nearest neighbor interpolation method, a bilinear interpolation method, a mean interpolation method, or a median interpolation method, and in the feature fusion, the specific operations are as follows: referring to fig. 1, the first feature map C5 is used as the bottom feature map, and after passing through the FPN network, it is directly used as the second feature map P5; and then, performing up-sampling on the first feature map C5, performing feature fusion with the second feature map C4 to obtain a second feature map P4, and finally, performing up-sampling on the second feature map P4, and performing feature fusion with the first feature map C3 to obtain a second feature map P3, so that three second feature maps can be obtained, feature fusion under three dimensions is realized, loss of feature information is avoided, more abundant feature information is obtained, and the positioning capability of the license plate in the image is improved.
In this embodiment, feature fusion of the three first feature maps is implemented by using channel fusion, which is also called as: concat channel fusion, which is a feature fusion method commonly used in each neural network model, merges the number of channels of convolutional layers.
After the three second feature maps are obtained, the three second feature maps can be input into an SSH Network (improved based on a Visual Geometry Group Network (VGG-16 Network)) for convolution processing, so that more fine information features can be obtained; the SSH network is provided with three parallel convolutional layers, that is, the three parallel convolutional layers can be used to perform convolution processing on the three second feature maps respectively, so as to implement multi-scale feature detection, thereby enhancing feature extraction, and improving context information of the feature maps, so as to improve quality of extracted features and angular point positioning accuracy.
Referring to fig. 3, three parallel convolution layers in the SSH network are, respectively, one convolution with 3 × 3 on the left, two convolutions with 3 × 3 in the middle, and three convolutions with 3 × 3 on the right, and in fig. 3, Conc2d ks is 3filters is 32, which represents that a convolution kernel with 3 × 3 is used, and conv2d convolution is performed to obtain a feature map with 32 channels; in practice, the three second feature maps are input into three parallel convolutional layers, so as to obtain three third feature maps.
After the features of the three networks are extracted, the head network can be used for predicting the key points of the license plate, namely detecting the prior frame (also called an anchor frame or a detection frame), the head network is a network for acquiring the output content of the network, and the features extracted by the three networks can be used for predicting to obtain a prediction result.
In this embodiment, the head network predicting the key points of the license plate includes the following three aspects: (1) the classification prediction is used for judging whether the inside of the prior frame contains a target (namely a license plate); (2) the regression of the prior frame is used for adjusting the prior frame to obtain a prediction frame, four parameters are needed, the first two are used for adjusting the center of the prior frame, the second two are used for adjusting the width and the height of the prior frame, and the adjustment of the prior frame can be realized by using a convolution of 1 x 1; (3) regression of license plate key points, that is, adjustment of a prediction frame in a priori frame is performed to obtain license plate key point coordinates, 8 parameters are required, each license plate key point requires two adjustment parameters (that is, horizontal and vertical coordinates), that is, x and y axes of the center of the prediction frame are adjusted to obtain license plate key point coordinates, that is, 8-dimensional key point vectors are output to obtain 4 license plate key point coordinates (that is, 4 license plate corner point coordinates) according to the 8-dimensional key point vector matching, that is, each license plate corner point coordinate is represented by a two-dimensional vector of (x, y), and of course, the final output result is represented by coordinates.
Meanwhile, in specific implementation, the embodiment further optimizes the model parameters of the license plate location model, and specifically includes: and adding constraints of the model about the corner points, improving the size of the prior frame and weighting each constraint.
Firstly, in this embodiment, the size of the prior anchor of the license plate location model is set as: 16 x8 (length x width), 32 x 16, 64 x 32, 128 x 64, 256 x 128, and 512 x 256, wherein the a priori anchors are used to characterize the anchor boxes that locate the license plate; through the design, the anchor point frame can be closer to the actual size of the license plate, and therefore the angular point positioning precision can be further improved.
Secondly, in this embodiment, on the basis of the original constraint condition of the Retinaface detection network, an angular point distance constraint condition and an angular point distance proportion constraint condition are newly added, and the reason is as follows: the traditional Retinaface detection network is a face recognition network, which only performs key point regression supervision, and has the following disadvantages: the key points are in relative positions, for a human face, the key points corresponding to the left eye and the right eye are on the top, the key points corresponding to the nose are in the middle, and the key points corresponding to the corner points of the left mouth and the right mouth are on the bottom, however, the traditional Retinaface detection network does not supervise and learn the positions of the key points, which can cause the position of the finally identified key points to have a small probability of dislocation error, thereby affecting the positioning accuracy, but the defect has a great influence when the license plate is positioned, and can directly cause the license plate positioning and the identification error; therefore, the embodiment adds the constraint conditions of the two key points so as to carry out the constraint of the key point positions; also, since the appearance of the license plate is geometrically a standard rectangle, even when tilted, it is a standard parallelogram, and therefore, this constraint is applicable when tilted.
The corner distance constraint function is used for constraining the distance between two parallel sides of the license plate to be recognized (namely ensuring the distance between the two parallel sides to be equal), the corner distance proportion constraint function is used for constraining the proportion between the length and the width of the license plate to be recognized, namely the proportion between the length and the width of the recognized license plate is equal to the proportion between the actual length and the actual width of the license plate, and the function specifically setting constraint conditions (namely the loss function of the whole model) is as follows:
Figure BDA0003383987140000091
in the above formula, L represents a loss function of the license plate location model, LclsRepresenting the target classification loss function, LboxRepresents the regression loss function of the detection box, LptsRepresenting the regression loss function of the key points, piRepresenting the probability that an anchor point i (representing the smallest unit point on the feature map) is predicted as a license plate corner,
Figure BDA0003383987140000092
a true tag denoted as a positive anchor at 1,
Figure BDA0003383987140000093
true tag representing a negative anchor at 0, tiRepresenting the predicted coordinates of the prediction box in the anchor box,
Figure BDA0003383987140000094
representing the true coordinates of the predicted frame in the anchor frame, liRepresenting the coordinates of the corner points of the license plate predicted in the anchor point frame,
Figure BDA0003383987140000095
indicating the coordinates of the marked corner points, λ, of the license plate1、λ2、λ3And λ4Represents a weight coefficient and is a constant.
Lpts1And representing a corner distance constraint function for constraining the distance between two parallel edges of the license plate to be recognized.
Lpts2And representing a corner distance proportion constraint function for constraining the proportion of the length and the width of the license plate to be recognized.
And in (1), Lpts1The specific functional expression of (a) is:
the corner distance constraint function is:
Lpts1(li)＝|||l0l1|-|l2l3|||+|||l0l2|-|l1l3||| (2)
in the above formula (2), l0l1Represents the distance between the first license plate corner point and the second license plate corner point, l2l3Represents the distance between the third license plate corner point and the fourth license plate corner point, l0l2Represents the distance between the first license plate corner point and the third license plate corner point, l1l3And (3) representing the distance between the second license plate corner and the fourth license plate corner, wherein, referring to fig. 4, the upper left corner of the license plate to be recognized is the first license plate corner (denoted as p0), the upper right corner is the second license plate corner (denoted as p1), the lower left corner is the third license plate corner (denoted as p2), and the lower right corner is the fourth license plate corner (denoted as p 3).
As can be seen from fig. 4, the distance between the first license plate corner and the second license plate corner, and the distance between the third license plate corner and the fourth license plate corner are two length sides of the license plate to be recognized, and the distance between the first license plate corner and the third license plate corner, and the distance between the second license plate corner and the fourth license plate corner are two width sides of the license plate to be recognized; thus, the constraint represented by the aforementioned formula (2) means: the distance between the two parallel sides should be equal, i.e. Lpts1The final value should be constrained to 0 or approach 0 so that the identified corner points are the most accurate.
Similarly, the corner distance proportional constraint function is:
Figure BDA0003383987140000101
in the above-mentioned formula (3),
Figure BDA0003383987140000102
the distance between the first license plate marking angular point and the second license plate marking angular point is represented,
Figure BDA0003383987140000103
and expressing the distance from the first license plate marking angular point to the third license plate marking angular point, wherein the license plate marking angular point represents the real angular point coordinates of the license plate and is obtained by the pre-measurement of workers.
Thus, the constraint of the aforementioned formula (3) means: the ratio of the length to the width of the license plate obtained according to the coordinates of the 4 license plate corner points predicted by the model and the ratio of the real length to the width of the license plate should be equal, that is, the value of the corner point distance proportional constraint function should be finally constrained to be 0 or approach to 0.
In this embodiment, the distance is obtained according to the coordinates of the corner points of the license plate, but may be obtained by using a coordinate distance formula, which is not described herein.
Therefore, through the improvement of the model constraint conditions, the regression precision of 4 corner points of the license plate can be improved, and the corner point error risk is reduced.
Finally, in the conventional Retinaface detection network, λ1＝0.25，λ2When the weight of the regression loss function of the key point is 0.1, the weight of the regression loss function of the key point is lower, and the corner point of the license plate is more important relative to the position frame of the license plate in the license plate positioning detection, so the lambda is set in the embodiment20.5, and setting the weight coefficient lambda of the newly added constraint term3＝0.01，λ40.01; therefore, corner regression can be restrained together, and the positioning accuracy of the license plate corners is further improved.
Therefore, through the detailed explanation of the license plate positioning model, the license plate positioning model has the following advantages:
(1) the invention adopts the lightweight MobileNet V1-0.25 network as the backbone network, not only can realize the positioning of the corner points of the license plate under the condition of almost no loss of precision, but also reduces the calculated amount and the parameter quantity of the model, and provides favorable conditions for the terminal to deploy the license plate recognition system.
(2) The branch output of the key points is improved from the traditional 10-dimensional vector to an 8-dimensional vector, 4 license plate corner points can be directly obtained, end-to-end license plate positioning is realized, and the problem of amplifying step positioning errors of target detection is avoided.
(3) The positioning precision of the license plate corner points can be remarkably improved and the license plate identification precision can be further improved by improving the prior frame size, increasing the corner point distance constraint condition, increasing the corner point distance proportion constraint condition and increasing the weight of the regression loss function of the key points.
Referring to fig. 6, in a second aspect of the present embodiment, on the basis of the first aspect of the present embodiment, a license plate recognition method based on license plate corner coordinates is provided, as shown in the following steps S201 to S205.
S201, obtaining an image to be recognized, wherein the image to be recognized at least comprises a license plate to be recognized.
S202, inputting the image to be recognized into a license plate positioning model for positioning detection to obtain 4 license plate corner coordinates corresponding to the license plate to be recognized.
And S203, acquiring the standard corner coordinates of the license plate in the image to be recognized.
And S204, performing perspective transformation on the image to be recognized by using the license plate standard corner coordinates and the 4 license plate corner coordinates to obtain a corrected image to be recognized, wherein the corrected image to be recognized only contains the license plate to be recognized.
S205, inputting the corrected image to be recognized into a license plate recognition model to obtain a license plate recognition result.
The license plate recognition principle of the steps is as follows:
the method comprises the steps of constructing a perspective transformation matrix M according to 4 license plate corner coordinates and 4 license plate standard corner coordinates by using a getPerspec transform function in opencv (which is a cross-platform computer vision and machine learning software library issued based on Apache2.0 license), and then transmitting an image to be recognized and the matrix M by using a warp Perspec function in opencv, so as to obtain a corrected image to be recognized.
In specific implementation, the coordinates of the standard corner points of the 4 license plates may be preset for a user according to the size of the license plate to be transformed, for example, if the license plate needs to be transformed into a license plate with a size of 94 × 24, the coordinates of the standard corner points of the 4 license plates are as follows: p0(0,0), p1(94,0), p2(24,0), p3(94,24), and at the same time, LPRnet (license Plate registration via Deep Neural networks) license Plate Recognition network can be used for license Plate Recognition.
Referring to fig. 5, fig. 5 is a schematic diagram of license plate positioning and recognition effects performed by using the positioning model and the recognition method provided in this embodiment, and it can be obviously seen from fig. 5 that end-to-end license plate positioning can be achieved, and tilt correction is performed, so that an image only including a license plate to be recognized is obtained, and license plate positioning and recognition accuracy is obviously improved.
As shown in fig. 7, a third aspect of the present embodiment provides a hardware device for implementing the license plate location method for license plate recognition described in the first aspect of the embodiment, including:
the device comprises an acquisition unit, a recognition unit and a recognition unit, wherein the acquisition unit is used for acquiring an image to be recognized, and the image to be recognized at least comprises a license plate to be recognized.
And the license plate positioning unit is used for inputting the image to be recognized into a license plate positioning model for positioning detection to obtain 4 license plate corner coordinates corresponding to the license plate to be recognized so as to complete license plate positioning in the image to be recognized according to the 4 license plate corner coordinates, wherein the license plate positioning model comprises a MobileNet V1-0.25 network, a FPN network, an SSH network and a head network.
For the working process, the working details, and the technical effects of the hardware apparatus provided in this embodiment, reference may be made to the first aspect of the embodiment, which is not described herein again.
As shown in fig. 8, a fourth aspect of the present embodiment provides another license plate location device for license plate recognition, taking the device as a computer host device as an example, including: the license plate locating method comprises a memory, a processor and a transceiver which are sequentially connected in a communication mode, wherein the memory is used for storing computer programs, the transceiver is used for receiving and sending messages, and the processor is used for reading the computer programs and executing the license plate locating method for license plate recognition in the first aspect of the embodiment.
For example, the Memory may include, but is not limited to, a Random Access Memory (RAM), a Read Only Memory (ROM), a Flash Memory (Flash Memory), a First In First Out (FIFO), and/or a First In Last Out (FILO), and the like; in particular, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, and so on. The processor may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field Programmable Gate Array), and a PLA (Programmable Logic Array), and may also include a main processor and a coprocessor, where the main processor is a processor for Processing data in an awake state and is also referred to as a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state.
In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit) which is responsible for rendering and drawing contents required to be displayed on the display screen, for example, the processor may not be limited to a processor adopting a model STM32F105 series microprocessor, a reduced instruction set computer (RSIC) microprocessor, an architecture processor such as X86, or a processor integrated with an embedded neural Network Processor (NPU); the transceiver may be, but is not limited to, a wireless fidelity (WIFI) wireless transceiver, a bluetooth wireless transceiver, a General Packet Radio Service (GPRS) wireless transceiver, a ZigBee wireless transceiver (ieee802.15.4 standard-based low power local area network protocol), a 3G transceiver, a 4G transceiver, and/or a 5G transceiver, etc. In addition, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.
For the working process, the working details, and the technical effects of the computer main device provided in this embodiment, reference may be made to the first aspect of the embodiment, which is not described herein again.
A fifth aspect of the present embodiment provides a storage medium storing instructions including the license plate location method for license plate recognition according to the first aspect of the present invention, that is, the storage medium stores instructions thereon, and when the instructions are executed on a computer, the license plate location method for license plate recognition according to the first aspect of the present invention is executed.
The storage medium refers to a carrier for storing data, and may include, but is not limited to, a floppy disk, an optical disk, a hard disk, a flash Memory, a flash disk and/or a Memory Stick (Memory Stick), etc., and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
For the working process, the working details, and the technical effects of the storage medium provided in this embodiment, reference may be made to the first aspect of the embodiment, which is not described herein again.
A sixth aspect of the present embodiment provides a computer program product comprising instructions for causing a computer to perform the license plate location method for license plate recognition according to the first aspect of the embodiment when the instructions are run on the computer, wherein the computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable device.
Finally, it should be noted that: the above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. A license plate positioning method for license plate recognition is characterized by comprising the following steps:
acquiring an image to be recognized, wherein the image to be recognized at least comprises a license plate to be recognized;
inputting the image to be recognized into a license plate positioning model for positioning detection to obtain 4 license plate corner coordinates corresponding to the license plate to be recognized so as to complete license plate positioning in the image to be recognized according to the 4 license plate corner coordinates, wherein the license plate positioning model comprises a MobileNet V1-0.25 network, an FPN network, an SSH network and a head network;
the MobileNet V1-0.25 network is used as a main feature extraction network of the license plate positioning model and is used for carrying out feature extraction on the image to be recognized under three receptive field conditions to obtain three first feature maps;
the FPN network is used for carrying out up-sampling and feature fusion on the three first feature maps to obtain three second feature maps;
the SSH network comprises three parallel convolutional layers and is used for carrying out convolution processing on the three second feature maps by utilizing the three parallel convolutional layers to obtain three third feature maps;
the head network is used for predicting license plate key points of the three third feature maps so as to output 8-dimensional key point vectors, and 4 license plate corner point coordinates are obtained according to the key point vectors.
2. The method of claim 1, wherein the loss function of the license plate location model is:
Figure FDA0003383987130000011
in the above formula, L represents a loss function of the license plate location model, LclsRepresenting the target classification loss function, LboxRepresents the regression loss function of the detection box, LptsRepresenting the regression loss function of the key points, piRepresenting the probability that anchor point i is predicted as a license plate corner,
Figure FDA0003383987130000012
a true tag denoted as a positive anchor at 1,
Figure FDA0003383987130000013
true tag representing a negative anchor at 0, tiRepresenting the predicted coordinates of the prediction box in the anchor box,
Figure FDA0003383987130000014
representing the true coordinates of the predicted frame in the anchor frame, liRepresenting the coordinates of the corner points of the license plate predicted in the anchor point frame,
Figure FDA0003383987130000015
indicating the coordinates of the marked corner points, λ, of the license plate1、λ2、λ3And λ4Represents a weight coefficient, and is a constant;
Lpts1representing a corner point distance constraint function which is used for constraining the distance between two parallel edges of the license plate to be recognized;
Lpts2representing a corner distance proportional constraint function for approximatingThe length and width ratio of the license plate to be identified.
3. The method of claim 2, wherein the corner distance constraint function is:
Lpts1(li)＝|||l0l1|-|l2l3|||+|||l0l2|-|l1l3|||
in the above formula, /)0l1Represents the distance between the first license plate corner point and the second license plate corner point, l2l3Represents the distance between the third license plate corner point and the fourth license plate corner point, l0l2Represents the distance between the first license plate corner point and the third license plate corner point, l1l3And representing the distance between the second license plate corner point and the fourth license plate corner point, wherein the upper left corner of the license plate to be identified is the first license plate corner point, the upper right corner is the second license plate corner point, the lower left corner is the third license plate corner point, and the lower right corner is the fourth license plate corner point.
4. The method of claim 3, wherein the corner-distance proportional constraint function is:
Figure FDA0003383987130000021
in the above-mentioned formula, the first and second groups,
Figure FDA0003383987130000022
the distance between the first license plate marking angular point and the second license plate marking angular point is represented,
Figure FDA0003383987130000023
and representing the distance between the first license plate marking angular point and the third license plate marking angular point.
5. The method of claim 2,weight coefficient lambda1、λ2、λ3And λ4The values of (a) are 0.25, 0.5, 0.01 and 0.01, respectively.
6. The method of claim 1, wherein the prior anchor of the license plate localization model has a size of: 16 x8, 32 x 16, 64 x 32, 128 x 64, 256 x 128 and 512 x 256, wherein the a priori anchors are used to characterize the anchor boxes where the license plate is located.
7. The method of claim 1, wherein after the image to be recognized is input into a license plate positioning model for positioning detection, and 4 license plate corner coordinates corresponding to the license plate to be recognized are obtained, the method further comprises:
acquiring license plate standard corner coordinates in the image to be recognized;
performing perspective transformation processing on the image to be recognized by using the license plate standard corner coordinates and the 4 license plate corner coordinates to obtain a corrected image to be recognized, wherein the corrected image to be recognized only contains the license plate to be recognized;
and inputting the corrected image to be recognized into a license plate recognition model to obtain a license plate recognition result.
8. A license plate positioning method for license plate recognition is characterized by comprising the following steps:
the device comprises an acquisition unit, a recognition unit and a recognition unit, wherein the acquisition unit is used for acquiring an image to be recognized, and the image to be recognized at least comprises a license plate to be recognized;
and the license plate positioning unit is used for inputting the image to be recognized into a license plate positioning model for positioning detection to obtain 4 license plate corner coordinates corresponding to the license plate to be recognized so as to complete license plate positioning in the image to be recognized according to the 4 license plate corner coordinates, wherein the license plate positioning model comprises a MobileNet V1-0.25 network, a FPN network, an SSH network and a head network.
9. A license plate positioning method for license plate recognition is characterized by comprising the following steps: the license plate positioning method comprises a memory, a processor and a transceiver which are sequentially connected in a communication mode, wherein the memory is used for storing computer programs, the transceiver is used for receiving and sending messages, and the processor is used for reading the computer programs and executing the license plate positioning method for license plate identification according to any one of claims 1-7.
10. A storage medium having stored thereon instructions for performing the license plate location method for license plate recognition according to any one of claims 1 to 7 when the instructions are run on a computer.
CN202111443006.2A 2021-11-30 2021-11-30 License plate positioning method and device for license plate recognition and storage medium Pending CN114155362A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111443006.2A CN114155362A (en) 2021-11-30 2021-11-30 License plate positioning method and device for license plate recognition and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111443006.2A CN114155362A (en) 2021-11-30 2021-11-30 License plate positioning method and device for license plate recognition and storage medium

Publications (1)

Publication Number Publication Date
CN114155362A true CN114155362A (en) 2022-03-08

Family

ID=80454989

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111443006.2A Pending CN114155362A (en) 2021-11-30 2021-11-30 License plate positioning method and device for license plate recognition and storage medium

Country Status (1)

Country Link
CN (1) CN114155362A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114332447A (en) * 2022-03-14 2022-04-12 浙江大华技术股份有限公司 License plate correction method, license plate correction device and computer readable storage medium
CN115577728A (en) * 2022-12-07 2023-01-06 深圳思谋信息科技有限公司 One-dimensional code positioning method, device, computer equipment and storage medium

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114332447A (en) * 2022-03-14 2022-04-12 浙江大华技术股份有限公司 License plate correction method, license plate correction device and computer readable storage medium
CN114332447B (en) * 2022-03-14 2022-08-09 浙江大华技术股份有限公司 License plate correction method, license plate correction device and computer readable storage medium
CN115577728A (en) * 2022-12-07 2023-01-06 深圳思谋信息科技有限公司 One-dimensional code positioning method, device, computer equipment and storage medium
CN115577728B (en) * 2022-12-07 2023-03-14 深圳思谋信息科技有限公司 One-dimensional code positioning method, device, computer equipment and storage medium

Similar Documents

Publication Publication Date Title
US11823363B2 (en) Infrared and visible light fusion method
CN111914795B (en) Method for detecting rotating target in aerial image
CN109978755B (en) Panoramic image synthesis method, device, equipment and storage medium
CN110020985B (en) Video stitching system and method of binocular robot
CN114155362A (en) License plate positioning method and device for license plate recognition and storage medium
CN109559349A (en) A kind of method and apparatus for calibration
US20210295529A1 (en) Method and system for image processing
CN112233062A (en) Surface feature change detection method, electronic device, and storage medium
CN111046746A (en) License plate detection method and device
CN114782548B (en) Global image-based radar data calibration method, device, equipment and medium
CN112560701A (en) Face image extraction method and device and computer storage medium
CN110189395B (en) Method for realizing dynamic analysis and quantitative design of landscape elevation based on human visual angle oblique photography
CN111626241A (en) Face detection method and device
Liu et al. A lightweight building instance extraction method based on adaptive optimization of mask contour
CN114897684A (en) Vehicle image splicing method and device, computer equipment and storage medium
CN117422858A (en) Dual-light image target detection method, system, equipment and medium
CN114690144A (en) Point cloud data labeling method, device and system
CN104021535B (en) The method of stepping framing ccd image splicing
CN116343095A (en) Vehicle track extraction method based on video stitching and related equipment
CN111062384A (en) Vehicle window accurate positioning method based on deep learning
CN115965831A (en) Vehicle detection model training method and vehicle detection method
CN116342632A (en) Depth information-based matting method and matting network training method
CN112699861B (en) Natural scene bill correction method based on neural network hotspot graph
CN113112531B (en) Image matching method and device
CN114332814A (en) Parking frame identification method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination