WO2021138893A1

WO2021138893A1 - Vehicle license plate recognition method and apparatus, electronic device, and storage medium

Info

Publication number: WO2021138893A1
Application number: PCT/CN2020/071330
Authority: WO
Inventors: 张恒瑞; 宋翔; 郭明坚; 林雨辉; 张劲松
Original assignee: 顺丰科技有限公司
Priority date: 2020-01-10
Filing date: 2020-01-10
Publication date: 2021-07-15
Also published as: CN115298705A

Abstract

A vehicle license plate recognition method and apparatus, an electronic device and a storage medium. The vehicle license plate recognition method comprises: acquiring a plurality of vehicle license plate images of a target vehicle (201); performing vehicle license plate recognition on the plurality of vehicle license plate images respectively, so as to obtain a plurality of vehicle license plate recognition results (202); and fusing the plurality of vehicle license plate recognition results, and determining a vehicle license plate recognition result of the target vehicle (203). As the finally determined recognition result of the vehicle license plate of the target vehicle is determined by fusing a plurality of recognition results of the vehicle license plate, the influence on the recognition result of the vehicle license plate caused by false detection of the vehicle license plate and motion blur of the vehicle license plate can be effectively eliminated, greatly improving the accuracy of recognition of vehicle license plates.

Description

License plate recognition method, device, electronic equipment and storage medium

Technical field

The invention relates to the technical field of license plate recognition, in particular to a license plate recognition method, device, electronic equipment and storage medium.

Background technique

License plate recognition is a popular application field in the field of Optical Character Recognition (OCR). There are a large number of license plate recognition solutions for parking lots and toll stations on the market. However, in these application scenarios, the license plate is about 1 meter away from the camera, coupled with the supporting lighting system and a proprietary camera, the image quality of the license plate image captured is very high, and naturally a high license plate recognition accuracy rate can be achieved.

In some actual application scenarios, the camera is sometimes far away from the vehicle or the license plate (for example, more than 3m), and the camera is a non-proprietary camera, such as the license plate recognition system applied to the loading and unloading port of the logistics transfer yard, used to identify loading and unloading vehicles License plate number. In order to save costs, the license plate recognition system uses a monitoring camera at the loading and unloading port instead of the proprietary camera in the aforementioned license plate recognition solution. The monitoring camera of the loading and unloading port is generally installed close to the ceiling. When the vehicle is parked in place, the distance between the camera and the license plate is greater than 3 meters, and the distance can be up to 10 meters or more during the parking and leaving of the vehicle, resulting in the license plate in the picture The pixel area is too small, which brings challenges to recognition. Secondly, the complex lighting conditions in the transit field, such as insufficient lighting, backlighting, and license plate reflections, also pose challenges for recognition. In addition, the vehicles in the transit yard are mainly large trucks, and only the rear license plate can be seen when the truck is loading and unloading. In the picture, the first line of Chinese characters and letters of the double-layer yellow license plate at the rear of the vehicle are too small, which makes it difficult to recognize, especially Chinese characters with many strokes, such as "Gan", "Zang", and "E" are stuck together, making it more difficult to recognize. At the same time, due to the high installation position of the camera, the upper part of the license plate of some models will be blocked by the rear bumper, and the characters in the first line cannot be recognized.

In addition, traditional license plate recognition methods include license plate detection, image binarization, character segmentation, character recognition and other steps. Because license plate detection will have misdetection, it is generally necessary to add an image quality judgment link before image binarization. The link uses a small classification network to eliminate non-license plate and fuzzy license plate images. The traditional method contains so many links, which leads to a lot of manpower for data labeling and model training in each link. In addition, the final recognition accuracy is obtained by multiplying the accuracy of each link. Because there are too many links, and each link cannot reach 100%, there is a bottleneck that is difficult to break through the accuracy of license plate recognition.

technical problem

It can be seen from the above that traditional license plate recognition methods have too many recognition links and are susceptible to environmental conditions, resulting in low accuracy of license plate recognition.

Technical solutions

The embodiments of the present application provide a license plate recognition method, device, electronic equipment, and storage medium, which can effectively eliminate the influence of misdetection of license plates and blurring of license plate motion on the result of license plate recognition, and greatly improve the accuracy of license plate recognition.

In one aspect, the present application provides a method for recognizing a license plate, and the method for recognizing a license plate includes:

Acquire multiple license plate images of the target vehicle;

Performing license plate recognition on the multiple license plate images to obtain multiple license plate recognition results;

Fusion of the multiple license plate recognition results to determine the license plate recognition result of the target vehicle.

In some embodiments of the present application, performing license plate recognition on the multiple license plate images to obtain multiple license plate recognition results includes:

Respectively taking the license plate images in the plurality of license plate images as target license plate images, extracting features of the target license plate images, and obtaining feature maps corresponding to the license plate images;

Extracting the attention map of the characters in the feature map;

Calculating, according to the attention map, a license plate character recognition result tensor corresponding to the target license plate image;

According to the license plate character recognition tensor, the license plate recognition result of the target license plate image is determined.

In some embodiments of the present application, the extracting features of the target license plate image to obtain a feature map corresponding to the license plate image includes:

The target license plate image is input into a preset convolutional neural network model to extract features corresponding to the license plate image using the convolutional neural network model, and output a feature map corresponding to the target license plate image.

In some embodiments of the present application, the calculating the tensor of the license plate character recognition result according to the attention map includes:

Weighting the attention map and the feature map to obtain a weighted map;

The weighted graph is input into a preset long-term short-term memory network model, and the license plate character recognition result tensor corresponding to the target license plate image is output.

In some embodiments of the present application, the license plate character recognition tensor includes a character recognition tensor at multiple positions, and the character recognition tensor at each position includes a plurality of candidate characters. According to the license plate character recognition tensor, it is determined The license plate recognition result of the target license plate image includes:

Parse the license plate character recognition tensor to obtain the confidence level of each candidate character in the character recognition tensor at each position;

Determine the candidate character with the highest confidence in each position according to the confidence of each candidate character in the character recognition tensor of each position;

The candidate character with the highest confidence in each position is taken as the character recognition result of the position, and the license plate recognition result of the target license plate image is obtained.

In some embodiments of the present application, the fusing the multiple license plate recognition results to determine the license plate recognition result of the target vehicle includes:

Fusing the multiple license plate recognition results to obtain a fused license plate recognition result;

According to the result of the license plate recognition after fusion, search in the preset license plate database to determine the license plate recognition result of the target vehicle.

In some embodiments of the present application, the fusion of the multiple license plate recognition results to obtain the fused license plate recognition result includes:

Calculate the number of qualified characters whose character recognition results meet the preset reliability requirements in each license plate recognition result;

Take the N license plate recognition results with the largest number of qualified characters, perform the fusion of the license plate recognition results, and obtain the fused license plate recognition results.

In some embodiments of the present application, the acquiring multiple license plate images of the target vehicle includes:

Acquiring the collected surveillance video of the target vehicle;

Detecting the license plate in the surveillance video to obtain multiple images including the license plate;

The license plate area in the multiple images is cropped and normalized to a preset size to obtain multiple license plate images of the target vehicle.

In some embodiments of the present application, the detecting the license plate in the surveillance video to obtain multiple images including the license plate includes:

A multi-target detection method is used to detect the license plate in the surveillance video to obtain multiple images including the license plate.

In another aspect, the present application provides a license plate recognition device, the license plate recognition device includes:

The acquiring unit is used to acquire multiple license plate images of the target vehicle;

A recognition unit, configured to perform license plate recognition on the multiple license plate images to obtain multiple license plate recognition results;

The determining unit is used for fusing the multiple license plate recognition results to determine the license plate recognition result of the target vehicle.

In some embodiments of the present application, the identification unit is specifically configured to:

Extracting the attention map of the characters in the feature map;

Weighting the attention map and the feature map to obtain a weighted map;

In some embodiments of the present application, the license plate character recognition tensor includes character recognition tensors at multiple positions, and the character recognition tensor at each position includes multiple candidate characters, and the recognition unit is specifically configured to:

In some embodiments of the present application, the determining unit is specifically configured to:

In some embodiments of the present application, the acquiring unit is specifically configured to:

Acquiring the collected surveillance video of the target vehicle;

On the other hand, this application also provides an electronic device, which includes:

One or more processors;

Memory; and

One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the steps in the license plate recognition method described above.

On the other hand, the present application also provides a computer-readable storage medium on which a computer program is stored, and the computer program is loaded by a processor to execute the steps in the license plate recognition method.

Beneficial effect

In this application, multiple license plate images of the target vehicle are acquired; license plate recognition is performed on the multiple license plate images to obtain multiple license plate recognition results; multiple license plate recognition results are merged to determine the license plate recognition result of the target vehicle. In this application, by recognizing multiple license plate images of the target vehicle, multiple license plate recognition results are obtained, and the multiple license plate recognition results are merged to determine the license plate recognition result of the target vehicle, because the final license plate recognition result of the target vehicle is It is determined by the fusion of multiple license plate recognition results, so it can effectively eliminate the influence of license plate misdetection and license plate motion blur on the license plate recognition result, and greatly improve the accuracy of license plate recognition.

Description of the drawings

In order to explain the technical solutions in the embodiments of the present invention more clearly, the following will briefly introduce the drawings needed in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

FIG. 1 is a schematic diagram of a scene of a license plate recognition system provided by an embodiment of the present invention;

2 is a schematic flowchart of an embodiment of a method for recognizing a license plate provided by an embodiment of the present invention;

FIG. 3 is a schematic flowchart of an embodiment in step 202 in an embodiment of the present invention;

4 is a schematic diagram of a specific scene of a license plate recognition scene in an embodiment of the present invention;

FIG. 5 is a schematic diagram of the attention map and the original image of the license plate in the embodiment of the present invention after being superimposed;

Fig. 6 is a schematic structural diagram of an embodiment of a license plate recognition device in an embodiment of the present invention;

Fig. 7 is a schematic structural diagram of an embodiment of an electronic device in an embodiment of the present invention.

Embodiments of the present invention

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the protection scope of the present invention.

In the description of the present invention, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", " The orientation or positional relationship indicated by “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inner”, and “outer” are based on the orientation shown in the drawings The or positional relationship is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the pointed device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with “first” and “second” may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more than two, unless otherwise specifically defined.

In this application, the word "exemplary" is used to mean "serving as an example, illustration, or illustration." Any embodiment described as "exemplary" in this application is not necessarily construed as being more preferred or advantageous over other embodiments. In order to enable any person skilled in the art to implement and use the present invention, the following description is given. In the following description, the details are listed for the purpose of explanation. It should be understood that those of ordinary skill in the art can realize that the present invention can also be implemented without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid unnecessary details to obscure the description of the present invention. Therefore, the present invention is not intended to be limited to the illustrated embodiments, but is consistent with the widest scope that conforms to the principles and features disclosed in the present application.

The embodiments of the present invention provide a license plate recognition method, device, electronic equipment, and storage medium, which will be described in detail below.

Please refer to FIG. 1, which is a schematic diagram of a scene of a license plate recognition system provided by an embodiment of the present invention. The license plate recognition system may include a monitoring device 100 and an electronic device 200. The monitoring device 100 and the electronic device 20 are connected to each other through a network. Integrated with a license plate recognition device, such as the electronic device in FIG. 1, the monitoring device 100 can access the electronic device 200.

In the embodiment of the present invention, the electronic device 200 is mainly used to obtain multiple license plate images of a target vehicle; perform license plate recognition on the multiple license plate images respectively to obtain multiple license plate recognition results; merge the multiple license plate recognition results, Determine the license plate recognition result of the target vehicle.

In the embodiment of the present invention, the monitoring device 100 is mainly used to shoot monitoring video images and transmit them to the electronic device 200.

In the embodiment of the present invention, the electronic device 200 may be an independent server, or a server network or server cluster composed of servers. For example, the electronic device 200 described in the embodiment of the present invention includes, but is not limited to, a computer and a network. A host, a single web server, a set of multiple web servers, or a cloud server composed of multiple servers. Among them, the cloud server is composed of a large number of computers or network servers based on Cloud Computing. In the embodiment of the present invention, the electronic device 200 and the monitoring device 100 can communicate through any communication method, including but not limited to, based on the 3rd Generation Partnership Project (3rd Generation Partnership Project, 3GPP), Long Term Evolution (Long Term) Evolution, LTE), Worldwide Interoperability for Microwave Access (WiMAX) mobile communications, or based on the TCP/IP protocol suite (TCP/IP Protocol Suite, TCP/IP), User Datagram Protocol (User Datagram Protocol, UDP) protocol computer network communication, etc.

It is understandable that the monitoring device 100 used in the embodiment of the present invention may be a device including both receiving and transmitting hardware, that is, a device having receiving and transmitting hardware capable of performing two-way communication on a two-way communication link. Such a monitoring device 100 may include a cellular or other communication device with a single-line display or a multi-line display or a cellular or other communication device without a multi-line display. The specific monitoring device 100 may specifically be a monitoring camera.

Those skilled in the art can understand that the application environment shown in Fig. 1 is only an application scenario with the solution of this application, and does not constitute a limitation on the application scenario of the solution of this application. Other application environments may also include those shown in Fig. 1 Show more or fewer electronic devices, or the network connection relationship of electronic devices. For example, only one electronic device and two monitoring devices are shown in Figure 1. It is understandable that the license plate recognition system may also include one or more other The electronic device, or/and one or more other monitoring devices connected to the electronic device network, are not specifically limited here.

In addition, as shown in FIG. 1, the license plate recognition system may also include a memory 300 for storing data, such as storing video data, for example, video files collected by a monitoring device.

It should be noted that the scene schematic diagram of the license plate recognition system shown in FIG. 1 is only an example. The license plate recognition system and the scene described in the embodiment of the present invention are intended to explain the technical solutions of the embodiments of the present invention more clearly, and do not constitute As for the limitation of the technical solutions provided by the embodiments of the present invention, those of ordinary skill in the art know that with the evolution of the license plate recognition system and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present invention are equally applicable to similar technical problems.

First, an embodiment of the present invention provides a license plate recognition method. The execution body of the license plate recognition method is a license plate recognition device, and the license plate recognition device is applied to electronic equipment. The license plate recognition method includes: acquiring multiple license plate images of a target vehicle; Performing license plate recognition on the multiple license plate images respectively to obtain multiple license plate recognition results; fusing the multiple license plate recognition results to determine the license plate recognition result of the target vehicle.

As shown in FIG. 2, it is a schematic flowchart of an embodiment of a license plate recognition method in an embodiment of the present invention. The license plate recognition method includes:

201. Acquire multiple license plate images of the target vehicle.

Specifically, in some embodiments of the present invention, the acquiring multiple license plate images of the target vehicle may include: acquiring the collected surveillance video of the target vehicle; detecting the license plate in the surveillance video to obtain the license plate The multiple images of the multiple images; the license plate area in the multiple images is cropped and normalized to a preset size to obtain multiple license plate images of the target vehicle. Wherein, the acquired monitoring video of the target vehicle may be a monitoring video shot by one or more monitoring devices.

Further, the detecting the license plate in the surveillance video to obtain multiple images including the license plate includes: detecting the license plate in the surveillance video by using a multi-target detection method to obtain multiple images including the license plate. Specifically, for the surveillance video collected by the camera, the SSD (Single Shot MultiBox Detector) multi-target detection network model can be used to continuously detect the license plate, and the detected license plate image is cropped from the original image and normalized to a uniform size. As input to the image sequence of the subsequent steps.

Among them, the SSD algorithm is a multi-target detection algorithm that directly predicts the target category and bounding box. The SSD algorithm uses feature maps of different convolutional layers for synthesis to achieve the same effect. The main network structure of the SSD algorithm is VGG16.

As the loading and unloading port of the logistics transfer yard, it will also be used to identify the license plate number of the loading and unloading vehicle. In order to save costs, the transit site generally uses surveillance cameras at the loading and unloading ports instead of proprietary cameras. The monitoring camera of the loading and unloading port is generally installed close to the ceiling. When the vehicle is parked in place, the distance between the camera and the license plate is greater than 3 meters, and the distance can be up to 10 meters or more during the parking and leaving of the vehicle, resulting in the license plate in the picture The pixel area is too small, which brings challenges to recognition. The complex lighting conditions in the transit field, such as insufficient lighting, backlighting, and license plate reflections, also pose challenges for recognition. In addition, the vehicles in the transit yard are mainly large trucks, and only the rear license plate can be seen when the truck is loading and unloading. In the picture, the first line of Chinese characters and letters of the double-layer yellow license plate at the rear of the vehicle are too small, which makes it difficult to recognize, especially Chinese characters with many strokes, such as "Gan", "Zang", and "E" are stuck together, making it more difficult to recognize. At the same time, due to the high installation position of the camera, the upper part of the license plate of some models will be blocked by the rear bumper, and the characters in the first line cannot be recognized.

Therefore, the main application scenario in the embodiment of the present invention can be the license plate number recognition scenario of the loading and unloading vehicle at the loading and unloading port. The target vehicle is the loading and unloading vehicle at the loading and unloading port, and the monitoring device is the monitoring device set at the loading and unloading port of the transfer yard. Application scenarios, such as scenarios with higher requirements for other license plate recognition.

202. Perform license plate recognition on the multiple license plate images to obtain multiple license plate recognition results.

In the embodiment of the present invention, after obtaining multiple license plate images of the target vehicle in step 201, license plate recognition can be performed on the multiple license plate images to obtain multiple license plate recognition results. Specifically, the license plate recognition is performed on each license plate image to obtain a license plate recognition result.

Wherein, as shown in FIG. 3, performing license plate recognition on the multiple license plate images to obtain multiple license plate recognition results may further include:

301. Using license plate images in the plurality of license plate images as target license plate images, extract features of the target license plate images to obtain feature maps corresponding to the license plate images.

Specifically, extracting features of the target license plate image to obtain a feature map corresponding to the license plate image includes: inputting the target license plate image into a preset convolutional neural network (Convolutional Neural Networks, CNN) model to use the The convolutional neural network model extracts features corresponding to the license plate image, and outputs a feature map corresponding to the target license plate image. Among them, the preset CNN model may be obtained after training using a large amount of collected license plate image data.

For example, the preset CNN model is used to extract image features, and the target license plate image is input into the preset CNN model to obtain the feature map corresponding to the target license plate image f={f _i,j,c }, where f={ f _i,j,c } is the feature map obtained through the CNN model, which is a 3-dimensional tensor, i, j represent the position coordinates of the feature map, and c represents the channel of the feature map.

302. Extract an attention map of characters in the feature map.

Wherein, the attention map is the visual attention mask of the characters in the feature map. Specifically, the following formula (1), formula (2), and formula (3) can be used to extract the attention map of the characters in the feature map. :

α _t ＝{α _t,i,j } (1)

α _t =soft max _i,j (α _t ) (3)

Among them, V _α is a preset vector, w _s , w _f1 , w _f2 , and w _f3 are the parameters of the weight matrix of the CNN model obtained in advance before the calculation, which will be updated during the CNN model training process, and _st is the preset Suppose the hidden state of the Long Short-Term Memory (LSTM) model at time t, e _i , e _j are the space coordinates i of the feature, and the column vector obtained by one-hot encoding of j, For example: i = 1, 2, 3, 4; j = 1, 2, 3, 4, then when i = 3, j = 4,

The attention map (also called attention mask) α _{t is} obtained after normalization by the softmax function, α _t is the attention map calculated at time t, which is a matrix, α _t,i,j are the elements of the matrix α _{t, i} , J is the row number and column number of the matrix respectively, which is a scalar.

Long Short-Term Memory (LSTM, Long Short-Term Memory) is a time cyclic neural network, which is specially designed to solve the long-term dependence problem of general RNN (recurrent neural network). All RNNs have a Repeat the chain form of neural network modules. In a standard RNN, this repeated structural module has only a very simple structure, such as a tanh layer.

303. Calculate the license plate character recognition result tensor corresponding to the target license plate image according to the attention map.

In the embodiment of the present invention, the calculation of the license plate character recognition result tensor according to the attention map includes: weighting the attention map and the feature map to obtain a weighted map; and inputting the weighted map to a preset length The short-term memory network model outputs the license plate character recognition result tensor corresponding to the target license plate image.

Steps

302 and 303 are the same LSTM. At time t, there are three LSTM inputs: the input value of the network at the current time t, the output value of the LSTM at the previous time, and the unit state at the previous time; the output of the LSTM has two A: The output value of LSTM at the current time t, and the unit state at the current time t. Of course, these input values and output values are all vectors.

LSTM input at time t

It is obtained by the following formula (4) and formula (5);

u _t,c =∑ _i,j α _t,i,j f _i,j,c (4)

Among them, c _t-1 is the one-hot encoding vector of the previous character number in the target license plate image, u _t,c is the feature vector of the weighted image obtained after the weighted summation of the attention map and the feature map,

Is the input vector of the LSTM at time t, w _c and w _u1 are the parameters of the weight matrix of the LSTM model obtained in advance, and are updated during the training of the LSTM model.

The status update and output of LSTM are obtained by the following formula (6):

Among them, _st is the hidden state of the LSTM at time t, which is a column vector, and o _t is the output vector of the LSTM at time t. The character recognition result tensor at time t is obtained by the following formula (7):

among them,

Is the character recognition result vector at time t, u _t is calculated by formula (4), o _t is obtained by formula (6), w _o and w _u2 are the parameters of the weight matrix of the LSTM model obtained in advance.

304. Determine the license plate recognition result of the target license plate image according to the license plate character recognition tensor.

Wherein, the license plate character recognition tensor includes multiple positions of the character recognition tensor, and the character recognition tensor at each position includes multiple candidate characters. In this case, the target license plate is determined according to the license plate character recognition tensor. The image's license plate recognition result can include:

(1) Analyze the license plate character recognition tensor to obtain the confidence of each candidate character in the character recognition tensor at each position.

(2) According to the confidence of each candidate character in the character recognition tensor at each position, determine the candidate character with the highest confidence in each position.

(3) Take the candidate character with the highest confidence in each position as the character recognition result at that position, and obtain the license plate recognition result of the target license plate image.

203. Fusion of the multiple license plate recognition results to determine the license plate recognition result of the target vehicle.

Wherein, fusing the multiple license plate recognition results to determine the license plate recognition result of the target vehicle may include:

(1) Fusion of the multiple license plate recognition results to obtain the merged license plate recognition result.

Wherein, the fusion of the plurality of license plate recognition results to obtain the fused license plate recognition result may further include: calculating the number of qualified characters whose character recognition results meet the preset reliability requirements in each license plate recognition result; taking qualified characters For the most N number of license plate recognition results, the license plate recognition results are fused, and the fused license plate recognition result is obtained.

In the experiment, the inventor also found that the rules are as follows: a) Clear and correctly recognized characters, the confidence of Top5 candidate characters, the confidence of Top1 character is much greater than the confidence of other candidate characters, for example, the confidence of Top1 is greater than 0.9, and The total confidence of the remaining candidate characters is 0.1; b) When the characters are blurred, the confidence of the Top5 candidate characters cannot be widened. For example, the confidence of Top1 is 0.6 and the confidence of Top2 is 0.3.

Therefore, in the embodiment of the present invention, a confidence threshold can be set, the number of qualified characters in the license plate recognition result that meets the confidence threshold can be calculated, and the number of qualified characters in the license plate recognition result can be used as an index for the quality of the license plate. For example, if the confidence threshold is 0.9, a certain license plate recognition result includes 6 characters, which are distributed in order of 0.95, 0.98, 0.97, 0.95, 0.89, 0.94. Among them, the number of characters that meet the confidence threshold is 5, that is, qualified characters The number is 5.

In the embodiment of the present invention, the N license plate recognition results with the largest number of qualified characters are taken, and the license plate recognition results are merged, and the following methods can be specifically adopted to obtain the merged license plate recognition result:

Since each license plate recognition result contains the position information of each character, based on this position information, character alignment is performed, and then the characters in each position are voted and sorted to obtain the best character at that position. The specific steps are as follows:

1. For the position information of each character in the license plate recognition result, the abscissa of the character vertex x and the character width width can be obtained according to the following formula, and the abscissa of the character center point can be calculated:

2. Take one of the N license plate recognition results as the reference license plate recognition result, and compare each character in the remaining N-1 license plate recognition results with the reference license plate recognition result from left to right. Specifically, set A threshold. If the minimum value of the abscissa distance between the center point of a character and the center points of all characters of the reference license plate recognition result is less than the threshold, then the character is assigned to the corresponding reference character;

3. After all the characters in the N-1 license plate recognition results are classified into the corresponding characters of the reference recognition result, then according to the character position of the reference license plate recognition result, vote for the characters in each position in turn from left to right, assuming There are m candidate characters (C ₁ , C ₁ ... C _m ) at a certain position. For a character C _i (i=1, 2..m), assuming the number of occurrences is l, the score of _{the character C i is} :

among them,

Is the score of the character C _i , and confidence _l is the confidence of the first occurrence of the character C _i.

4. The candidate character with the highest score is subsequently obtained as the optimal character at that position, and the optimal characters in all positions are arranged in the order from left to right, and the result of the fusion license plate recognition can be obtained.

(2) According to the fusion license plate recognition result, search in the preset license plate database to determine the license plate recognition result of the target vehicle.

In the above step (1), the multiple license plate recognition results are merged, and after the merged license plate recognition result is obtained, the merged license plate recognition result may be a complete license plate information, or it may only include a part of the license plate information. In this case, In order to ensure that the complete license plate recognition result is obtained, the license plate recognition result of the target vehicle can be determined by searching in a preset license plate database according to the fusion license plate recognition result. For example, take the license plate recognition scene at the loading and unloading port of the transfer yard as an example, according to the result of the license plate recognition after fusion, it can be matched in the vehicle database of the logistics transfer yard to obtain the complete license plate number.

In the embodiment of the present invention, multiple license plate images of the target vehicle are acquired; license plate recognition is performed on the multiple license plate images to obtain multiple license plate recognition results; multiple license plate recognition results are merged to determine the license plate recognition result of the target vehicle. In the embodiment of the invention, multiple license plate recognition results are obtained by recognizing multiple license plate images of the target vehicle, and the multiple license plate recognition results are merged to determine the license plate recognition result of the target vehicle. Because the final license plate recognition of the target vehicle is determined The result is determined by the fusion of multiple license plate recognition results, so it can effectively eliminate the influence of license plate misdetection and license plate motion blur on the license plate recognition result, and greatly improve the accuracy of license plate recognition.

The following describes the license plate recognition solution in the embodiment of the present invention in combination with a specific application scenario:

Traditional license plate recognition methods include license plate detection, image binarization, character segmentation, character recognition and other steps. Because license plate detection will have misdetection, it is generally necessary to add an image quality judgment link before image binarization. Using a small classification network to eliminate non-license plate and fuzzy license plate images, the traditional method contains so many links, which leads to a lot of manpower for data labeling and model training in each link. In addition, the final recognition accuracy rate is obtained by multiplying the accuracy rates of each link. Because there are too many links, and each link cannot reach 100%, there is a bottleneck that is difficult to break through in the recognition accuracy rate.

Therefore, as shown in FIG. 4, in the embodiment of the present invention, the image quality judgment, image binarization, character segmentation and character recognition in the existing license plate recognition process are integrated into a deep neural network (Attention-OCR). -In the OCR model training process, only the license plate number needs to be labeled, instead of the position of each character, which simplifies the workload of labeling, and uses the advantages of big data (large collection of license plate images and license plate number data). The license plate image is fully trained on the Attention-OCR network to achieve a high accuracy rate.

The Attention-OCR network structure is shown in Figure 1, including the following parts: a CNN network model and an LSTM network model. By inputting an image sequence in CNN, outputting the feature map corresponding to the image sequence, extracting the attention map from each feature map , Input the attention map into the LSTM to get the character recognition result tensor at time t, analyze the character recognition result tensor at time t to get the confidence of each candidate character, and take the character with the highest confidence as the character recognition result at that position. After visualizing the attention map, it is superimposed on the original license plate image, as shown in Figure 5, where the small square is the attention map.

The number of qualified characters in the license plate recognition result that meets the confidence threshold is calculated, the number of qualified characters in the license plate recognition result is used as the index of license plate quality judgment, and the five license plate recognition results with the largest number of qualified characters are taken to merge the license plate recognition results. According to the result of the license plate recognition after fusion, search in the preset license plate database to determine the license plate recognition result of the target vehicle.

In order to better implement the license plate recognition method in the embodiment of the present invention, on the basis of the license plate recognition method, an embodiment of the present invention also provides a license plate recognition device. As shown in FIG. 6, the license plate recognition device 600 includes:

The acquiring unit 601 is configured to acquire multiple license plate images of the target vehicle;

The recognition unit 602 is configured to perform license plate recognition on the multiple license plate images to obtain multiple license plate recognition results;

The determining unit 603 is configured to merge the multiple license plate recognition results to determine the license plate recognition result of the target vehicle.

In some embodiments of the present application, the identification unit 602 is specifically configured to:

Extracting the attention map of the characters in the feature map;

Weighting the attention map and the feature map to obtain a weighted map;

In some embodiments of the present application, the license plate character recognition tensor includes character recognition tensors at multiple positions, and the character recognition tensor at each position includes multiple candidate characters, and the recognition unit 602 is specifically configured to:

In some embodiments of the present application, the determining unit 603 is specifically configured to:

In some embodiments of the present application, the acquiring unit 601 is specifically configured to:

Acquiring the collected surveillance video of the target vehicle;

In the embodiment of the present invention, multiple license plate images of the target vehicle are acquired by the acquiring unit 601; the recognition unit 602 performs license plate recognition on the multiple license plate images to obtain multiple license plate recognition results; the determining unit 603 merges the multiple license plate recognition results, Determine the result of the license plate recognition of the target vehicle. In the embodiment of the invention, multiple license plate recognition results are obtained by recognizing multiple license plate images of the target vehicle, and the multiple license plate recognition results are merged to determine the license plate recognition result of the target vehicle. Because the final license plate recognition of the target vehicle is determined The result is determined by the fusion of multiple license plate recognition results, so it can effectively eliminate the influence of license plate misdetection and license plate motion blur on the license plate recognition result, and greatly improve the accuracy of license plate recognition.

An embodiment of the present invention also provides an electronic device that integrates any of the license plate recognition devices provided in the embodiments of the present invention, and the electronic device includes:

One or more processors;

Memory

And one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to execute the license plate described in any one of the foregoing license plate recognition method embodiments Identify the steps in the method.

The embodiment of the present invention also provides an electronic device that integrates any of the license plate recognition methods provided in the embodiment of the present invention. As shown in FIG. 7, it shows a schematic structural diagram of an electronic device involved in an embodiment of the present invention, specifically:

The electronic device may include one or more processing core processors 701, one or more computer-readable storage medium memory 702, power supply 703, input unit 704 and other components. Those skilled in the art can understand that the structure of the electronic device shown in FIG. 7 does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or a combination of certain components, or different component arrangements. among them:

The processor 701 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and/or modules stored in the memory 702, and calls Data, perform various functions of electronic equipment and process data, so as to monitor the electronic equipment as a whole. Optionally, the processor 701 may include one or more processing cores; preferably, the processor 701 may integrate an application processor and a modem processor, where the application processor mainly processes the operating system, user interface, application programs, etc. , The modem processor mainly deals with wireless communication. It can be understood that the foregoing modem processor may not be integrated into the processor 701.

The memory 702 may be used to store software programs and modules. The processor 701 executes various functional applications and data processing by running the software programs and modules stored in the memory 702. The memory 702 may mainly include a program storage area and a data storage area. The program storage area may store an operating system, an application program required by at least one function (such as a sound playback function, an image playback function, etc.), etc.; Data created by the use of electronic equipment, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory 702 may further include a memory controller to provide the processor 701 with access to the memory 702.

The electronic device also includes a power supply 703 for supplying power to various components. Preferably, the power supply 703 can be logically connected to the processor 701 through a power management system, so that functions such as charging, discharging, and power consumption management can be managed through the power management system. The power supply 703 may also include any components such as one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, and a power status indicator.

The electronic device may further include an input unit 704, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

Although not shown, the electronic device may also include a display unit, etc., which will not be repeated here. Specifically in this embodiment, the processor 701 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 702 according to the following instructions, and the processor 701 will run and store the executable file in the memory 702. The application programs in the memory 702, thereby realizing various functions, are as follows:

Acquire multiple license plate images of the target vehicle;

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by instructions, or by instructions to control related hardware, and the instructions can be stored in a computer-readable storage medium. It is loaded and executed by the processor.

To this end, an embodiment of the present invention provides a computer-readable storage medium, which may include: read only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disks or optical disks, etc. . A computer program is stored thereon, and the computer program is loaded by the processor to execute the steps in any of the license plate recognition methods provided in the embodiments of the present invention. For example, the computer program can be loaded by the processor to perform the following steps:

Acquire multiple license plate images of the target vehicle;

In the above-mentioned embodiments, the description of each embodiment has its own focus. For a part that is not described in detail in an embodiment, please refer to the detailed description of other embodiments above, which will not be repeated here.

In specific implementation, each of the above units or structures can be implemented as independent entities, or can be combined arbitrarily, and implemented as the same or several entities. For the specific implementation of each of the above units or structures, please refer to the previous method embodiments. No longer.

For the specific implementation of the above operations, please refer to the previous embodiments, which will not be repeated here.

The above provides a detailed introduction to the license plate recognition method, device, electronic device, and storage medium provided by the embodiments of the present invention. Specific examples are used in this article to illustrate the principles and implementations of the present invention. The description of the above embodiments is only It is used to help understand the method and its core idea of the present invention; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation and the scope of application. In summary, this specification The content should not be construed as limiting the present invention.

Claims

A method for recognizing a license plate, wherein the method for recognizing a license plate includes:

Acquire multiple license plate images of the target vehicle;

Performing license plate recognition on the multiple license plate images to obtain multiple license plate recognition results;

Fusion of the multiple license plate recognition results to determine the license plate recognition result of the target vehicle.
The license plate recognition method according to claim 1, wherein said performing license plate recognition on the plurality of license plate images respectively to obtain multiple license plate recognition results comprises:

Respectively taking the license plate images in the plurality of license plate images as target license plate images, extracting features of the target license plate images, and obtaining feature maps corresponding to the license plate images;

Extracting the attention map of the characters in the feature map;

Calculating, according to the attention map, a license plate character recognition result tensor corresponding to the target license plate image;

According to the license plate character recognition tensor, the license plate recognition result of the target license plate image is determined.
The license plate recognition method according to claim 2, wherein said extracting the features of the target license plate image to obtain the feature map corresponding to the license plate image comprises:

The target license plate image is input into a preset convolutional neural network model to extract features corresponding to the license plate image using the convolutional neural network model, and output a feature map corresponding to the target license plate image.
The license plate recognition method according to claim 2, wherein the calculating the license plate character recognition result tensor according to the attention map comprises:

Weighting the attention map and the feature map to obtain a weighted map;

The weighted graph is input into a preset long-term short-term memory network model, and the license plate character recognition result tensor corresponding to the target license plate image is output.
The license plate recognition method according to claim 2, wherein the license plate character recognition tensor includes multiple positions of the character recognition tensor, and the character recognition tensor at each position includes a plurality of candidate characters, and the character recognition tensor of the license plate includes multiple candidate characters. The recognition tensor to determine the license plate recognition result of the target license plate image includes:

Parse the license plate character recognition tensor to obtain the confidence level of each candidate character in the character recognition tensor at each position;

Determine the candidate character with the highest confidence in each position according to the confidence of each candidate character in the character recognition tensor of each position;

The candidate character with the highest confidence in each position is taken as the character recognition result of the position, and the license plate recognition result of the target license plate image is obtained.
The license plate recognition method according to claim 1, wherein the fusing the multiple license plate recognition results to determine the license plate recognition result of the target vehicle comprises:

Fusing the multiple license plate recognition results to obtain a fused license plate recognition result;

According to the result of the license plate recognition after fusion, search in the preset license plate database to determine the license plate recognition result of the target vehicle.
The license plate recognition method according to claim 6, wherein the fusing the multiple license plate recognition results to obtain the fused license plate recognition result comprises:

Calculate the number of qualified characters whose character recognition results meet the preset reliability requirements in each license plate recognition result;

Take the N license plate recognition results with the largest number of qualified characters, perform the fusion of the license plate recognition results, and obtain the fused license plate recognition results.
The license plate recognition method according to any one of claims 1 to 7, wherein said acquiring multiple license plate images of the target vehicle comprises:

Acquiring the collected surveillance video of the target vehicle;

Detecting the license plate in the surveillance video to obtain multiple images including the license plate;

The license plate area in the multiple images is cropped and normalized to a preset size to obtain multiple license plate images of the target vehicle.
The license plate recognition method according to claim 8, wherein the detecting the license plate in the surveillance video to obtain a plurality of images including the license plate includes:

A multi-target detection method is used to detect the license plate in the surveillance video to obtain multiple images including the license plate.
A license plate recognition device, wherein the license plate recognition device includes:

The acquiring unit is used to acquire multiple license plate images of the target vehicle;

A recognition unit, configured to perform license plate recognition on the multiple license plate images to obtain multiple license plate recognition results;

The determining unit is used for fusing the multiple license plate recognition results to determine the license plate recognition result of the target vehicle.
The license plate recognition device according to claim 10, wherein the recognition unit is specifically configured to:

Respectively taking the license plate images in the plurality of license plate images as target license plate images, extracting features of the target license plate images, and obtaining feature maps corresponding to the license plate images;

Extracting the attention map of the characters in the feature map;

Calculating, according to the attention map, a license plate character recognition result tensor corresponding to the target license plate image;

According to the license plate character recognition tensor, the license plate recognition result of the target license plate image is determined.
The license plate recognition device according to claim 11, wherein the recognition unit is specifically configured to:

The target license plate image is input into a preset convolutional neural network model to extract features corresponding to the license plate image using the convolutional neural network model, and output a feature map corresponding to the target license plate image.
The license plate recognition device according to claim 11, wherein the recognition unit is specifically configured to:

Weighting the attention map and the feature map to obtain a weighted map;

The weighted graph is input into a preset long-term short-term memory network model, and the license plate character recognition result tensor corresponding to the target license plate image is output.
The license plate recognition device according to claim 11, wherein the license plate character recognition tensor includes multiple positions of the character recognition tensor, and the character recognition tensor of each position includes a plurality of candidate characters, and the recognition unit is specifically configured to :

Parse the license plate character recognition tensor to obtain the confidence level of each candidate character in the character recognition tensor at each position;

Determine the candidate character with the highest confidence in each position according to the confidence of each candidate character in the character recognition tensor of each position;

The candidate character with the highest confidence in each position is taken as the character recognition result of the position, and the license plate recognition result of the target license plate image is obtained.
The license plate recognition device according to claim 10, wherein the determining unit is specifically configured to:

Fusing the multiple license plate recognition results to obtain a fused license plate recognition result;

According to the result of the license plate recognition after fusion, search in the preset license plate database to determine the license plate recognition result of the target vehicle.
The license plate recognition device according to claim 15, wherein the determining unit is specifically configured to:

Calculate the number of qualified characters whose character recognition results meet the preset reliability requirements in each license plate recognition result;

Take the N license plate recognition results with the largest number of qualified characters, perform the fusion of the license plate recognition results, and obtain the fused license plate recognition results.
The license plate recognition device according to any one of claims 10 to 16, wherein the acquiring unit is specifically configured to:

Acquiring the collected surveillance video of the target vehicle;

Detecting the license plate in the surveillance video to obtain multiple images including the license plate;

The license plate area in the multiple images is cropped and normalized to a preset size to obtain multiple license plate images of the target vehicle.
The license plate recognition device according to claim 17, wherein the acquiring unit is specifically configured to:

A multi-target detection method is used to detect the license plate in the surveillance video to obtain multiple images including the license plate.
An electronic device, wherein the electronic device includes:

One or more processors;

Memory; and

One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the license plate recognition method according to any one of claims 1 to 9 .
A computer-readable storage medium, wherein a computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the license plate recognition method according to any one of claims 1 to 9.