CN116311205A

CN116311205A - License plate recognition method, license plate recognition device, electronic equipment and storage medium

Info

Publication number: CN116311205A
Application number: CN202310212915.8A
Authority: CN
Inventors: 张兴; 郑毅; 王伟; 陈镭; 李昕尧; 张黔; 陈焕坤; 曾志贤
Original assignee: Guangdong Runlian Information Technology Co ltd; China Resources Digital Technology Co Ltd
Current assignee: Guangdong Runlian Information Technology Co ltd; China Resources Digital Technology Co Ltd
Priority date: 2023-02-27
Filing date: 2023-02-27
Publication date: 2023-06-23

Abstract

The application provides a license plate recognition method, a license plate recognition device, electronic equipment and a storage medium, which belong to the technical field of artificial intelligence, and are characterized in that an initial license plate image of a target vehicle is obtained, the initial license plate image comprises a license plate of the target vehicle, the length-width ratio of the license plate in the initial license plate image is calculated to obtain license plate length-width ratio data, the license plate type is determined according to the license plate length-width ratio data, the license plate type comprises double-row license plates, if the license plate type is the double-row license plate, the initial license plate image is cut to obtain a first intermediate license plate image and a second intermediate license plate image, the first intermediate license plate image and the second intermediate license plate image are subjected to image stitching to obtain the target license plate image, the feature extraction is carried out on the target license plate image to obtain a first license plate feature map, the first license plate feature map is input into a preset bidirectional long short-time memory network to carry out license plate recognition, and license plate information of the license plates is obtained, and the accuracy of identifying the double-row license plates is improved.

Description

License plate recognition method, license plate recognition device, electronic equipment and storage medium

Technical Field

The application relates to the technical field of artificial intelligence, in particular to a license plate recognition method, a license plate recognition device, electronic equipment and a storage medium.

Background

The existing license plate recognition algorithm based on character segmentation or uplink and downlink cutting lines has good license plate recognition effect only on a single-row license plate, and the accuracy of the license plate recognition algorithm on the double-row license plate is reduced because the characters of the upper and lower double-row license plates are inconsistent in size and number, so that the structure of the double-row license plate is complex.

Disclosure of Invention

The embodiment of the application mainly aims to provide a license plate recognition method, a license plate recognition device, electronic equipment and a storage medium, and aims to improve the accuracy of license plate recognition on double-row license plates.

In order to achieve the above object, a first aspect of an embodiment of the present application provides a license plate recognition method, where the method includes:

acquiring an initial license plate image of a target vehicle, wherein the initial license plate image comprises a license plate of the target vehicle;

calculating the length-width ratio of the license plate in the initial license plate image to obtain license plate length-width ratio data;

determining the license plate type of the license plate according to the length-width ratio data of the license plate, wherein the license plate type comprises double-row license plates;

If the license plate type is the double-row license plate, cutting the initial license plate image to obtain a first intermediate license plate image and a second intermediate license plate image;

performing image stitching on the first intermediate license plate image and the second intermediate license plate image to obtain a target license plate image;

extracting features of the target license plate image to obtain a first license plate feature map;

and inputting the first license plate feature map into a preset two-way long and short time memory network to perform license plate recognition, so as to obtain license plate information of the license plate.

In some embodiments, the license plate type further comprises a single row license plate, and the license plate recognition method further comprises, after determining the license plate type of the license plate from the license plate aspect ratio data:

if the license plate type is the single-row license plate, extracting features of the initial license plate image to obtain a second license plate feature map;

inputting the second license plate feature map into the two-way long and short-time memory network for license plate recognition to obtain license plate characters of the license plate;

and performing de-duplication processing on the license plate characters to obtain the license plate information.

In some embodiments, the acquiring an initial license plate image of the target vehicle includes:

Acquiring a vehicle image of a target vehicle, wherein the vehicle image comprises a license plate of the target vehicle;

detecting license plates of the vehicle images to obtain first license plate images;

and preprocessing the first license plate image to obtain the initial license plate image.

In some embodiments, the preprocessing the first license plate image to obtain the initial license plate image includes:

carrying out gray level transformation on the first car image to obtain a second car image;

extracting edge characteristics of the second license plate image to obtain a third license plate image;

performing angle detection on the third license plate image to obtain the inclination angle of the license plate in the third license plate image;

and correcting the third license plate image according to the inclination angle to obtain the initial license plate image.

In some embodiments, the performing gray level transformation on the first card image to obtain a second card image includes:

extracting an image channel of the first license plate image to obtain a red channel license plate image, a green channel license plate image and a blue channel license plate image;

and carrying out graying treatment on the red channel license plate image, the green channel license plate image and the blue channel license plate image based on preset weight data to obtain the second license plate image.

In some embodiments, the extracting the edge feature of the second license plate image to obtain a third license plate image includes:

performing edge detection on the second card image to obtain a first edge image;

non-edge suppression is carried out on the first edge image, and a second edge image is obtained;

and removing edges of the second edge image to obtain the third license plate image.

In some embodiments, the license plate type further comprises a single row license plate, the determining the license plate type of the license plate from the license plate aspect ratio data comprises:

calculating the difference between the plate length-width ratio data and a preset single-row plate threshold value to obtain a first difference value;

calculating the difference between the license plate length-width ratio data and a preset double-row license plate threshold value to obtain a second difference value;

if the first difference value is smaller than the second difference value, determining the license plate type as the single-row license plate;

and if the first difference value is larger than the second difference value, determining the license plate type as the double-row license plate.

To achieve the above object, a second aspect of the embodiments of the present application proposes a license plate recognition device, the device including:

The acquisition module is used for acquiring an initial license plate image of a target vehicle, wherein the initial license plate image comprises a license plate of the target vehicle;

the calculating module is used for calculating the length-width ratio of the license plate in the initial license plate image to obtain license plate length-width ratio data;

the license plate type determining module is used for determining the license plate type of the license plate according to the length-width ratio data of the license plate, and the license plate type comprises double-row license plates;

the clipping module is used for clipping the initial license plate image if the license plate type is the double-row license plate to obtain a first intermediate license plate image and a second intermediate license plate image;

the image stitching module is used for stitching the first intermediate license plate image and the second intermediate license plate image to obtain a target license plate image;

the feature extraction module is used for extracting features of the target license plate image to obtain a first license plate feature map;

and the license plate recognition module is used for inputting the first license plate feature map into a preset two-way long and short-time memory network to perform license plate recognition, so as to obtain license plate information of the license plate.

To achieve the above object, a third aspect of the embodiments of the present application provides an electronic device, where the electronic device includes a memory and a processor, where the memory stores a computer program, and the processor implements the license plate recognition method according to the first aspect when executing the computer program.

To achieve the above object, a fourth aspect of the embodiments of the present application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the license plate recognition method described in the first aspect.

According to the license plate recognition method, the license plate recognition device, the electronic equipment and the computer readable storage medium, the length-width ratio of the license plate in the initial license plate image is calculated through acquiring the initial license plate image of the target vehicle, the license plate length-width ratio data is obtained, the license plate type of the license plate is determined according to the license plate length-width ratio data, and the single-row license plate and the double-row license plate are distinguished through the license plate length-width ratio data so as to carry out different treatments on the single-row license plate and the double-row license plate. Further, if the license plate type is a double-row license plate, the initial license plate image is cut to obtain a first middle license plate image and a second middle license plate image, and the upper row and the lower row of the double-row license plate are split into two independent rows, so that each row can be conveniently and respectively processed. Furthermore, the first middle license plate image and the second middle license plate image are subjected to image stitching to obtain a target license plate image, and the double-row license plate can be converted into a single-row license plate. Finally, feature extraction is carried out on the target license plate image to obtain a first license plate feature map, effective license plate features can be extracted, the first license plate feature map is input into a preset two-way long and short-time memory network to carry out license plate recognition, license plate information of a license plate is obtained, and the license plate recognition is carried out through the effective license plate features instead of the license plate image, so that the accuracy and the stability of double-row license plate recognition are improved while the data processing amount is reduced.

Drawings

Fig. 1 is a flowchart of a license plate recognition method provided in an embodiment of the present application;

fig. 2 is a flowchart of step S110 in fig. 1;

fig. 3 is a flowchart of step S230 in fig. 2;

fig. 4 is a flowchart of step S310 in fig. 3;

fig. 5 is a flowchart of step S320 in fig. 3;

fig. 6 is a flowchart of step S130 in fig. 1;

FIG. 7 is another flowchart of a license plate recognition method according to an embodiment of the present disclosure;

fig. 8 is a schematic structural diagram of a license plate recognition device according to an embodiment of the present application;

fig. 9 is a schematic hardware structure of an electronic device according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.

It should be noted that although functional block division is performed in a device diagram and a logic sequence is shown in a flowchart, in some cases, the steps shown or described may be performed in a different order than the block division in the device, or in the flowchart. The terms first, second and the like in the description and in the claims and in the above-described figures, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order.

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of the present application only and is not intended to be limiting of the present application.

First, several nouns referred to in this application are parsed:

artificial intelligence (Artificial Intelligence, AI): is a new technical science for researching and developing theories, methods, technologies and application systems for simulating, extending and expanding the intelligence of people; artificial intelligence is a branch of computer science that attempts to understand the nature of intelligence and to produce a new intelligent machine that can react in a manner similar to human intelligence, research in this field including robotics, language recognition, image recognition, natural language processing, and expert systems. Artificial intelligence can simulate the information process of consciousness and thinking of people. Artificial intelligence is also a theory, method, technique, and application system that utilizes a digital computer or digital computer-controlled machine to simulate, extend, and expand human intelligence, sense the environment, acquire knowledge, and use knowledge to obtain optimal results.

License plate recognition (Vehicle License Plate Recognition, VLPR): the method is an important component of a modern intelligent traffic system, and is a technology capable of detecting vehicles on a monitored road surface and automatically extracting and processing vehicle license plate information (Chinese characters, english letters, arabic numerals, license plate colors), and the technology is based on digital image processing, pattern recognition, computer vision and other technologies, and analyzes vehicle images or video sequences shot by a camera to obtain unique license plate numbers of each vehicle. The license plate recognition technology is applied to the fields of parking lot charge management, expressway toll gate charge, traffic flow control index measurement, vehicle positioning, automobile theft prevention, expressway overspeed automatic supervision, traffic light electronic police and the like, and has important significance for maintaining traffic safety and urban security, preventing traffic jam and realizing traffic automatic management.

Join time class loss function (Connectionist Temporal Classification, CTC): the method is a loss function, model output and label alignment are not needed in the calculation process, the workload of data alignment labeling is greatly reduced, and the efficiency is greatly improved.

The license plate number is a unique identity of the vehicle, and the license plate automatic recognition technology (license plate recognition technology) can realize automatic registration and verification of the identity of the vehicle under the condition that the vehicle is not changed at all. The license plate is structurally divided into a single-row license plate and a double-row license plate, compared with the single-row license plate, the double-row license plate is structurally complex, for example, the character sizes and the character numbers of the characters of the upper and lower two-row license plates of the double-row license plate are inconsistent, and the traditional license plate recognition algorithm based on character segmentation or uplink and downlink cutting lines is poor in stability and low in accuracy due to the influence of factors such as fuzzy, inclination and shielding of the license plate caused by environmental noise.

Based on this, the embodiment of the application provides a license plate recognition method, a license plate recognition device, electronic equipment and a computer readable storage medium, aiming at improving the accuracy of double-row license plate recognition.

The license plate recognition method, the license plate recognition device, the electronic equipment and the computer readable storage medium provided by the embodiment of the application are specifically described through the following embodiments, and the license plate recognition method in the embodiment of the application is described first.

The embodiment of the application can acquire and process the related data based on the artificial intelligence technology. Among these, artificial intelligence (Artificial Intelligence, AI) is the theory, method, technique and application system that uses a digital computer or a digital computer-controlled machine to simulate, extend and extend human intelligence, sense the environment, acquire knowledge and use knowledge to obtain optimal results.

Artificial intelligence infrastructure technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a robot technology, a biological recognition technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and other directions.

The embodiment of the application provides a license plate recognition method, which relates to the technical field of artificial intelligence. The license plate recognition method provided by the embodiment of the application can be applied to a terminal, a server side and software running in the terminal or the server side. In some embodiments, the terminal may be a smart phone, tablet, notebook, desktop, etc.; the server side can be configured as an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, and a cloud server for providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, basic cloud computing services such as big data and artificial intelligent platforms and the like; the software may be an application or the like that implements the license plate recognition method, but is not limited to the above form.

The subject application is operational with numerous general purpose or special purpose computer system environments or configurations. For example: personal computers, server computers, hand-held or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like. The application may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The application may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

In the embodiments of the present application, when related processing is required according to user information, user behavior data, user history data, user location information, and other data related to user identity or characteristics, permission or consent of the user is obtained first, and the collection, use, processing, and the like of the data comply with related laws and regulations and standards of related countries and regions. In addition, when the embodiment of the application needs to acquire the sensitive personal information of the user, the independent permission or independent consent of the user is acquired through a popup window or a jump to a confirmation page or the like, and after the independent permission or independent consent of the user is explicitly acquired, necessary user related data for enabling the embodiment of the application to normally operate is acquired.

Fig. 1 is an optional flowchart of a license plate recognition method provided in an embodiment of the present application, where the method in fig. 1 may include, but is not limited to, steps S110 to S170.

Step S110, acquiring an initial license plate image of a target vehicle, wherein the initial license plate image comprises a license plate of the target vehicle;

step S120, calculating the length-width ratio of a license plate in an initial license plate image to obtain license plate length-width ratio data;

step S130, determining license plate type of the license plate according to the length-width ratio data of the license plate, wherein the license plate type comprises double-row license plates;

step S140, if the license plate type is a double-row license plate, cutting the initial license plate image to obtain a first intermediate license plate image and a second intermediate license plate image;

step S150, performing image stitching on the first intermediate license plate image and the second intermediate license plate image to obtain a target license plate image;

step S160, extracting features of a target license plate image to obtain a first license plate feature map;

step S170, inputting the first license plate feature map into a preset two-way long and short time memory network to perform license plate recognition, and obtaining license plate information of the license plate.

Step S110 to step S170 illustrated in the embodiment of the present application, by obtaining an initial license plate image of a target vehicle, calculating an aspect ratio of a license plate in the initial license plate image, obtaining license plate aspect ratio data, determining a license plate type of the license plate according to the license plate aspect ratio data, and distinguishing a single-row license plate from a double-row license plate according to the license plate aspect ratio data, so as to perform different treatments on the single-row license plate and the double-row license plate. Further, if the license plate type is a double-row license plate, the initial license plate image is cut to obtain a first middle license plate image and a second middle license plate image, and the upper row and the lower row of the double-row license plate are split into two independent rows, so that each row can be conveniently and respectively processed. Furthermore, the first middle license plate image and the second middle license plate image are subjected to image stitching to obtain a target license plate image, and the double-row license plate can be converted into a single-row license plate. Finally, feature extraction is carried out on the target license plate image to obtain a first license plate feature map, effective license plate features can be extracted, the first license plate feature map is input into a preset two-way long-short-time memory network to carry out license plate recognition, license plate information of a license plate is obtained, the license plate recognition is carried out through the effective license plate features instead of the license plate image, the data processing capacity of the network is reduced, the license plate recognition is carried out through the two-way long-short-time memory network, and the accuracy and the stability of double-row license plate recognition are improved.

Referring to fig. 2, in some embodiments, step S110 may include, but is not limited to, steps S210 to S230:

step S210, acquiring a vehicle image of a target vehicle, wherein the vehicle image comprises a license plate of the target vehicle;

step S220, license plate detection is carried out on the vehicle image to obtain a first license plate image;

step S230, preprocessing the first license plate image to obtain an initial license plate image.

In step S210 of some embodiments, a vehicle image of a target vehicle on a monitored road surface, which is captured by a camera, is acquired, where the monitored road surface may be a road section such as a traffic highway entrance, an automobile parking lot entrance, a garage entrance, etc., and the vehicle image may be a still image including a license plate of the target vehicle, or may be a video frame including a license plate of the target vehicle in a dynamic video sequence.

In step S220 of some embodiments, the vehicle image is input to the trained license plate detection model to perform license plate detection, so as to obtain position information of key points of the license plate, namely, positions of the license plate in the vehicle image, and the license plate region is cut out from the vehicle image according to the positions, so as to obtain a first license plate image. And carrying out license plate key point training by using a Yo l oV7 algorithm to obtain a license plate detection model, wherein the training process of the license plate detection model is as follows: obtaining sample license plate data, wherein the sample license plate data comprises sample license plate pictures and sample license plate position information, the sample license plate position information is used for representing the positions of license plates in the sample license plate pictures, the sample license plate data is input into a YoloV7 target detection model, license plate position recognition is carried out on the sample license plate pictures through the YoloV7 target detection model, predicted license plate position information is obtained, loss data are calculated according to the sample license plate position information and the predicted license plate position information, and when the loss data are minimum or the iteration number of the YoloV7 target detection model is larger than an iteration number threshold value, updating of parameters of the YoloV7 target detection model is stopped, and a license plate detection model is obtained. It can be understood that the loss data can be obtained by performing loss calculation by using the sample license plate position information and the predicted license plate position information as input parameters of the loss function. The YoloV7 target detection model has the characteristics of high detection precision, high speed, strong real-time performance and the like, and by adopting the YoloV7 target detection model as the license plate detection model, license plate detection can be performed in real time and accurately, so that the speed and the precision of license plate detection are improved.

It should be noted that, the picture with the license plate in the monitoring images of the traffic highway entrance, the car parking lot entrance, the garage entrance and the like can be extracted to be used as an initial sample license plate picture, the picture with the license plate can be obtained in a web crawler mode to be used as an initial sample license plate picture to expand sample data, and the initial sample license plate picture is subjected to license plate position marking to obtain sample license plate position information of the initial sample license plate picture. And cleaning the data of the initial sample license plate picture to remove unqualified data. And sorting all qualified initial sample license plate pictures and sample license plate position information thereof to obtain sample license plate data.

In step S230 of some embodiments, in order to reduce the influence of factors such as license plate blurring, tilting, shielding, etc. caused by environmental noise on the license plate recognition effect, the first license plate image is preprocessed to obtain an initial license plate image.

In the steps S210 to S230, the license plate can be located from the vehicle image by detecting the license plate of the vehicle image, so as to obtain the first license plate image, and reduce the data processing amount in the subsequent steps. By preprocessing the first license plate image, the influence of factors such as license plate blurring, tilting and shielding on license plate recognition effect can be reduced, and the accuracy and stability of license plate recognition are improved.

Referring to fig. 3, in some embodiments, step S230 may include, but is not limited to, steps S310 to S340:

step S310, gray level transformation is carried out on the first car plate image to obtain a second car plate image;

step S320, extracting edge characteristics of the second license plate image to obtain a third license plate image;

step S330, angle detection is carried out on the third license plate image, and the inclination angle of the license plate in the third license plate image is obtained;

and S340, correcting the third license plate image according to the inclination angle to obtain an initial license plate image.

In step S310 of some embodiments, in order to increase the speed of license plate recognition and reduce the amount of calculation, if the first license plate image is a color image, the first license plate image is subjected to gray level conversion, and the color image is converted into a gray level image, so as to obtain the second license plate image. If the first car image is a gray image, edge feature extraction is directly carried out on the first car image.

In step S320 of some embodiments, the image edge is a region where the local gray level of the image changes significantly, is the most basic feature of the image, and contains important information for image recognition. Because the image edge has invariance, the image edge is not influenced by light ray transformation or other external factors, and the vision system is most sensitive to the image edge, the license plate edge feature can be obtained by extracting the edge feature of the second license plate image, the license plate is conveniently identified based on the license plate edge feature, and the accuracy and the stability of license plate identification are improved.

In step S330 of some embodiments, in order to reduce the influence of license plate inclination on license plate recognition effect, the hough transform is utilized to perform up-down straight line detection and inclination angle detection, so as to obtain an angle of an included angle between the upper and lower straight lines of the license plate and the horizontal straight line, and the angle is taken as the inclination angle.

In step S340 of some embodiments, the perspective transformation-based inclination angle correction algorithm rotates the third license plate image by an angle equal to the inclination angle value, so as to complete license plate inclination correction, and obtain an initial license plate image.

Step S310 to step S340, if the first license plate image is a color image, performing gray level conversion on the first license plate image to obtain a second license plate image, so as to simplify the image matrix and increase the operation speed, thereby increasing the speed of license plate recognition. Because the edge features are not easily influenced by external factors and contain information for license plate recognition, the edge features of the license plate image are acquired by extracting the edge features of the second license plate image, and the license plate recognition is performed according to the edge features. The problem of poor license plate recognition effect caused by license plate inclination can be solved by correcting the third license plate image according to the inclination angle.

Referring to fig. 4, in some embodiments, step S310 may include, but is not limited to, steps S410 to S420:

step S410, extracting an image channel of the first license plate image to obtain a red channel license plate image, a green channel license plate image and a blue channel license plate image;

step S420, gray processing is carried out on the red channel license plate image, the green channel license plate image and the blue channel license plate image based on preset weight data, and a second license plate image is obtained.

In step S410 of some embodiments, the color image includes a red channel (R channel) image component, a green channel image (G channel) image component, and a blue channel (B channel) image component. And if the first car image is a color image, carrying out weighted average on R, G, B three channels of the first car image according to a certain weight value so as to carry out gray level conversion on the first car image and obtain a second car image. Specifically, if the first license plate image is a color image, image channel extraction is performed on the first license plate image to obtain a red channel (R channel) license plate image, a green channel (G channel) license plate image and a blue channel (B channel) license plate image.

In step S420 of some embodiments, the preset weight data includes first weight data, second weight data, and third weight data Multiplying the first weight data with the gray value of each pixel point in the red channel license plate image to obtain a first image, multiplying the second weight data with the gray value of each pixel point in the green channel license plate image to obtain a second image, multiplying the third weight data with the gray value of each pixel point in the blue channel license plate image to obtain a third image, and carrying out image fusion on the first image, the second image and the third image to obtain a second license plate image. If the red channel license plate image is expressed as I _R (x, y), green channel license plate image is denoted as I _G (x, y), blue channel license plate image is denoted as I _B The calculation method of the second card image is shown in the formula (1).

I(x,y)＝0.3×I _R (x,y)+0.59×I _G (x,y)+0.11×I _B (x, y) formula (1)

Wherein (x, y) represents coordinates of a pixel point in the image space, I (x, y) is a pixel value of the pixel point at the (x, y) position in the second vehicle image, the first weight data is 0.3, the second weight data is 0.59, and the third weight data is 0.11.

Through the steps S410 to S420, the color image can be converted into the gray image, so that the number of image channels is reduced, the calculated amount is reduced, and the license plate recognition speed is improved.

Referring to fig. 5, in some embodiments, step S320 may include, but is not limited to, steps S510 to S530:

Step S510, performing edge detection on the second card image to obtain a first edge image;

step S520, performing non-edge suppression on the first edge image to obtain a second edge image;

and step S530, performing edge removal on the second edge image to obtain a third license plate image.

In step S510 of some embodiments, edge detection is performed on the second tile image by using a canny edge detection operator to obtain a first edge image, where the canny edge detection operator includes an x-direction convolution operator Sx and a y-direction convolution operator Sy, the x-direction convolution operator Sx is used for performing pixel convolution calculation along an x-axis direction of the second tile image, the y-direction convolution operator Sy is used for performing pixel convolution calculation along a y-axis direction of the second tile image, the pixel convolution calculation includes gradient magnitude calculation and gradient direction calculation, the x-direction convolution operator Sx is shown in formula (2), and the y-direction convolution operator Sy is shown in formula (3).

The x-direction convolution operator Sx slides in the x-axis direction of the second car image, the y-direction convolution operator Sy slides in the y-axis direction of the second car image, when the Sx completely covers the area of the second car image, the area is used as a first target area, dot product operation is carried out on the pixel value of the pixel point of the first target area and the pixel value of the Sx, so as to obtain a first gradient amplitude, wherein the first gradient amplitude is the gradient amplitude of the origin in the x-direction, and the first gradient amplitude is shown in a formula (4).

P (x, y) = (I (x, y+1) -I (x, y) +I (x+1, y+1) -I (x+1, y))/2 formula (4)

Wherein P (x, y) is the first gradient amplitude of the (x, y) pixel point in the second image, the first gradient amplitude of all the pixel points in the second image forms a first order partial derivative matrix of the second image in the x direction, the length of the first order partial derivative matrix is the length of the second image, and the width of the first order partial derivative matrix is the width of the second image.

When the Sy completely covers the area of the second vehicle image, taking the area as a second target area, performing dot product operation on the pixel value of the pixel point of the second target area and the pixel value of the Sy to obtain a second gradient amplitude, wherein the first target area and the second target area have the same origin, and the second gradient amplitude is the gradient amplitude of the origin in the y direction, as shown in a formula (5).

Q (x, y) = (I (x, y) -I (x+1, y) +I (x, y+1) -I (x+1, y+1))/2 formula (5)

Wherein Q (x, y) is a second gradient amplitude value of the (x, y) pixel points in the second image, the second gradient amplitude values of all the pixel points in the second image form a first order partial derivative matrix of the second image in the y direction, the length of the first order partial derivative matrix is the length of the second image, and the width of the first order partial derivative matrix is the width of the second image.

And obtaining a target gradient amplitude according to the first gradient amplitude and the second gradient amplitude, wherein the target gradient amplitude is the gradient amplitude of the origin, as shown in a formula (6).

Wherein M (x, y) is a target gradient magnitude at a pixel point (x, y) in the second vehicle image.

And obtaining a target gradient direction according to the first gradient amplitude and the second gradient amplitude, wherein the target gradient direction is the gradient direction of the origin, as shown in a formula (7).

θ (x, y) =arctan (Q (x, y)/P (x, y)) formula (7)

And θ (x, y) is the target gradient direction of the (x, y) pixel point in the second car image.

If the target gradient amplitude is larger than the first threshold, taking pixel points corresponding to the target gradient amplitude and the target gradient direction as edge points, and taking all the edge points as edge features when all the pixels on the second vehicle image are traversed, so as to obtain a first edge image.

In step S520 of some embodiments, the larger the target gradient magnitude of the pixel point, the more likely the pixel point is an edge point, and the non-edge point needs to be removed by the non-maximum threshold to obtain the second edge image. Specifically, edge detection is performed through non-maximum suppression to obtain a local maximum value, namely a maximum value, of the pixel point, the local maximum value can be set to 128, a gray value corresponding to the non-maximum pixel point is set to 0, and non-edge points are removed to obtain a second edge image, wherein the second edge image is a binary image.

In step S530 of some embodiments, the gray values of the non-edge pixels in the second edge image are all 0, and the gray values of the edge pixels are 128, but the edge pixels may be false edge points caused by noise interference or other factors, so that the false edges need to be removed by the dual-threshold method. Specifically, when the target gradient amplitude of the edge pixel point is greater than or equal to a second threshold value, marking the edge pixel point as a strong edge point, when the target gradient amplitude of the edge pixel point is greater than or equal to a third threshold value and smaller than the second threshold value, marking the edge pixel point as a weak edge point, wherein the third threshold value is smaller than the second threshold value, when the weak edge point is connected with the strong edge point, the weak edge point is reserved, when the strong edge point is not connected with the weak edge point, the weak edge point is deleted, and when the edge pixel point is smaller than the third threshold value, the edge pixel point is restrained, and finally, a third license plate image is obtained.

Through the steps S510 to S530, edge features of the license plate image can be accurately extracted, and influence of non-edge features on license plate recognition is avoided.

In step S120 of some embodiments, the length and width of the license plate in the initial license plate image are obtained, and the ratio of the length to the width is used as the license plate aspect ratio data. If the length of the license plate is denoted as H, the width is denoted as W, and the aspect ratio data of the license plate is denoted as K, k=h/W.

Referring to fig. 6, in some embodiments, the license plate type further includes a single row license plate, and step S130 may include, but is not limited to, steps S610 to S640:

step S610, calculating a difference between the aspect ratio data of the license plate and a preset single-row license plate threshold value to obtain a first difference value;

step S620, calculating the difference between the aspect ratio data of the license plate and a preset double-row license plate threshold value to obtain a second difference value;

step S630, if the first difference is smaller than the second difference, determining the license plate type as a single-row license plate;

in step S640, if the first difference is greater than the second difference, the license plate type is determined as a double-row license plate.

In step S610 of some embodiments, the absolute value of the single-row license plate threshold subtracted from the license plate aspect ratio data is used as the first difference, where the single-row license plate threshold may be adjusted according to the actual situation, and the specific value of the single-row license plate threshold is not limited in the embodiments of the present application.

In step S620 of some embodiments, the absolute value of the two-row license plate threshold subtracted from the license plate aspect ratio data is used as the second difference, where the two-row license plate threshold is different from the one-row license plate threshold, and the two-row license plate threshold can be adjusted according to the actual situation.

In step S630 of some embodiments, if the first difference is smaller than the second difference, which indicates that the license plate aspect ratio data is closer to the single-row license plate threshold, the probability that the license plate type is a single-row license plate is greater, and the license plate type is determined to be a single-row license plate.

In step S640 of some embodiments, if the first difference is greater than the second difference, which indicates that the license plate aspect ratio data is closer to the double license plate threshold, and the probability of the license plate type being the double license plate is greater, the license plate type is determined to be the double license plate. If the first difference is equal to the second difference, the set single-row license plate threshold or double-row license plate threshold is unreasonable, and the single-row license plate threshold or the double-row license plate threshold needs to be reset. Or when the first difference is equal to the second difference, acquiring the license plate types of the first N license plates, wherein N is an integer larger than 1, when the probability of the license plate type being a single-row license plate is larger than that of the license plate type being a double-row license plate, determining the current license plate type as the single-row license plate, when the probability of the license plate type being the double-row license plate is larger than that of the single-row license plate, determining the current license plate type as the double-row license plate, when the probability of the license plate type being the single-row license plate is equal to that of the double-row license plate, and resetting a single-row license plate threshold or a double-row license plate threshold.

Compared with the traditional license plate recognition method, the steps S610 to S640 can automatically detect whether the license plate type is a single-row license plate or a double-row license plate, so as to respectively process the single-row license plate and the double-row license plate differently.

Referring to fig. 7, in some embodiments, after step S130, the license plate recognition method may include, but is not limited to, steps S710 to S730:

step S710, if the license plate type is a single-row license plate, extracting features of the initial license plate image to obtain a second license plate feature map;

step S720, inputting the second license plate feature map into a bidirectional long-short time memory network to perform license plate recognition, and obtaining license plate characters of a license plate;

and step S730, performing de-duplication processing on the license plate characters to obtain license plate information.

In step S710 of some embodiments, if the license plate type is a single-row license plate, license plate recognition is performed on the initial license plate image through a license plate recognition model, wherein the license plate recognition model includes a feature extraction network and a bidirectional long-short-time memory network. Specifically, the initial license plate image is input to a feature extraction network for feature extraction to obtain a second license plate feature map, wherein the feature extraction network can be ResNet18, mobileNet or VGG16, and the second license plate feature map is license plate feature information of a single-row license plate.

In step S720 of some embodiments, the bidirectional Long-Short-Term Memory network may be formed by a forward Long-Short-Term Memory network (LSTM) and a backward Long-Term Memory network, or may be formed by stacking multiple layers of bidirectional Long-Term Memory networks. It can be appreciated that the deep network structure has higher license plate recognition accuracy than the shallow network structure.

Specifically, a second vehicle feature map is input into a forward long-short time memory network to perform feature extraction to obtain a first feature map, the second vehicle feature map is input into a backward long-short time memory network to perform feature extraction to obtain a second feature map, feature combination is performed on the first feature map and the second feature map to obtain a third feature map, the third feature map is input into a current bidirectional long-short time memory network to perform feature extraction to obtain an output of the current bidirectional long-short time memory network, the output is used as an input of a next bidirectional long-short time memory network until the output of an L-layer bidirectional long-short time memory network is obtained, L is more than or equal to 2, normalization processing is performed on the output through a softmax function to obtain a character posterior probability matrix, wherein the character posterior probability matrix is used for representing the probability of outputting characters in different time steps, characters with the maximum probability of each time step in the character posterior probability matrix are selected to obtain a character sequence, and the character sequence is used as a license plate. In the training phase, the loss function of the bidirectional long-short-time memory network adopts a CTC loss function.

In step S730 of some embodiments, redundant characters exist in the character sequence, in order to avoid repeated recognition of the characters, in the embodiments of the present application, redundant data is removed by a blank mechanism, so as to obtain single-row license plate information, and the single-row license plate information is output. For example, five time steps t0, t1, t2, t3 and t4 exist, the character output by the time steps t0, t1 and t2 is a, the character output by the time steps t3 and t4 is b, the character sequence is aaabb, and the continuously repeated characters are combined to obtain license plate information ab.

Through the steps S710 to S730, accuracy of license plate character recognition can be improved, and repeated recognition of characters can be avoided.

In step S140 of some embodiments, since the character sizes and the number of characters of the two-row license plate characters are not consistent, the structure is complex, and in order to facilitate the processing of the two-row license plate characters respectively, when the license plate type is the two-row license plate, the initial license plate image is cut up and down to obtain the first intermediate license plate image and the second intermediate license plate image.

In step S150 of some embodiments, the first intermediate license plate image and the second intermediate license plate image are transversely stitched to obtain a target license plate image. The existing double-row license plate end-to-end recognition based on deep learning is complex in network structure, a large amount of double-row license plate data are needed, applicability is low, the double-row license plate is converted into a single-row license plate in a splicing mode, a license plate recognition model is not required to be trained by the large amount of double-row license plate data, the high-precision recognition of the double-row license plate can be achieved by training the license plate recognition model by only adopting the single-row license plate data, and applicability is improved.

In step S160 of some embodiments, the feature extraction manner of the target license plate image is the same as that of step S710, and will not be described here again.

In step S170 of some embodiments, the manner of license plate recognition based on the bidirectional long-short-time memory network is the same as that of step S720, and will not be described here again. And performing character deduplication on the character sequence output by the bidirectional long-short-time memory network through a blank mechanism to obtain license plate information of the double-row license plate, and outputting the license plate information.

Referring to fig. 8, an embodiment of the present application further provides a license plate recognition device, which may implement the license plate recognition method, where the device includes:

the acquiring module 810 is configured to acquire an initial license plate image of the target vehicle, where the initial license plate image includes a license plate of the target vehicle;

the calculating module 820 is configured to calculate an aspect ratio of a license plate in the initial license plate image, and obtain license plate aspect ratio data;

the license plate type determining module 830 is configured to determine a license plate type of a license plate according to the aspect ratio data of the license plate, where the license plate type includes a double-row license plate;

the cropping module 840 is configured to crop the initial license plate image if the license plate type is a double-row license plate, and obtain a first intermediate license plate image and a second intermediate license plate image;

The image stitching module 850 is configured to perform image stitching on the first intermediate license plate image and the second intermediate license plate image to obtain a target license plate image;

the feature extraction module 860 is configured to perform feature extraction on the target license plate image to obtain a first license plate feature map;

the license plate recognition module 870 is configured to input the first license plate feature map to a preset two-way long and short time memory network to perform license plate recognition, so as to obtain license plate information of the license plate.

The specific implementation of the license plate recognition device is basically the same as the specific embodiment of the license plate recognition method, and is not repeated here.

The embodiment of the application also provides electronic equipment, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the license plate recognition method when executing the computer program. The electronic equipment can be any intelligent terminal including a tablet personal computer, a vehicle-mounted computer and the like.

Referring to fig. 9, fig. 9 illustrates a hardware structure of an electronic device according to another embodiment, the electronic device includes:

the processor 910 may be implemented by a general-purpose CPU (central processing unit), a microprocessor, an application-specific integrated circuit (ApplicationSpecificIntegratedCircuit, ASIC), or one or more integrated circuits, etc. for executing related programs to implement the technical solutions provided by the embodiments of the present application;

Memory 920 may be implemented in the form of read-only memory (ReadOnlyMemory, ROM), static storage, dynamic storage, or random access memory (RandomAccessMemory, RAM). The memory 920 may store an operating system and other application programs, and when the technical solutions provided in the embodiments of the present application are implemented by software or firmware, relevant program codes are stored in the memory 920, and the processor 910 invokes the license plate recognition method to execute the embodiments of the present application;

an input/output interface 930 for inputting and outputting information;

the communication interface 940 is configured to implement communication interaction between the present device and other devices, and may implement communication in a wired manner (e.g., USB, network cable, etc.), or may implement communication in a wireless manner (e.g., mobile network, WI F I, bluetooth, etc.);

a bus 950 for transferring information between components of the device (e.g., processor 910, memory 920, input/output interface 930, and communication interface 940);

wherein processor 910, memory 920, input/output interface 930, and communication interface 940 implement communication connections among each other within the device via a bus 950.

The embodiment of the application also provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program realizes the license plate recognition method when being executed by a processor.

The memory, as a non-transitory computer readable storage medium, may be used to store non-transitory software programs as well as non-transitory computer executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory optionally includes memory remotely located relative to the processor, the remote memory being connectable to the processor through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.

According to the license plate recognition method, the license plate recognition device, the electronic equipment and the computer readable storage medium, the length-width ratio of the license plate in the initial license plate image is calculated through acquiring the initial license plate image of the target vehicle, the license plate length-width ratio data is obtained, the license plate type of the license plate is determined according to the license plate length-width ratio data, and the single-row license plate and the double-row license plate are distinguished through the license plate length-width ratio data so as to carry out different treatments on the single-row license plate and the double-row license plate. Further, if the license plate type is a double-row license plate, the initial license plate image is cut to obtain a first middle license plate image and a second middle license plate image, and the upper row and the lower row of the double-row license plate are split into two independent rows, so that each row can be conveniently and respectively processed. Furthermore, the first middle license plate image and the second middle license plate image are subjected to image stitching to obtain a target license plate image, and the double-row license plate can be converted into a single-row license plate. Finally, feature extraction is carried out on the target license plate image to obtain a first license plate feature map, effective license plate features can be extracted, the first license plate feature map is input into a preset two-way long-short-time memory network to carry out license plate recognition, license plate information of a license plate is obtained, the license plate recognition is carried out through the effective license plate features instead of the license plate image, the data processing capacity of the network is reduced, the license plate recognition is carried out through the two-way long-short-time memory network, and the accuracy of double-row license plate recognition is improved.

The embodiments described in the embodiments of the present application are for more clearly describing the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application, and as those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are equally applicable to similar technical problems.

It will be appreciated by those skilled in the art that the technical solutions shown in the figures do not constitute limitations of the embodiments of the present application, and may include more or fewer steps than shown, or may combine certain steps, or different steps.

The above described apparatus embodiments are merely illustrative, wherein the units illustrated as separate components may or may not be physically separate, i.e. may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Those of ordinary skill in the art will appreciate that all or some of the steps of the methods, systems, functional modules/units in the devices disclosed above may be implemented as software, firmware, hardware, and suitable combinations thereof.

The terms "first," "second," "third," "fourth," and the like in the description of the present application and in the above-described figures, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that embodiments of the present application described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

It should be understood that in this application, "at least one" means one or more, and "a plurality" means two or more. "and/or" for describing the association relationship of the association object, the representation may have three relationships, for example, "a and/or B" may represent: only a, only B and both a and B are present, wherein a, B may be singular or plural. The character "/" generally indicates that the context-dependent object is an "or" relationship. "at least one of" or the like means any combination of these items, including any combination of single item(s) or plural items(s). For example, at least one (one) of a, b or c may represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, c may be single or plural.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the above-described division of units is merely a logical function division, and there may be another division manner in actual implementation, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described above as separate components may or may not be physically separate, and components shown as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including multiple instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of the various embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing a program.

Preferred embodiments of the present application are described above with reference to the accompanying drawings, and thus do not limit the scope of the claims of the embodiments of the present application. Any modifications, equivalent substitutions and improvements made by those skilled in the art without departing from the scope and spirit of the embodiments of the present application shall fall within the scope of the claims of the embodiments of the present application.

Claims

1. The license plate recognition method is characterized by comprising the following steps of:

2. The license plate recognition method according to claim 1, wherein the license plate type further comprises a single row license plate, and the license plate recognition method further comprises, after determining the license plate type of the license plate from the license plate aspect ratio data:

3. The license plate recognition method according to claim 1, wherein the acquiring the initial license plate image of the target vehicle includes:

4. The license plate recognition method of claim 3, wherein the preprocessing the first license plate image to obtain the initial license plate image comprises:

5. The license plate recognition method of claim 4, wherein the performing gray scale transformation on the first license plate image to obtain a second license plate image comprises:

6. The license plate recognition method according to claim 4, wherein the performing edge feature extraction on the second license plate image to obtain a third license plate image includes:

7. The license plate recognition method according to any one of claims 1 to 6, wherein the license plate type further includes a single row license plate, and the determining the license plate type of the license plate from the license plate aspect ratio data includes:

8. License plate recognition device, characterized in that, the device includes:

9. An electronic device comprising a memory and a processor, the memory storing a computer program, the processor implementing the license plate recognition method of any one of claims 1 to 7 when the computer program is executed.

10. A computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the license plate recognition method of any one of claims 1 to 7.