CN111797829A

CN111797829A - License plate detection method and device, electronic equipment and storage medium

Info

Publication number: CN111797829A
Application number: CN202010587354.6A
Authority: CN
Inventors: 王嘉伟; 邵明; 王耀农
Original assignee: Zhejiang Dahua Technology Co Ltd
Current assignee: Zhejiang Dahua Technology Co Ltd
Priority date: 2020-06-24
Filing date: 2020-06-24
Publication date: 2020-10-20

Abstract

The invention discloses a license plate detection method and device, electronic equipment and a storage medium, which are used for improving the license plate detection precision. The method comprises the steps of extracting features of an image to be detected including a license plate region, determining detection information of the image to be detected according to the extracted image feature information, wherein the detection information comprises at least one detection frame, and a confidence coefficient and a prediction cross-over ratio corresponding to each detection frame; determining a target detection frame from at least one detection frame according to the intersection ratio of the confidence degree corresponding to each detection frame and the prediction; and taking the position of the target detection frame in the image to be detected as the position of the license plate area in the image to be detected. The confidence coefficient represents the confidence coefficient of the license plate contained in the detection frame, the prediction cross-over ratio represents the coincidence degree of the detection frame and the license plate region, the target detection frame selected according to the confidence coefficient of the detection frame and the prediction cross-over ratio is the detection frame with higher detection accuracy, the position of the target detection frame is used as the result of license plate detection, and the license plate detection accuracy is higher.

Description

License plate detection method and device, electronic equipment and storage medium

Technical Field

The invention relates to the technical field of image processing, in particular to a license plate detection method and device, electronic equipment and a storage medium.

Background

The intelligent traffic service system with video monitoring as a core provides important guarantee for city safety, with the increase of vehicle holding amount, the vehicle management by acquiring vehicle information through video monitoring becomes the development trend of vehicle management at present, and the license plate information is the main content of vehicle attribute information and is the main dependence on monitoring and tracking vehicles.

At present, relatively large noise may exist in a license plate detection environment, for example, an interfering rectangular object exists, and the environmental noise may interfere with the license plate detection, so that the precision of the license plate detection is relatively low.

Disclosure of Invention

The invention provides a license plate detection method and device, electronic equipment and a storage medium, which are used for improving the license plate detection precision.

In a first aspect, an embodiment of the present invention provides a license plate detection method, including:

extracting features of an image to be detected containing a license plate region, and determining detection information of the image to be detected according to the extracted image feature information, wherein the detection information comprises at least one detection frame, confidence coefficient of the license plate contained in the detection frame corresponding to each detection frame, and prediction intersection and comparison of coincidence degree of the detection frame and the license plate region;

determining a target detection frame from the at least one detection frame according to the intersection ratio of the confidence degree corresponding to each detection frame and the prediction;

and taking the position of the target detection frame in the image to be detected as the position of the license plate area in the image to be detected.

The license plate detection method provided by the embodiment of the invention comprises the steps of extracting the characteristics of an image to be detected containing a license plate area, determining the detection information of the image to be detected according to the extracted image characteristic information, wherein the detection information comprises at least one detection frame, the confidence coefficient corresponding to each detection frame and the prediction cross-over ratio corresponding to each detection frame, determining a target detection frame from at least one detection frame obtained through detection according to the confidence coefficient and the prediction cross-over ratio corresponding to each detection frame, and taking the position of the target detection frame as the position of the license plate in the image to be detected to finish the license plate detection. According to the license plate detection method provided by the embodiment of the invention, aiming at least one detection frame included in detection information of an image to be detected, a target detection frame is selected from the at least one detection frame according to the confidence coefficient and the prediction cross ratio of each detection frame, the prediction cross ratio represents the coincidence degree of the detection frame and a license plate region as the confidence coefficient of the detection frame including the license plate, the target detection frame selected according to the confidence coefficient and the prediction cross ratio of the detection frame is the detection frame with higher detection precision in the detection frame, the position of the target detection frame is taken as the position of the license plate region in the image to be detected, and the license plate detection precision is higher. In addition, a target detection frame is selected from at least one detection frame, and the detection frames which are detected by mistake and caused by noise interference in the detection information are removed, so that the precision of the license plate detection is improved.

In an optional implementation manner, the detection information further includes a probability value corresponding to each license plate type for each detection frame;

the method further comprises the following steps:

and determining the license plate type of the license plate in the image to be detected according to the probability value of the target detection frame corresponding to each license plate type.

According to the license plate detection method provided by the embodiment of the invention, the detection information obtained according to the image characteristic information of the image to be detected also comprises the license plate type, so that the detection result is enriched, the use of the license plate detection result is expanded, and for example, the region to which the vehicle belongs can be judged according to the license plate type.

An optional implementation manner is that, the performing feature extraction on the image to be detected including the license plate region, and determining the detection information of the image to be detected according to the extracted image feature information includes:

and performing feature extraction on the image to be detected through a trained full convolution neural network, and determining at least one detection frame corresponding to the image to be detected through the trained full convolution neural network according to the extracted image feature information, and the confidence coefficient, the prediction cross-over ratio and the probability value of each detection frame corresponding to each license plate type corresponding to each detection frame.

The license plate detection method provided by the embodiment of the invention can extract the features of the license plate through the trained full convolution neural network and determine the detection information according to the extracted image feature information, before the full convolution neural network is used, the full convolution neural network is trained based on a large amount of sample data, the training image containing the license plate area is taken as the input of the full convolution neural network, at least one detection frame corresponding to the training image, the confidence coefficient and the cross-over ratio corresponding to each detection frame and the probability value corresponding to each license plate type are taken as the output of the full convolution neural network, the full convolution neural network is trained for multiple times, after the full convolution neural network is converged, the full convolution neural network is determined to be trained, and the trained full convolution neural network has the capability of extracting the features of the image to be detected and acquiring the detection information, and the accuracy of license plate detection is improved based on the computing power of the full convolution neural network.

In an optional embodiment, the trained full convolutional neural network comprises at least one first convolutional layer, at least one pooling layer, and at least one second convolutional layer, wherein the first convolutional layer and the pooling layer are arranged in a crossed manner;

determining at least one detection frame corresponding to the image to be detected, the confidence coefficient and the prediction cross-over ratio corresponding to each detection frame and the probability value of each license plate type corresponding to each detection frame through the trained full convolution neural network, wherein the determination comprises the following steps:

performing convolution operation on the image to be detected according to the convolution kernel used for extracting the license plate features in the first convolution layer by adopting the first convolution layer to obtain a feature mapping matrix corresponding to the image to be detected;

adopting the pooling layer to perform down-sampling processing on the feature mapping matrix obtained by the last first convolution layer;

and performing feature fusion on the feature mapping matrix subjected to the downsampling processing by adopting the second convolution layer to obtain at least one detection frame corresponding to the image to be detected, the confidence coefficient and the prediction cross-over ratio corresponding to each detection frame and the probability value of each detection frame corresponding to each license plate type.

According to the license plate detection method provided by the embodiment of the invention, the license plate detection of the image to be detected is realized based on the full convolution neural network with the structure, and the down sampling is adopted in the whole network, so that the calculated amount is reduced.

An optional embodiment is that, the determining a target detection frame from the at least one detection frame according to the intersection ratio of the confidence degree corresponding to each detection frame to the prediction includes:

selecting at least one detection frame as a target detection frame; wherein, the selected detection frame comprises the detection frame with the largest screening parameter; if a plurality of detection frames are selected, the intersection ratio of any two detection frames in the plurality of detection frames is smaller than a first preset threshold value; the screening parameters are determined according to the confidence coefficient and the prediction cross-over ratio;

determining the license plate type of the license plate in the image to be detected according to the probability value of the target detection frame corresponding to each license plate type, wherein the determining comprises the following steps:

and taking the license plate type with the maximum probability value corresponding to the target detection frame as the type of the license plate.

According to the license plate detection method provided by the embodiment of the invention, the confidence coefficient represents the possibility of whether the detection frame contains the license plate, and the prediction cross ratio represents the positioning accuracy of the detection frame, so that the screening parameter of each detection frame is determined according to the confidence coefficient and the prediction cross ratio corresponding to each detection frame in at least one detection frame, the target detection frame is selected from at least one detection frame according to the screening parameter, and the license plate detection accuracy can be determined according to the comparison between the confidence coefficient and the prediction cross ratio, so that the detection frame with large screening parameter is used as the target detection frame, and the license plate detection accuracy is higher.

An alternative embodiment is that the full convolution neural network is trained according to the following:

taking a training image containing a license plate region, a pre-labeled position of an actual frame corresponding to the license plate region in the training image, and a pre-labeled license plate type as input of a full convolution neural network, taking at least one training detection frame corresponding to the training image, a training confidence coefficient, a training cross-over ratio and a training probability value corresponding to each license plate type as output of the full convolution neural network, and performing multi-round training on the full convolution neural network;

determining a positioning loss value, a cross-over ratio loss value, a category loss value and a confidence coefficient loss value according to the position of at least one training detection frame corresponding to the training image obtained in each round of training, the training confidence coefficient corresponding to each training detection frame, the training cross-over ratio and the training probability value corresponding to each license plate type, and the position of a pre-labeled actual frame and the pre-labeled license plate type; taking the sum of the positioning loss value, the intersection ratio loss value, the category loss value and the confidence coefficient loss value as a comprehensive loss value;

and adjusting parameters of the full convolution neural network according to the comprehensive loss value until the comprehensive loss value is determined to be within a preset range to obtain the trained full convolution neural network.

In the embodiment of the invention, in the process of training the full convolution neural network, the full convolution neural network is trained based on a large amount of sample data, after the full convolution neural network is converged, the training of the full convolution neural network is determined to be completed, and the trained full convolution neural network has the capability of extracting the characteristics of the image to be detected and acquiring the detection information. When the parameters of the full convolution neural network are adjusted, the training of the optimization network is assisted through positioning loss, intersection ratio loss, category loss and confidence coefficient loss, and the accuracy of the full convolution neural network in license plate detection is improved.

In an alternative embodiment, the positioning loss value is determined in the following manner:

determining a positioning loss value according to the position of the training detection frame with the maximum training confidence coefficient and the position of the pre-labeled actual frame;

determining the intersection ratio loss value according to the following modes:

determining the real intersection ratio of the training detection frame with the maximum training confidence coefficient and the pre-labeled actual frame; determining a cross-over ratio loss value according to the training cross-over ratio corresponding to the training detection box with the maximum training confidence coefficient and the real cross-over ratio;

determining a class loss value according to:

determining a category loss value according to a training probability value of the training detection frame with the maximum training confidence corresponding to each license plate type and the pre-labeled license plate type;

determining a confidence loss value according to:

determining a confidence loss value according to training confidence degrees corresponding to a plurality of training detection frames with training intersection ratios larger than a second preset threshold and real confidence degrees corresponding to the plurality of training detection frames, wherein the real confidence degrees corresponding to the training detection frames are determined according to the positions of the pre-labeled actual frames.

In the license plate detection method provided by the embodiment of the invention, the loss value is determined based on the information of the pre-labeled real frame and the information of the detection frame in the training process, the parameters of the full convolution neural network are adjusted according to the loss value, and the neural network is optimized, so that the full convolution neural network has the license plate detection capability.

In an optional embodiment, the training image including the license plate region includes: a training image containing a complete vehicle region, and a training image containing an incomplete vehicle region.

In the license plate detection method provided by the embodiment of the invention, the training samples comprise the training images including the complete vehicle area and the training images including the cut vehicle area, so that the distribution diversity of the training samples can be enriched, and the robustness of the full convolution neural network obtained by training on the license plate detection is improved. The method is carried out based on the training images containing the complete vehicle areas and the training images containing the incomplete vehicles, and the trained full convolution neural network can realize the license plate detection of any images to be detected containing the license plate areas, reduce the license plate detection errors caused by vehicle detection and improve the license plate detection precision.

In a second aspect, an embodiment of the present invention further provides a license plate detection apparatus, including:

the detection module is used for extracting the characteristics of an image to be detected containing a license plate region and determining the detection information of the image to be detected according to the extracted image characteristic information, wherein the detection information comprises at least one detection frame, confidence coefficients indicating that the detection frames corresponding to each detection frame contain the license plate and a prediction intersection and comparison indicating the coincidence degree of the detection frames and the license plate region;

the selecting module is used for determining a target detection frame from the at least one detection frame according to the intersection ratio of the confidence degree corresponding to each detection frame and the prediction;

and the determining module is used for taking the position of the target detection frame in the image to be detected as the position of the license plate area in the image to be detected.

the determination module is further to: and determining the license plate type of the license plate in the image to be detected according to the probability value of the target detection frame corresponding to each license plate type.

An optional implementation manner is that the detection module is specifically configured to:

the detection module is specifically configured to:

An optional implementation manner is that the selecting module is specifically configured to:

the determining module is specifically configured to: and taking the license plate type with the maximum probability value corresponding to the target detection frame as the type of the license plate.

In an optional embodiment, the apparatus further includes a training module, and the training module is configured to train the full convolution neural network according to the following manner:

In an optional implementation manner, the training module is specifically configured to determine the positioning loss value according to the following manner:

the training module is specifically configured to determine an intersection-to-intersection ratio loss value according to the following manner:

the training module is specifically configured to determine a category loss value according to the following:

the training module is specifically configured to determine a confidence loss value according to the following:

In a third aspect, another embodiment of the present invention further provides an electronic device, including at least one processor; and a memory communicatively coupled to the at least one processor; the storage stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any license plate detection method provided by the first aspect of the embodiment of the invention.

In a fourth aspect, another embodiment of the present invention further provides a computer storage medium, where the computer storage medium stores computer-executable instructions, and the computer-executable instructions are configured to enable a computer to execute any license plate detection method provided in the first aspect of the embodiments of the present invention.

Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments of the present invention will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.

Fig. 1 is a schematic diagram of a license plate detection result including an image of an incomplete vehicle according to an embodiment of the present invention;

fig. 2 is a schematic flow chart of a license plate detection method according to an embodiment of the present invention;

fig. 3 is a schematic diagram of a method for determining an intersection ratio of any two regions according to an embodiment of the present invention;

fig. 4 is a schematic structural diagram of a full convolution neural network according to an embodiment of the present invention;

FIG. 5 is a diagram illustrating a coordinate representation of a center of a rectangular frame in an image according to an embodiment of the present invention;

FIG. 6 is a normalized center coordinate representation of a rectangular frame in an image according to an embodiment of the present invention;

FIG. 7 is a complete flowchart of a license plate detection method according to an embodiment of the present invention;

FIG. 8 is a flowchart illustrating a method for training a full convolutional neural network according to an embodiment of the present invention;

fig. 9 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

fig. 10 is a schematic structural diagram of a license plate detection device according to an embodiment of the present invention;

fig. 11 is a schematic structural diagram of another license plate detection device according to an embodiment of the present invention.

Detailed Description

The technical solution in the embodiments of the present application will be described in detail and removed with reference to the accompanying drawings. In the description of the embodiments herein, "/" means "or" unless otherwise specified, for example, a/B may mean a or B; "and/or" in the text is only an association relationship describing an associated object, and means that three relationships may exist, for example, a and/or B may mean: three cases of a alone, a and B both, and B alone exist, and in addition, "a plurality" means two or more than two in the description of the embodiments of the present application.

In the following, the terms "first", "second" are used for descriptive purposes only and are not to be understood as implying or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of embodiments of the application, unless stated otherwise, "plurality" means two or more.

The term "and/or" in the embodiments of the present invention describes an association relationship of associated objects, and indicates that three relationships may exist, for example, a and/or B may indicate: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

When detecting a license plate, if a large noise exists in a detection environment, the precision of license plate detection may be low. For example, when an interfering rectangular object exists in an image or a video for license plate detection, the image or the video may be mistakenly detected as a license plate; or for example, in some cases, the license plate detection is obtained by performing secondary detection based on a vehicle detection result, for some images with a large snapshot angle, a cut vehicle may be obtained during vehicle detection, and the incomplete vehicle detection may also form noise, thereby causing a large license plate positioning error and a low license plate detection accuracy. As shown in fig. 1, a schematic diagram of a license plate detection result containing an image of an incomplete vehicle is shown, and since a vehicle is cut off during vehicle detection, the license plate detection result is a detection frame shown in fig. 1, and has a relatively large difference from the position of a real license plate, and the detection precision is relatively low. The embodiment of the invention provides a license plate detection method, which can obtain a license plate detection result with higher detection precision.

As shown in fig. 2, a flowchart of a license plate detection method provided in an embodiment of the present invention includes:

in step S201, feature extraction is performed on an image to be detected including a license plate region, and detection information of the image to be detected is determined according to the extracted image feature information;

the detection information comprises at least one detection frame, confidence degree of a license plate contained in the detection frame corresponding to each detection frame, and prediction intersection ratio of coincidence degree of the detection frame and the license plate region;

in step S202, determining a target detection frame from at least one detection frame according to a coincidence ratio of the confidence degree corresponding to each detection frame to the prediction;

in step S203, the position of the target detection frame in the image to be detected is taken as the position of the license plate region in the image to be detected.

In the license plate detection method provided by the embodiment of the invention, the feature extraction is carried out on the image to be detected containing the license plate region, the detection information of the image to be detected is determined according to the extracted image feature information, the detection information comprises at least one detection frame, the confidence corresponding to each detection frame and the prediction cross-over ratio corresponding to each detection frame, the target detection frame is determined from at least one detection frame obtained by detection according to the confidence corresponding to each detection frame and the prediction cross-over ratio, and the position of the target detection frame is used as the position of the license plate in the image to be detected, so that the license plate detection is completed. According to the license plate detection method provided by the embodiment of the invention, aiming at least one detection frame included in detection information of an image to be detected, a target detection frame is selected from the at least one detection frame according to the confidence coefficient and the prediction cross ratio of each detection frame, the prediction cross ratio represents the coincidence degree of the detection frame and a license plate region as the confidence coefficient of the detection frame including the license plate, the target detection frame selected according to the confidence coefficient and the prediction cross ratio of the detection frame is the detection frame with higher detection precision in the detection frame, the position of the target detection frame is taken as the position of the license plate region in the image to be detected, and the license plate detection precision is higher. In addition, a target detection frame is selected from at least one detection frame, and the detection frames which are detected by mistake and caused by noise interference in the detection information are removed, so that the precision of the license plate detection is improved.

It should be noted that the image to be detected including the license plate region in the embodiment of the present invention may be an image to be detected including an entire vehicle region, or may be an image to be detected including an incomplete vehicle region but including a license plate region. In addition, the license plate detection method provided by the embodiment of the invention can also realize license plate detection on any image to be detected containing a license plate area, and the image to be detected is not specifically limited by the embodiment of the invention.

Due to the fact that noise influencing the license plate detection precision exists, the determined detection information of the image to be detected comprises at least one detection frame, the target detection frame is determined from the at least one detection frame contained in the detection information, and the target detection frame is used as a detection result for license plate detection of the image to be detected.

The detection information of the image to be detected also comprises confidence corresponding to each detection frame, a prediction intersection ratio corresponding to each detection frame and a probability value corresponding to each license plate type of each detection frame.

The confidence corresponding to each detection frame represents the confidence that the detection frame contains the license plate, and the higher the confidence is, the higher the probability that the detection frame contains the license plate is, for example, the confidence corresponding to the detection frame 1 is 0.9, and the confidence corresponding to the detection frame 2 is 0.8, so that the probability that the detection frame 1 contains the license plate is higher than the probability that the detection frame 2 contains the license plate.

The prediction cross-over ratio corresponding to each detection frame represents the coincidence degree of the detection frame and the license plate region, the larger the prediction cross-over ratio is, the larger the coincidence degree of the detection frame and the license plate region is, for example, the prediction cross-over ratio corresponding to the detection frame 1 is 0.8, the prediction cross-over ratio corresponding to the detection frame 2 is 0.6, and the coincidence degree of the detection frame 1 and the license plate region is larger than the coincidence degree of the detection frame 2 and the license plate region.

The intersection-to-union ratio (IOU) represents the ratio of the area of the intersection of any two regions to the area of the union of the two regions, such as two rectangular regions shown in fig. 3, where a rectangular region 1 is composed of a region a and a region B, a rectangular region 2 is composed of a region B and a region C, the region B is the intersection of the rectangular region 1 and the rectangular region 2, and the region a, the region B and the region C are the union of the rectangular region 1 and the rectangular region 2, and then the intersection-to-union ratio of the rectangular region 1 to the rectangular region 2 can be determined according to formula (1):

wherein, the IOU_ABRepresents the intersection ratio between the rectangular area 1 and the rectangular area 2; s_A、S_B、S_CThe areas of the regions A, B and C are shown, respectively.

It can be seen that the intersection ratio IOU can represent the degree of coincidence of the two regions, and the larger the intersection ratio between the two regions is, the higher the degree of coincidence of the two regions is. In the embodiment of the invention, the prediction cross-over ratio corresponding to any detection frame is the cross-over ratio between the detection frame and the license plate region obtained by prediction according to the image characteristic information of the image to be detected, and the positioning accuracy of the license plate detection is represented.

In addition, the confidence coefficient represents the possibility of whether the detection frame contains the license plate, and the detection accuracy of the license plate can be determined by the combination of the detection frame and the detection frame through the prediction of intersection accuracy and the indication of the positioning accuracy of the detection frame.

The probability value of each detection frame corresponding to each license plate type is the possibility of representing the license plate type of the license plate contained in the image to be detected, which is determined according to the image characteristic information of the image to be detected. The types of license plates are various, such as green plates (plates for new energy vehicles), yellow plates (plates for large-sized vehicles, trailers, and the like), blue plates (plates for small-sized vehicles), and the like. It should be noted that the above license plate types are only examples, and do not limit the scope of the embodiments of the present invention.

The probability value of any detection frame corresponding to each license plate type determined according to the image characteristic information of the image to be detected is assumed as follows: the detection frame comprises a green plate 0.9, a yellow plate 0.1 and a blue plate 0, if the license plate is detected by the detection frame, the probability that the license plate type of the license plate contained in the image to be detected is the green plate is 0.9, the probability that the license plate is the yellow plate is 0.1, and the probability that the license plate is the blue plate is 0. The license plate type of the license plate contained in the image to be detected can be determined according to the probability value of at least one detection frame corresponding to each license plate type.

The embodiment of the invention extracts the characteristics of the image to be detected containing the license plate region, determines the detection information of the image to be detected according to the extracted image characteristic information, and can be realized by a deep learning technology.

An optional implementation manner is that features of an image to be detected are extracted through a trained full convolution neural network, and at least one detection box corresponding to the image to be detected, a confidence coefficient corresponding to each detection box, a prediction cross-over ratio and a probability value corresponding to each type of license plate of each detection box are determined according to extracted image feature information through the trained full convolution neural network.

Specifically, an image to be detected is input into a trained full convolution neural network, the trained full convolution neural network performs feature extraction on the image to be detected, at least one detection frame corresponding to the image to be detected is determined according to extracted image feature information, and a confidence coefficient, a prediction cross-over ratio and a probability value of each detection frame corresponding to each license plate type are determined.

It should be noted that before the full convolution neural network is called, the full convolution neural network needs to be trained based on a large amount of sample data, a training image including a license plate region is used as an input of the full convolution neural network, at least one detection box corresponding to the training image, a confidence coefficient corresponding to each detection box, a cross-over ratio and a probability value corresponding to each license plate type are used as an output of the full convolution neural network, the full convolution neural network is trained for multiple times, and after the full convolution neural network is converged, it is determined that the full convolution neural network is trained completely.

In implementation, before the image to be detected including the license plate region is input into the trained full convolution neural network, the image to be detected can be preprocessed, so that the image to be detected meets the input requirement of the trained full convolution neural network. The preprocessing can include cropping the image to be detected, randomly flipping the image, converting the color space, scaling, etc. In order to prevent the deformation of the image to be detected in the process of processing the image to be detected by the full convolution neural network and further cause errors on license plate detection, the image to be detected containing the license plate area with the same width and height can be obtained through preprocessing, the influence of noise is reduced, and the precision of license plate detection is improved.

The embodiment of the invention provides a full convolution neural network, which comprises at least one first convolution layer, at least one pooling layer and at least one second convolution layer, wherein the first convolution layer and the pooling layer are arranged in a crossed mode.

When the detection information of the image to be detected is determined, a first convolution layer is adopted, and convolution operation is carried out on the image to be detected according to convolution check used for extracting license plate features in the first convolution layer to obtain a feature mapping matrix corresponding to the image to be detected; adopting the pooling layer to perform down-sampling processing on the feature mapping matrix obtained by the last first convolution layer; and performing feature fusion on the feature mapping matrix subjected to the downsampling processing by adopting a second convolution layer to obtain at least one detection frame corresponding to the image to be detected, the confidence coefficient and the prediction cross-over ratio corresponding to each detection frame and the probability value of each detection frame corresponding to each license plate type.

Specifically, the preprocessed image to be detected is input into a full convolution neural network, and the full convolution neural network performs feature extraction on the image to be detected to obtain detection information of the image to be detected.

Aiming at the first convolution layers, each first convolution layer comprises a plurality of convolution kernels, the convolution kernels are matrixes used for extracting pixel characteristics in an image to be detected, the image to be detected input into the full convolution neural network is an image matrix formed by pixel values, and the pixel values can be gray values, RGB values and the like of pixels in the image to be detected; performing convolution operation on the image to be detected by a plurality of convolution kernels in the first convolution layer, wherein the convolution operation refers to performing matrix convolution operation on an image matrix and a convolution kernel matrix; the image matrix is subjected to convolution operation of a convolution kernel to obtain a feature mapping matrix, a plurality of convolution kernels perform convolution operation on an image to be detected to obtain a plurality of feature mapping matrices corresponding to the image to be detected, each convolution kernel can extract specific features, and different convolution kernels extract different features. In the embodiment of the present invention, the convolution kernel included in the first convolution layer may be a convolution kernel for extracting features of texture features, color features, contour features, and the like of the image to be detected.

For the pooling layer, the feature mapping matrix obtained by the last first convolution layer may be down-sampled. The pooling layer is used for reducing the size of the model, improving the calculation speed and improving the robustness of the extracted features. The pooling layer in embodiments of the present invention may be a maximum pooling layer. When the feature mapping matrix is subjected to down-sampling processing through the maximum pooling layer, the feature mapping matrix can be divided into a plurality of regions, the maximum value in each region is taken, the parameter values in the feature mapping matrix are reduced, and down-sampling is realized.

And aiming at the second convolution layer, performing feature fusion on the feature mapping matrix subjected to the downsampling processing to obtain detection information of the image to be detected, wherein the detection information comprises at least one detection frame, confidence corresponding to each detection frame, a prediction intersection ratio and a probability value corresponding to each license plate type of each detection frame.

As shown in fig. 4, a structure of a full convolution neural network provided in an embodiment of the present invention includes 3 first convolution layers, 3 pooling layers, and 3 second convolution layers, in order to reduce a calculation amount, the first convolution layer in the embodiment of the present invention may perform feature extraction and may also perform down-sampling processing, a step size of the first convolution layer may be set to 2, and the first convolution layer and the 3 pooling layers achieve overall down-sampling of the full convolution neural network to 16.

After the detection information of the image to be detected is determined according to the license plate detection method provided by the embodiment of the invention, a target detection frame is determined from at least one detection frame of the detection result according to the intersection ratio of the confidence degree corresponding to each detection frame in the detection information and the prediction.

In an optional implementation manner, at least one detection frame is selected as a target detection frame; wherein, the selected detection frame comprises the detection frame with the largest screening parameter; if a plurality of detection frames are selected, the intersection ratio of any two detection frames in the plurality of detection frames is smaller than a first preset threshold value; screening parameters were determined based on confidence and predictive cross-over ratios.

Specifically, a screening parameter of each detection frame is determined according to a confidence coefficient and a prediction cross-over ratio corresponding to each detection frame in at least one detection frame, a target detection frame is selected from the at least one detection frame according to the screening parameter, and the license plate detection accuracy can be determined according to the confidence coefficient and the prediction cross-over ratio, so that the detection frame with large screening parameters is used as the target detection frame, and the license plate detection accuracy is higher. If one target detection frame is determined in at least one detection frame, taking the maximum screening parameter in the at least one detection frame as the target detection frame; if a plurality of target detection frames are determined in at least one detection frame, the intersection ratio of any two detection frames in the plurality of detection frames is smaller than a first preset threshold value, and repeated detection frames possibly corresponding to the same object can be removed according to the intersection ratio between any two detection frames in the plurality of detection frames.

In an implementation, the product of the confidence corresponding to the detection frame and the prediction cross may be used as the screening parameter of the detection frame, for example, if the confidence corresponding to any detection frame is 0.9, and the prediction cross ratio is 0.7, the screening parameter corresponding to the detection frame is 0.63.

It should be noted that, if the intersection ratio between any two detection frames in the multiple detection frames is not less than the first preset threshold, the detection frames with smaller screening parameters in the two detection frames are rejected, and the detection frames with larger screening parameters are reserved. The first preset threshold may be an empirical value of one skilled in the art, such as 0.9, and the embodiment of the present invention is not limited in particular.

The embodiment of the invention provides a specific method for determining a target detection frame, which comprises the steps of firstly determining a screening parameter corresponding to each detection frame according to a confidence coefficient corresponding to each detection frame and a prediction cross-over ratio, supposing that detection information comprises five detection frames, namely a detection frame 1, a detection frame 2, a detection frame 3, a detection frame 4 and a detection frame 5, wherein the screening parameter of the detection frame 1 is 0.7, the screening parameter of the detection frame 2 is 0.9, the screening parameter of the detection frame 3 is 0.5, the screening parameter of the detection frame 4 is 0.8 and the screening parameter of the detection frame 5 is 0.6.

Firstly, selecting the detection frame with the largest sequencing parameter from the unselected detection frames, and rejecting the detection frames which are not smaller than a first preset threshold value and intersect with the selected detection frame from the rest detection frames.

In the first round of filtering processing, the detection frame with the largest screening parameter in the unselected detection frames is the detection frame 2, the intersection-parallel ratio between the rest detection frames 1, 3, 4, 5 and 2 is determined respectively, and if the intersection-parallel ratio between the detection frame 1 and 2 is 0.98, the intersection-parallel ratio between the detection frame 3 and 2 is 0.91, the intersection-parallel ratio between the detection frame 4 and 2 is 0.5, the intersection-parallel ratio between the detection frame 5 and 2 is 0.4, and the first preset threshold value is 0.9, the detection frame 1 and 3 are rejected, and the detection frame 2, 4 and 5 are retained.

Then, judging whether the unselected detection frames exist, if so, returning to the step of selecting the detection frame with the largest screening parameter of the unselected detection frames; and if not, taking the rest detection frames as target detection frames.

In the remaining detection frames 2, 4 and 5, the detection frame 4 and the detection frame 5 are not selected, the detection frame with the largest screening parameter in the unselected detection frames is the detection frame 4, the intersection and parallel ratio between the remaining detection frames 2, 5 and 4 is respectively determined, and if the intersection and parallel ratio between the detection frame 2 and the detection frame 4 is 0.5, the intersection and parallel ratio between the detection frame 5 and the detection frame 4 is 0.91 and the first preset threshold value is 0.9, the detection frame 5 is removed and the detection frames 2 and 4 are retained.

And when judging whether the unselected detection frames exist, selecting the detection frames 2 and 4, and taking the detection frames 2 and 4 as target detection frames if the unselected detection frames do not exist.

In addition, when the intersection ratio between the non-selected detection frame with the largest screening parameter and the other detection frames is judged for screening, the intersection ratio between the non-selected detection frame and the selected detection frame may not be judged, for example, when the intersection ratios between the remaining detection frames and the detection frame 4 are determined, the intersection ratio between the detection frame 2 and the detection frame 4 may not be judged, when the detection frame 2 is selected for filtering, the intersection ratio between the detection frame 2 and the detection frame 4 has already been judged, therefore, if the detection frame 4 is not rejected, the intersection ratio between the detection frame 4 and the detection frame 2 is proved to be smaller than the first preset threshold, and in order to simplify the calculation amount, the intersection ratio between the detection frame 2 and the detection frame 4 may not be judged. In implementation, at least one detection frame may be sorted according to the screening parameters from large to small, in each filtering, the detection frame with the largest screening parameter is selected from the unselected detection frames, a predetermined intersection ratio between the other detection frames arranged after the selected detection frame and the selected detection frame is determined, if the intersection ratio between the other detection frames arranged after the selected detection frame and the selected detection frame is not less than a first predetermined threshold, the other detection frames arranged after the selected detection frame are removed until the filtering is completed, and the remaining detection frames are used as target detection frames.

The embodiment of the invention also provides a method for determining the target detection frame, wherein at least one detection frame contained in the detection information is combined pairwise, and if the number of the at least one detection frame is N, the number of the at least one detection frame is N

And combining the group detection frames, calculating the intersection ratio between the two detection frames in each group, if the intersection ratio between any two detection frames is not less than a first preset threshold value, rejecting the detection frames with smaller screening parameters, performing the operation on each group of detection frames, and taking the rest detection frames as target detection frames.

In addition, if only one detection frame is output by the full convolution neural network and corresponds to the image to be detected, the detection frame can be used as a target detection frame.

It should be noted that the method for determining the target detection frame provided in the foregoing embodiments is only an example, and does not limit the scope of the embodiments of the present invention. In fact, any screening method that can obtain the target detection frame in the embodiments of the present invention is within the scope of the embodiments of the present invention.

And after determining the target detection frame, taking the position of the target detection frame in the image to be detected as the position of the license plate area in the image to be detected, and taking the license plate type with the maximum probability value corresponding to the target detection frame as the license plate type of the license plate in the image to be detected.

In the embodiment of the present invention, the position of the target detection frame may be represented by a central coordinate, for example, the coordinate of the target detection frame 1 is (c)_x，c_yW, h) in which (c)_x，c_y) The method comprises the following steps of representing the central coordinate of a target detection frame, w representing the width of the target detection frame, h representing the height of the target detection frame, representing by adopting relative coordinates in order to reduce inaccurate positioning caused by pixel coordinate change in the image scaling process, normalizing the size of an image to be detected, representing the target detection frame by using the normalized coordinates, and introducing the normalized coordinate representation:

the image shown in fig. 5 has a rectangular frame, the size of the image is 970 wide and 546 high, the rectangular frame is surrounded by four points (640, 356) (870, 356) (640, 520) (870, 520), the coordinates of the center point of the rectangular frame are (755, 438), the width is 230, the height is 164, and the rectangular frame is represented by the way of the center coordinates as (755, 438, 230, 164). Since the image scaling causes the number of pixels of the image to change and the positioning is not accurate due to the representation of absolute coordinates, the size of the image is normalized, and as shown in fig. 6, the size of the image is normalized, the coordinates (755, 438) of the center point of the rectangular box are normalized and represented as (0.78, 0.8), the width is normalized and represented as 0.24, and the height is normalized and represented as 0.30, and the rectangular box is represented as (0.78, 0.8, 0.24, 0.30) by the coordinates of the center point.

In the embodiment of the invention, the probability value of each license plate type corresponding to the target detection frame is assumed as follows: and if the green card is 0.9, the yellow card is 0.1 and the blue card is 0, determining that the type of the license plate detected by the target detection frame is the green card.

It should be noted that, in the license plate detection method provided in the embodiment of the present invention, since at least one detection frame may be determined by using a full convolution neural network, and then a target detection frame is determined therefrom as a result of license plate detection, the license plate detection accuracy is high, the number and the position of the target detection frame may be regarded as the number and the position of a license plate in an image to be detected, and if one target detection frame is detected, it indicates that the image to be detected may only include one license plate; if the number of the target detection frames is multiple, the fact that the image to be detected possibly comprises a plurality of license plates is shown.

As shown in fig. 7, a complete flowchart of a license plate detection method provided in the embodiment of the present invention includes:

in step S701, inputting an image to be detected including a license plate region into a trained full convolution neural network;

in step S702, at least one detection frame corresponding to an image to be detected output by the trained full convolution neural network, a confidence corresponding to each detection frame, a prediction cross-over ratio, and a probability value corresponding to each license plate type for each detection frame are obtained;

in step S703, determining a screening parameter of each detection frame according to a confidence corresponding to each detection frame in the at least one detection frame and a prediction merging ratio;

in step S704, selecting the detection frame with the largest sequencing parameter from the unselected detection frames, and removing the detection frames that are not smaller than the first preset threshold value when compared with the selected detection frame from the remaining detection frames;

in step S705, it is determined whether there is an unselected detection frame, and if yes, the process returns to step S704; if not, executing step S706;

in step S706, the remaining detection frames are set as target detection frames;

in step S707, the position of the target detection frame in the image to be detected is used as the position of the license plate region in the image to be detected, and the license plate type with the maximum probability value corresponding to the target detection frame is used as the type of the license plate in the image to be detected.

The embodiment of the invention also provides a method for training the full convolution neural network, and after the training of the full convolution neural network is finished, the license plate detection method shown in the figures 2 and 7 can be executed through the trained full convolution neural network.

One optional implementation way is that a training image containing a license plate region, a position of an actual frame corresponding to the license plate region in a pre-labeled training image, and a pre-labeled license plate type are used as the input of a full convolution neural network, at least one training detection frame corresponding to the training image, a training confidence coefficient corresponding to each training detection frame, a training cross-over ratio, and a training probability value corresponding to each license plate type are used as the output of the full convolution neural network, and the full convolution neural network is subjected to multiple rounds of training; determining a positioning loss value, a cross-over ratio loss value, a category loss value and a confidence coefficient loss value according to the position of at least one training detection frame corresponding to a training image obtained in each round of training, the training confidence coefficient corresponding to each training detection frame, a training cross-over ratio and the training probability value corresponding to each license plate type, and the position of a pre-labeled actual frame and the pre-labeled license plate type; taking the sum of the positioning loss value, the intersection ratio loss value, the category loss value and the confidence coefficient loss value as a comprehensive loss value; and adjusting parameters of the full convolution neural network according to the comprehensive loss value until the comprehensive loss value is determined to be within a preset range to obtain the trained full convolution neural network.

Specifically, a full convolution neural network is trained through a large number of training images, a training image containing a license plate region, the position of an actual frame corresponding to the license plate region in a pre-labeled training image and a pre-labeled license plate type are input into the full convolution neural network in the training process, the full convolution neural network extracts the characteristics of the training image, at least one training detection frame corresponding to the training image and the training confidence coefficient, the training cross-over ratio and the training probability value corresponding to each license plate type are determined according to the extracted image characteristic information and output, a comprehensive loss value is determined through detection information and labeling information output by the full convolution neural network, model parameters of the full convolution neural network are adjusted until the comprehensive loss value is within a preset range, and the convergence training of the neural network is determined to be completed, the trained full convolution neural network has the capability of determining at least one detection frame in an image to be detected, the confidence degree corresponding to each detection frame, the prediction cross-over ratio and the probability value corresponding to each license plate type, and after the full convolution neural network is trained, the license plate detection method provided by the embodiment of the invention can be realized according to the trained full convolution neural network, so that the license plate detection precision is improved.

Before inputting the training image into the full convolution neural network, the training image needs to be determined, and the training image in the embodiment of the invention can be a training image containing a complete vehicle area and a training image containing an incomplete vehicle area.

Because the license plate detection is usually performed by secondary analysis based on the result of the vehicle detection, and the vehicle detection error can cause that the output vehicle is a cut vehicle, thereby causing the error of license plate positioning and even causing the phenomenon of missing detection of the license plate, the full convolution neural network provided by the embodiment of the invention can enrich the diversity of the distribution of training samples during training, and can realize the license plate detection of any image to be detected containing the license plate area based on the training image containing the complete vehicle area and the training image containing the incomplete vehicle, thereby reducing the license plate detection error caused by the vehicle detection and improving the precision of the license plate detection.

According to the embodiment of the invention, the collected training images can be preprocessed, and the training images are cut randomly, so that the diversity of sample distribution is enriched. An optional implementation manner is that, for an input image, the image is randomly cropped along the horizontal direction at a preset probability, the training image size is judged after cropping, if the training image is different in width and height, the training image is further cropped along the vertical direction until the height is equal to the width after cropping, the deformation influence caused by the zoom of the training image in the full convolution neural network is reduced, and the image is randomly flipped, color space converted, scaled and the like, so that the training image meets the input requirement of the full convolution neural network.

In the process of clipping the training image, one tenth of the training image can be randomly selected from the training image for clipping, and in the implementation, the image can be randomly clipped along the horizontal direction, and the clipping proportion is randomly sampled between 0 and 1/3 of the width of the original image. In order to ensure that the integrity of the license plate cannot be changed by cutting, the position of the actual frame corresponding to the marked license plate area is read before cutting. The image may be cropped from the left side of the training image and from the right side if the license plate is cropped. If the license plate is cut as it is, then the random cutting in the horizontal direction is not performed, and the training image after cutting contains the license plate area.

It should be noted that, in the embodiment of the present invention, the manner of preprocessing the training image is merely an example, and does not limit the scope of the present invention.

Inputting the position of an actual frame of a license plate region in a preprocessed training image and a pre-labeled vehicle type into a full convolution neural network, extracting the characteristics of the training image by the full convolution neural network, outputting at least one training detection frame corresponding to the training image, a training confidence coefficient, a training cross-over ratio and a training probability value corresponding to each license plate type, and determining a comprehensive loss value according to detection information output by the full convolution neural network and labeled detection information. Each of the following ways of determining the loss value is described:

and when the positioning loss value is determined, determining the positioning loss value according to the position of the training detection frame with the maximum training confidence coefficient and the position of the pre-labeled actual frame.

Specifically, for any training image, the full convolution neural network may output at least one training detection box, and determine a positioning loss value according to a position of a training detection box with a maximum training confidence coefficient in the at least one detection box and a position of an actual box in a pre-labeled training image.

In practice, the positioning loss value may be determined according to equation (2):

b represents the total number of training detection frames with the maximum training confidence coefficient obtained according to the training images in one training process; i represents a training detection box with the maximum training confidence corresponding to any training image; (x)_i，y_i，w_i，h_i) Representing the position of the training detection box with the maximum training confidence corresponding to any training image;

and indicating the position of the actual frame corresponding to any one of the marked training images.

In the embodiment of the invention, in order to ensure that the algorithm keeps robustness on objects with different scales, evolution operation is carried out on the width and height prediction results in loss so as to reduce the problem of loss imbalance caused by scale factors.

When determining the cross-over ratio loss value, determining the real cross-over ratio of the training detection frame with the maximum training confidence coefficient and the pre-labeled actual frame; and determining a coincidence ratio loss value according to the training coincidence ratio and the real coincidence ratio corresponding to the training detection box with the maximum training confidence coefficient.

Specifically, for any training image, the full convolution neural network may output at least one training detection frame, determine the position of the training detection frame and the position of the pre-labeled actual frame to determine a true intersection ratio, and determine an intersection ratio loss value according to the training intersection ratio and the true intersection ratio of the training detection frame output by the full convolution neural network.

In practice, the intersection ratio loss value may be determined according to equation (3):

b represents the total number of training detection frames with the maximum training confidence coefficient obtained according to the training images in one training process; i represents a training detection box with the maximum training confidence corresponding to any training image; IoU_iRepresenting the training cross-over ratio of the training detection box with the maximum training confidence corresponding to any training image;

and representing the real intersection ratio of the training detection frame with the maximum training confidence corresponding to any training image, wherein the real intersection ratio is determined according to the position of the training detection frame and the position of a pre-labeled actual frame.

And when the category loss value is determined, determining the category loss value according to the training probability value of each license plate type corresponding to the training detection frame with the maximum training confidence coefficient and the pre-labeled license plate type.

Specifically, for any training image, the full convolution neural network may output at least one training detection frame, and the category loss value is determined according to a training probability value corresponding to each license plate type of the training detection frame with the maximum training confidence coefficient in the at least one detection frame and a pre-labeled license plate type.

In implementation, the class loss value may be determined according to equation (4):

b represents the total number of training detection frames with the maximum training confidence coefficient obtained according to the training images in one training process; i represents a training detection box with the maximum training confidence corresponding to any training image; p_i(c) Representing the training probability value of each license plate type corresponding to the training detection box with the maximum training confidence corresponding to any training image;

representing the license plate type of any training image label.

For example, the training probability values corresponding to each license plate type for the training detection frame with the maximum training confidence corresponding to any training image are green plate 0.5, yellow plate 0.3 and blue plate 0.2, and if the type of the license plate in the pre-labeled training image is green plate, the category loss value for the training detection frame with the maximum training confidence corresponding to the training image is:

and when the confidence coefficient loss value is determined, determining the confidence coefficient loss value according to the training confidence coefficients corresponding to the training detection frames with the training intersection ratio being greater than a second preset threshold value and the real confidence coefficients corresponding to the training detection frames. And determining the real confidence corresponding to the training detection frame according to the position of the pre-labeled actual frame.

Specifically, for any training image, the full convolution neural network may output at least one training detection box, and determine a confidence loss value according to a training confidence of a plurality of training detection boxes in the at least one detection box, which is greater than a second preset threshold in training intersection, and a true confidence corresponding to the plurality of training detection boxes.

The confidence degree of any training detection frame corresponding to the training prediction frame is 1 or 0, the full convolution neural network performs down-sampling processing on the training image in the process of performing feature extraction on the image, for example, the full convolution neural network with down-sampling of 16 in the embodiment of the present invention processes the training image of 304 × 304 into a feature mapping matrix of 19 × 19, the center point of the actual frame in the original training image falls at a corresponding position in the feature mapping matrix of 19 × 19, it is assumed that the center point of the actual frame falls at a pixel point a of the feature mapping matrix of 19 × 19, the last convolution layer of the full convolution neural network performs license plate detection pixel by pixel point, the confidence degree corresponding to the training prediction frame detected by the pixel point a is 1, and the confidence degrees corresponding to the training prediction frames detected by the pixel points at other positions are 0.

In implementation, the confidence loss value may be determined according to equation (5):

the S represents the total number of training detection frames with the training intersection ratio larger than a second preset threshold value, which are obtained according to the training images in the one-round training process; j represents any training detection box with the training cross-over ratio larger than a second preset threshold value; c_jRepresenting a training confidence of any training detection box;

representing the true confidence of any of the training detection boxes.

In each round of training, the sum of the positioning loss value D, the cross-comparison loss value L, the category loss value P and the confidence coefficient loss value C is used as a comprehensive loss value, parameters of the full convolution neural network are adjusted according to the comprehensive loss value, after the parameters of the full convolution neural network are adjusted, a training image is input into the full convolution neural network to perform next round of training until the determined comprehensive loss value is within a preset range, the full convolution neural network is determined to be trained, the full convolution neural network is trained through a large number of training images, the full convolution neural network learns the license plate detection capability, and the trained full convolution neural network is used for license plate detection to obtain license plate detection information with higher detection precision.

As shown in fig. 8, an overall flowchart of a full convolutional neural network training method provided in an embodiment of the present invention includes:

in step S801, a training image including a plurality of license plate regions is collected;

in step S802, preprocessing the acquired training image;

in step S803, inputting the training image, the position of the actual frame corresponding to the license plate region in the pre-labeled training image, and the pre-labeled license plate type into the full convolution neural network, and obtaining the detection information of the training image output by the full convolution neural network;

in step S804, a comprehensive loss value is determined according to a position of at least one training detection frame corresponding to a training image in the detection information of the training image, a training confidence corresponding to each training detection frame, a training intersection ratio, and a training probability value corresponding to each license plate type, and a position of a pre-labeled actual frame and a pre-labeled license plate type;

in step S805, parameters of the full convolution neural network are adjusted according to the synthetic loss value, and the neural network is optimized.

Based on the same inventive concept, the embodiment of the invention also provides an electronic device, and as the principle of solving the problem of the electronic device is the same as the license plate detection method of the embodiment of the invention, the implementation of the electronic device can refer to the implementation of the method, and repeated details are not repeated.

Referring to fig. 9, an electronic device 900 according to an embodiment of the present invention includes:

at least one processor 901, and

a memory 902 communicatively coupled to the at least one processor;

the memory 902 stores instructions executable by the at least one processor 901, and the at least one processor 901 implements the license plate detection method according to the embodiment of the invention by executing the instructions stored in the memory.

Based on the same inventive concept, as shown in fig. 10, an embodiment of the present invention provides a license plate detection device, including:

the detection module 1001 is used for extracting features of an image to be detected containing a license plate region, and determining detection information of the image to be detected according to the extracted image feature information, wherein the detection information comprises at least one detection frame, confidence coefficients indicating that the detection frames corresponding to the detection frames contain license plates, and a prediction intersection and comparison indicating the coincidence degree of the detection frames and the license plate region;

a selecting module 1002, configured to determine a target detection frame from the at least one detection frame according to a coincidence ratio between a confidence corresponding to each detection frame and a prediction;

a determining module 1003, configured to use a position of the target detection frame in the image to be detected as a position of the license plate area in the image to be detected.

the determining module 1003 is further configured to: and determining the license plate type of the license plate in the image to be detected according to the probability value of the target detection frame corresponding to each license plate type.

In an optional implementation manner, the detection module 1001 is specifically configured to:

In an optional embodiment, the trained full convolutional neural network comprises at least one first convolutional layer, at least one pooling layer, and at least one second convolutional layer, wherein the first convolutional layer is arranged to intersect with the pooling layer;

the detection module 1001 is specifically configured to:

An optional implementation manner is that the selecting module 1002 is specifically configured to:

the determining module 1003 is specifically configured to: and taking the license plate type with the maximum probability value corresponding to the target detection frame as the type of the license plate.

As shown in fig. 11, an embodiment of the present invention provides a license plate detecting apparatus, further including a training module 1004, where the training module 1004 is configured to train a full convolution neural network according to the following manner:

In an optional embodiment, the training module 1004 is specifically configured to determine the positioning loss value according to the following manner:

the training module 1004 is specifically configured to determine the cross-over ratio loss value according to the following manner:

the training module 1004 is specifically configured to determine the class loss value according to the following:

the training module 1004 is specifically configured to determine the confidence loss value according to the following:

An embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the method for detecting a license plate is implemented.

As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims

1. A license plate detection method is characterized by comprising the following steps:

2. The method of claim 1, wherein the detection information further comprises a probability value for each detection box for each license plate type;

the method further comprises the following steps:

3. The method of claim 2, wherein the extracting the features of the image to be detected including the license plate region, and determining the detection information of the image to be detected according to the extracted image feature information comprises:

4. The method of claim 3, wherein the trained full convolutional neural network comprises at least one first convolutional layer, at least one pooling layer, and at least one second convolutional layer, wherein the first convolutional layer is interleaved with the pooling layer;

5. The method of claim 1, wherein determining a target detection box from the at least one detection box according to the confidence level and the prediction intersection ratio corresponding to each detection box comprises:

6. The method of any of claims 2 to 5, wherein the full convolutional neural network is trained according to the following:

7. The method of claim 6, wherein the location loss value is determined according to:

determining the intersection ratio loss value according to the following modes:

determining a class loss value according to:

determining a confidence loss value according to:

8. The method of claim 7, wherein the training image containing the license plate region comprises: a training image containing a complete vehicle region, and a training image containing an incomplete vehicle region.

9. A license plate detection device, comprising:

10. An electronic device, comprising:

at least one processor, and

a memory communicatively coupled to the at least one processor;

the storage stores instructions executable by the at least one processor, and the at least one processor implements the license plate detection method according to any one of claims 1-8 by executing the instructions stored in the storage.

11. A computer storage medium storing a computer program, wherein the computer program is configured to perform the license plate detection method according to any one of claims 1 to 8 when executed by a computer.