CN111079744B - Intelligent vehicle license plate identification method and device suitable for complex illumination environment - Google Patents

Intelligent vehicle license plate identification method and device suitable for complex illumination environment Download PDF

Info

Publication number
CN111079744B
CN111079744B CN201911243936.6A CN201911243936A CN111079744B CN 111079744 B CN111079744 B CN 111079744B CN 201911243936 A CN201911243936 A CN 201911243936A CN 111079744 B CN111079744 B CN 111079744B
Authority
CN
China
Prior art keywords
image
license plate
detected
sample
information
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201911243936.6A
Other languages
Chinese (zh)
Other versions
CN111079744A (en
Inventor
都荣胜
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Ludong University
Original Assignee
Ludong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Ludong University filed Critical Ludong University
Priority to CN201911243936.6A priority Critical patent/CN111079744B/en
Publication of CN111079744A publication Critical patent/CN111079744A/en
Application granted granted Critical
Publication of CN111079744B publication Critical patent/CN111079744B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
    • G06V20/63—Scene text, e.g. street names
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
    • G06V10/469—Contour-based spatial representations, e.g. vector-coding
    • G06V10/473—Contour-based spatial representations, e.g. vector-coding using gradient analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
    • G06V20/625—License plates

Abstract

The embodiment of the invention discloses a vehicle license plate intelligent identification method and a device suitable for a complex illumination environment, wherein the method comprises the following steps: obtaining an image to be detected; performing edge detection on the image to be detected by using a preset edge detection algorithm, and determining a gradient image to be detected corresponding to the image to be detected; the method comprises the steps of determining whether an image to be detected contains a license plate or not by utilizing the image to be detected, a gradient image to be detected and a pre-trained license plate detection model, and determining the position information of the license plate contained in the image to be detected under the condition that the image to be detected contains the license plate, wherein the pre-trained license plate detection model is as follows: the model obtained is trained on the basis of the sample original image marked with the position of the sample vehicle and the sample edge image corresponding to the sample original image, so that the detection rate of the license plate detection in a complex illumination environment, such as an environment scene with large illumination change, is improved.

Description

Intelligent vehicle license plate identification method and device suitable for complex illumination environment
Technical Field
The invention relates to the technical field of image detection, in particular to an intelligent vehicle license plate recognition method and device suitable for a complex illumination environment.
Background
The license plate detection technology is one of key technologies for vehicle management in the fields of security, smart cities and the like, and has important significance in academic research and products in the industrial field.
At present, a license plate detection technology generally depends on a deep learning technology to detect a license plate in an image, namely, a license plate detection network model based on a deep convolutional neural network algorithm is obtained by training a large number of sample images containing the license plate and calibration information containing position information of the position of the license plate in the corresponding sample images, wherein the license plate detection network model learns the image characteristics of an image of a region where the license plate is located in the sample images; and then, the license plate detection network model is utilized to carry out license plate detection on the image to be detected, the image characteristics of the image of the area where the license plate is located in the sample image are learned by the license plate detection network model, and the license plate and the position information of the license plate in the image to be detected are determined.
However, the image features learned by the license plate detection network model based on the deep convolutional neural network algorithm are generally texture features of the image, and the texture features of the license plate in the image are changed to some extent due to the change of illumination and the shadow, which causes the condition that the license plate is easy to miss detection in a complex illumination environment, for example, an environment scene with large illumination change.
Disclosure of Invention
The invention provides an intelligent vehicle license plate recognition method and device suitable for a complex illumination environment, and aims to improve the detection rate of license plate detection in the complex illumination environment, such as an environment scene with large illumination change. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present invention provides an intelligent vehicle license plate recognition method suitable for a complex illumination environment, where the method includes:
obtaining an image to be detected;
performing edge detection on the image to be detected by using a preset edge detection algorithm, and determining a gradient image to be detected corresponding to the image to be detected;
determining whether the image to be detected contains a license plate or not by using the image to be detected, the gradient image to be detected and a pre-trained license plate detection model, and determining the position information of the license plate contained in the image to be detected under the condition that the image to be detected contains the license plate, wherein the pre-trained license plate detection model is as follows: and training the obtained model based on the sample original image marked with the position of the sample vehicle and the sample edge image corresponding to the sample original image.
Optionally, the step of performing edge detection on the image to be detected by using a preset edge detection algorithm to determine a gradient image to be detected corresponding to the image to be detected includes:
utilizing a first filter operator to carry out edge detection on the image to be detected in the horizontal direction, and determining a first gradient image corresponding to the image to be detected;
utilizing a second filter operator to carry out edge detection on the image to be detected in the vertical direction, and determining a second gradient image corresponding to the image to be detected;
and determining the gradient image to be detected corresponding to the image to be detected by utilizing the first gradient image and the second gradient image.
Optionally, before the step of determining whether the image to be detected contains a license plate by using the image to be detected, the gradient image to be detected and a pre-trained license plate detection model, and determining the position information of the license plate contained in the image to be detected under the condition that the image to be detected contains the license plate, the method further includes:
a process of training to obtain the pre-trained license plate detection model, wherein the process comprises:
obtaining a plurality of sample original images and calibration information thereof, wherein each sample original image contains a license plate, and each calibration information comprises: calibrating position information, calibrating type information and calibrating confidence information of the position of the sample license plate in the corresponding sample original image;
performing edge detection on each sample original image by using the preset edge detection algorithm, and determining a sample gradient image corresponding to each sample original image;
and training an initial license plate detection model by using each sample original image, the sample gradient image corresponding to each sample original image and calibration position information, calibration category information and calibration confidence information which are included in calibration information corresponding to each sample original image to obtain the pre-trained license plate detection model.
Optionally, the initial license plate detection model includes: a feature extraction layer and a feature regression and recognition layer;
the method for obtaining the pre-trained license plate detection model by training an initial license plate detection model by using the sample original image, the sample gradient image corresponding to the sample original image and the calibration position information included in the calibration information corresponding to the sample original image comprises the following steps:
inputting each sample image into the feature extraction layer to obtain a sample image feature corresponding to each sample image, wherein each sample image comprises: a sample original image and a sample gradient image corresponding to the sample original image;
inputting the sample image characteristics corresponding to each sample image into the characteristic regression and identification layer to obtain the detection information of the position of the license plate of the sample in each sample image, wherein the detection information comprises: detecting position information, detection category information and detection confidence information;
determining a target loss value corresponding to each sample image by using a preset loss function, the detection position information, the detection category information and the detection confidence information of the position of the sample license plate in each sample image, and the calibration position information, the calibration category information and the calibration confidence information included in the calibration information corresponding to each sample image, wherein the calibration information corresponding to each sample image is as follows: the calibration information corresponding to the original sample image included in the sample image;
and adjusting network parameters of an initial license plate detection model by using a target loss value corresponding to each sample image and a preset loss threshold value until the initial license plate detection model converges to obtain a pre-trained license plate detection model comprising the feature extraction layer and the feature regression and recognition layer.
Optionally, the detection information of the position of the license plate in each sample image includes: the sample image comprises first detection position information, first detection category information and first detection confidence information of the position of the license plate in the sample original image, and second detection position information, second detection category information and second detection confidence information of the position of the license plate in the sample gradient image corresponding to the sample original image;
the step of determining the target loss value corresponding to each sample image by using the preset loss function, the detection position information, the detection category information and the detection confidence information of the position of the sample license plate in each sample image, and the calibration position information, the calibration category information and the calibration confidence information included in the calibration information corresponding to each sample image comprises:
for each sample image, determining a first loss value corresponding to the sample original image included in the sample image by using first detection position information, first detection category information and first detection confidence information of the position of the sample license plate in the sample original image included in the sample image, and calibration position information, calibration category information and calibration confidence information included in calibration information corresponding to the sample image;
determining a second loss value corresponding to the sample gradient image corresponding to the sample original image by using second detection position information, second detection category information and second detection confidence information of the position of the sample license plate in the sample gradient image corresponding to the sample original image included in the sample image, and calibration position information, calibration category information and calibration confidence information included in the calibration information corresponding to the sample image;
and determining a target loss value corresponding to the sample image by using a first loss value corresponding to the sample original image included in the sample image and a second loss value corresponding to the sample gradient image corresponding to the sample original image.
Optionally, the pre-trained license plate detection model corresponds to at least one set of anchor point size information, each set of anchor point size information includes at least one pair of width and height information, wherein at least one pair of width and height information included by each set of anchor point size information is: clustering the license plate size information of the sample license plate in the sample original image of the pre-trained license plate detection model according to the training;
the step of determining whether the image to be detected contains the license plate or not by using the image to be detected, the gradient image to be detected and the pre-trained license plate detection model, and determining the position information of the license plate contained in the image to be detected under the condition that the image to be detected contains the license plate comprises the steps of:
inputting the image to be detected and the gradient image to be detected into a feature extraction layer of a pre-trained license plate detection model to obtain a first feature map of a first size corresponding to the image to be detected and a second feature map of the first size corresponding to the gradient image to be detected;
determining whether the image to be detected contains the license plate or not by utilizing the first feature map, the second feature map, a feature regression and recognition layer of a pre-trained license plate detection model and each pair of width and height information in at least one pair of width and height information included by the at least one group of anchor point size information;
and if the image to be detected contains the license plate, determining the position information of the license plate contained in the image to be detected.
Optionally, the method further includes:
if the image to be detected does not contain the license plate by utilizing the first feature diagram, the second feature diagram, the feature regression and recognition layer of the pre-trained license plate detection model and each pair of width and height information in at least one pair of width and height information included by the at least one group of anchor point size information, obtaining the at least one pair of width and height information included by the first anchor point size information, and obtaining the first width and height information obtained after scaling based on a first scaling coefficient corresponding to the first anchor point size information, wherein the first anchor point size information is: a largest group of anchor size information or a smallest group of anchor size information in the at least one group of anchor size information;
obtaining a third feature map of a second size corresponding to the image to be detected and a fourth feature map of the second size corresponding to the gradient image to be detected, wherein the third feature map and the fourth feature map are both: inputting the image to be detected and the gradient image to be detected into a feature extraction layer of a pre-trained license plate detection model to obtain a feature map;
determining whether the image to be detected contains the license plate or not by utilizing the third feature map, the fourth feature map, a feature regression and recognition layer of a pre-trained license plate detection model and the first width and height information;
and if the image to be detected contains the license plate, determining the position information of the license plate contained in the image to be detected.
Optionally, the method further includes:
if the third feature map, the fourth feature map, a feature regression and recognition layer of a pre-trained license plate detection model and the first width and height information are utilized to determine that the image to be detected does not contain a license plate, at least one pair of width and height information included in the second anchor point size information is obtained, and the second width and height information is obtained after scaling is performed on the basis of a second scaling coefficient corresponding to the second anchor point size information, wherein the second anchor point size information is as follows: a minimum set of anchor size information or a maximum set of anchor size information of the at least one set of anchor size information;
obtaining a fifth feature map of a third size corresponding to the image to be detected and a sixth feature map of the third size corresponding to the gradient image to be detected, wherein the fifth feature map and the sixth feature map are both: inputting the image to be detected and the gradient image to be detected into a feature extraction layer of a pre-trained license plate detection model to obtain a feature map;
determining whether the image to be detected contains the license plate or not by utilizing the fifth feature map, the sixth feature map, a feature regression and recognition layer of a pre-trained license plate detection model and the second width and height information;
and if the image to be detected contains the license plate, determining the position information of the license plate contained in the image to be detected.
In a second aspect, an embodiment of the present invention provides an intelligent vehicle license plate recognition apparatus suitable for a complex lighting environment, where the apparatus includes:
a first obtaining module configured to obtain an image to be detected;
the first determining module is configured to perform edge detection on the image to be detected by using a preset edge detection algorithm, and determine a gradient image to be detected corresponding to the image to be detected;
the second determining module is configured to determine whether the image to be detected contains a license plate or not by using the image to be detected, the gradient image to be detected and a pre-trained license plate detection model, and determine position information of the license plate contained in the image to be detected under the condition that the image to be detected contains the license plate, wherein the pre-trained license plate detection model is as follows: and training the obtained model based on the sample original image marked with the position of the sample vehicle and the sample edge image corresponding to the sample original image.
Optionally, the first determining module is specifically configured to perform edge detection on the image to be detected in the horizontal direction by using a first filter operator, and determine a first gradient image corresponding to the image to be detected;
utilizing a second filter operator to carry out edge detection on the image to be detected in the vertical direction, and determining a second gradient image corresponding to the image to be detected;
and determining the gradient image to be detected corresponding to the image to be detected by utilizing the first gradient image and the second gradient image.
Optionally, the apparatus further comprises:
the training module is configured to determine whether the image to be detected contains a license plate or not by using the image to be detected, the gradient image to be detected and a pre-trained license plate detection model, and train to obtain the pre-trained license plate detection model before determining the position information of the license plate contained in the image to be detected under the condition that the image to be detected contains the license plate, wherein the training module comprises:
the system comprises an obtaining unit and a calibration unit, wherein the obtaining unit is configured to obtain a plurality of sample original images and calibration information thereof, each sample original image contains a license plate, and each calibration information comprises: calibrating position information, calibrating type information and calibrating confidence information of the position of the sample license plate in the corresponding sample original image;
the determining unit is configured to perform edge detection on each sample original image by using the preset edge detection algorithm, and determine a sample gradient image corresponding to each sample original image;
and the training unit is configured to train an initial license plate detection model by using the calibration position information, the calibration category information and the calibration confidence information included in the calibration information corresponding to each sample original image, the sample gradient image corresponding to each sample original image and the calibration information corresponding to each sample original image, so as to obtain the pre-trained license plate detection model.
Optionally, the initial license plate detection model includes: a feature extraction layer and a feature regression and recognition layer;
the training unit includes:
a first input sub-module, configured to input each sample image into the feature extraction layer, to obtain a sample image feature corresponding to each sample image, where each sample image includes: a sample original image and a sample gradient image corresponding to the sample original image;
the second input submodule is configured to input the sample image features corresponding to each sample image into the feature regression and recognition layer, so as to obtain detection information of the position of the license plate of the sample in each sample image, wherein the detection information includes: detecting position information, detection category information and detection confidence information;
the determining submodule is configured to determine a target loss value corresponding to each sample image by using a preset loss function, the detection position information, the detection category information and the detection confidence information of the position of the license plate of the sample in each sample image, and the calibration position information, the calibration category information and the calibration confidence information included in the calibration information corresponding to each sample image, wherein the calibration information corresponding to each sample image is: the calibration information corresponding to the original sample image included in the sample image;
and the adjusting submodule is configured to adjust network parameters of an initial license plate detection model by using a target loss value corresponding to each sample image and a preset loss threshold value until the initial license plate detection model converges to obtain the pre-trained license plate detection model comprising the feature extraction layer and the feature regression and recognition layer.
Optionally, the detection information of the position of the license plate in each sample image includes: the sample image comprises first detection position information, first detection category information and first detection confidence information of the position of the license plate in the sample original image, and second detection position information, second detection category information and second detection confidence information of the position of the license plate in the sample gradient image corresponding to the sample original image;
the determining sub-module is specifically configured to determine, for each sample image, a first loss value corresponding to the sample original image included in the sample image by using first detection position information, first detection category information, and first detection confidence information of a position where a sample license plate is located in the sample original image included in the sample image, and calibration position information, calibration category information, and calibration confidence information included in calibration information corresponding to the sample image;
determining a second loss value corresponding to the sample gradient image corresponding to the sample original image by using second detection position information, second detection category information and second detection confidence information of the position of the sample license plate in the sample gradient image corresponding to the sample original image included in the sample image, and calibration position information, calibration category information and calibration confidence information included in the calibration information corresponding to the sample image;
and determining a target loss value corresponding to the sample image by using a first loss value corresponding to the sample original image included in the sample image and a second loss value corresponding to the sample gradient image corresponding to the sample original image.
Optionally, the pre-trained license plate detection model corresponds to at least one set of anchor point size information, each set of anchor point size information includes at least one pair of width and height information, wherein at least one pair of width and height information included by each set of anchor point size information is: clustering the license plate size information of the sample license plate in the sample original image of the pre-trained license plate detection model according to the training;
the second determining module is specifically configured to input the image to be detected and the gradient image to be detected into a feature extraction layer of a pre-trained license plate detection model, so as to obtain a first feature map of a first size corresponding to the image to be detected and a second feature map of the first size corresponding to the gradient image to be detected;
determining whether the image to be detected contains the license plate or not by utilizing the first feature map, the second feature map, a feature regression and recognition layer of a pre-trained license plate detection model and each pair of width and height information in at least one pair of width and height information included by the at least one group of anchor point size information;
and if the image to be detected contains the license plate, determining the position information of the license plate contained in the image to be detected.
Optionally, the second determining module is specifically configured to determine that the image to be detected does not include the license plate if the first feature map, the second feature map, the feature regression and recognition layer of the pre-trained license plate detection model and each pair of width and height information in at least one pair of width and height information included in the at least one group of anchor point size information are utilized, obtain at least one pair of width and height information included in the first anchor point size information, and obtain the first width and height information obtained after scaling is performed based on a first scaling coefficient corresponding to the first anchor point size information, where the first anchor point size information is: a largest group of anchor size information or a smallest group of anchor size information in the at least one group of anchor size information;
obtaining a third feature map of a second size corresponding to the image to be detected and a fourth feature map of the second size corresponding to the gradient image to be detected, wherein the third feature map and the fourth feature map are both: inputting the image to be detected and the gradient image to be detected into a feature extraction layer of a pre-trained license plate detection model to obtain a feature map;
determining whether the image to be detected contains the license plate or not by utilizing the third feature map, the fourth feature map, a feature regression and recognition layer of a pre-trained license plate detection model and the first width and height information;
and if the image to be detected contains the license plate, determining the position information of the license plate contained in the image to be detected.
Optionally, the second determining module is specifically configured to determine that the image to be detected does not include the license plate if the third feature map, the fourth feature map, a feature regression and recognition layer of a pre-trained license plate detection model and the first width and height information are used, obtain at least one pair of width and height information included in the second anchor point size information, and obtain the second width and height information obtained after scaling based on a second scaling coefficient corresponding to the second anchor point size information, where the second anchor point size information is: a minimum set of anchor size information or a maximum set of anchor size information of the at least one set of anchor size information;
obtaining a fifth feature map of a third size corresponding to the image to be detected and a sixth feature map of the third size corresponding to the gradient image to be detected, wherein the fifth feature map and the sixth feature map are both: inputting the image to be detected and the gradient image to be detected into a feature extraction layer of a pre-trained license plate detection model to obtain a feature map;
determining whether the image to be detected contains the license plate or not by utilizing the fifth feature map, the sixth feature map, a feature regression and recognition layer of a pre-trained license plate detection model and the second width and height information;
and if the image to be detected contains the license plate, determining the position information of the license plate contained in the image to be detected.
According to the content, the intelligent vehicle license plate recognition method and device provided by the embodiment of the invention are suitable for complex illumination environments, and can obtain the image to be detected; performing edge detection on the image to be detected by using a preset edge detection algorithm, and determining a gradient image to be detected corresponding to the image to be detected; the method comprises the steps of determining whether an image to be detected contains a license plate or not by utilizing the image to be detected, a gradient image to be detected and a pre-trained license plate detection model, and determining the position information of the license plate contained in the image to be detected under the condition that the image to be detected contains the license plate, wherein the pre-trained license plate detection model is as follows: and training the obtained model based on the sample original image marked with the position of the sample vehicle and the sample edge image corresponding to the sample original image.
By applying the embodiment of the invention, the detection of the image to be detected and the gradient image to be detected thereof can be realized based on the sample original image marked with the position of the sample vehicle and the sample edge image corresponding to the sample original image which is less influenced by the change of the illumination condition, and the pre-trained license plate detection model is trained, so that the position information of the license plate contained in the image to be detected is determined under the condition that the image to be detected contains the license plate. The method comprises the steps of training to obtain edge characteristics based on images through a sample original image and a sample edge image corresponding to the sample original image, detecting a pre-trained license plate detection model of the position where a license plate is located, wherein the sample edge image is less influenced by the change of illumination conditions, and even if the sample edge image is located in a complex illumination environment, namely a scene with large illumination change, the license plate contained in the sample edge image and the position where the license plate is located can be detected through the pre-trained license plate detection model, so that the detection rate of license plate detection in the complex illumination environment, such as the environment scene with large illumination change, is improved. Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.
The innovation points of the embodiment of the invention comprise:
1. the detection of the image to be detected and the gradient image to be detected can be realized based on training the obtained pre-trained license plate detection model through the sample original image marked with the position of the sample vehicle and the sample edge image corresponding to the sample original image which is less affected by the change of the illumination condition, and the position information of the license plate contained in the image to be detected is determined under the condition that the image to be detected contains the license plate. The method comprises the steps of training to obtain edge characteristics based on images through a sample original image and a sample edge image corresponding to the sample original image, detecting a pre-trained license plate detection model of the position where a license plate is located, wherein the sample edge image is less influenced by the change of illumination conditions, and even if the sample edge image is located in a complex illumination environment, namely a scene with large illumination change, the license plate contained in the sample edge image and the position where the license plate is located can be detected through the pre-trained license plate detection model, so that the detection rate of license plate detection in the complex illumination environment, such as the environment scene with large illumination change, is improved.
2. Training by utilizing the sample original image and the sample edge image corresponding to the sample original image to obtain a pre-trained license plate detection model, determining a target loss value corresponding to the sample original image by utilizing loss values corresponding to the sample original image and the sample edge image corresponding to the sample original image, and adjusting parameters of the license plate detection model based on the target loss value corresponding to the sample original image to enable the pre-trained license plate detection model to have the following characteristics: the capability of detecting the license plate from the sample edge image with small influence of illumination condition change provides a basis for subsequent license plate detection in a complex illumination environment, such as an environment scene with large illumination change, and improves the detection rate of license plate detection in the complex illumination environment, such as a scene with large illumination change.
3. The size information of the fixed anchor point and the size information of the self-adaptive anchor point are used for jointly detecting the license plate and the position information thereof contained in the image to be detected, the size of a sample vehicle contained in the original sample image is weakened to a certain extent, the limitation on the size of the license plate which can be detected by a pre-trained license plate detection model is avoided, the detection rate of the license plate in the actual detection process is improved when the size of the license plate is different from the size of the license plate in the original sample image in the actual detection process, and the condition that the license plate is missed to be detected when the size of the license plate is different from the size of the license plate in the original sample image in the actual.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below. It is to be understood that the drawings in the following description are merely exemplary of some embodiments of the invention. For a person skilled in the art, without inventive effort, further figures can be obtained from these figures.
Fig. 1 is a schematic flow chart of a method for intelligently identifying a license plate of a vehicle in a complex illumination environment according to an embodiment of the present invention;
fig. 2 is another schematic flow chart of a vehicle license plate intelligent recognition method suitable for a complex illumination environment according to an embodiment of the present invention;
fig. 3 is a schematic data interaction diagram for jointly detecting license plates and position information thereof included in an image to be detected through fixed anchor point size information and adaptive anchor point size information according to an embodiment of the present invention;
fig. 4 is a schematic flow chart of the intelligent vehicle license plate recognition device suitable for complex lighting environments according to the embodiment of the present invention.
Detailed Description
The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. It is to be understood that the described embodiments are merely a few embodiments of the invention, and not all embodiments. All other embodiments, which can be obtained by a person skilled in the art without inventive effort based on the embodiments of the present invention, are within the scope of the present invention.
It is to be noted that the terms "comprises" and "comprising" and any variations thereof in the embodiments and drawings of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
The invention provides an intelligent vehicle license plate recognition method and device suitable for a complex illumination environment, and aims to improve the detection rate of license plate detection in the complex illumination environment, such as an environment scene with large illumination change. The following provides a detailed description of embodiments of the invention.
Fig. 1 is a schematic flow chart of a vehicle license plate intelligent recognition method suitable for a complex illumination environment according to an embodiment of the present invention. The method may comprise the steps of:
s101: and obtaining an image to be detected.
In the embodiment of the present invention, the method may be applied to any type of electronic device with computing capability, and the electronic device may be a server or a terminal device. The electronic device may be provided in a vehicle, or may be provided in a non-vehicle device without being provided in the vehicle.
In one case, the electronic device may be an image capture device, or a device connected to an image capture device; under the condition that the electronic equipment is image acquisition equipment, the electronic equipment can monitor the current road corresponding to the electronic equipment in real time, acquire an image when a vehicle passes through the current road, and perform a subsequent intelligent vehicle license plate identification process suitable for a complex illumination environment as the image to be detected; under the condition that the electronic equipment is connected with the image acquisition equipment, the image acquisition equipment connected with the electronic equipment can monitor the current road corresponding to the image acquisition equipment in real time, acquire images when a vehicle passes through the current road and send the acquired images to the electronic equipment, and the electronic equipment acquires the images acquired by the connected image acquisition equipment, and the images serve as images to be detected and are used as an intelligent vehicle license plate identification process applicable to a complex illumination environment after execution. The image acquisition equipment can be a camera, a camera and the like. In one case, the image to be detected may be a color image, such as an RGB (Red Green Blue) image.
S102: and carrying out edge detection on the image to be detected by using a preset edge detection algorithm, and determining the gradient image to be detected corresponding to the image to be detected.
In this step, after the electronic device obtains the image to be detected, edge detection may be performed on the image to be detected by using a preset edge detection algorithm, so as to obtain a gradient image to be detected corresponding to the image to be detected. The preset edge detection algorithm may be a Sobel edge detection algorithm. The Sobel edge detection algorithm has a large inhibition effect on noise and a smooth area in an image, when the Sobel edge detection algorithm is applied to the image containing the license plate, color and texture features in the image are greatly compressed, edge features are highlighted, and the edge features are slightly influenced by illumination.
In an implementation manner of the present invention, the step S102 may include the following steps 011-:
011: and performing edge detection on the image to be detected in the horizontal direction by using a first filter operator, and determining a first gradient image corresponding to the image to be detected. Wherein, the horizontal direction can refer to the horizontal axis direction of the image to be detected.
012: and performing edge detection on the image to be detected in the vertical direction by using a second filter operator, and determining a second gradient image corresponding to the image to be detected. Wherein the vertical direction may refer to a longitudinal axis direction of the image to be detected.
013: and determining the gradient image to be detected corresponding to the image to be detected by utilizing the first gradient image and the second gradient image.
The first filter operator and the second filter operator may be operators set by a worker according to experience, and in one case, the first filter operator may be an operator set by a worker according to experience
Figure BDA0002306999920000101
The second filter operator may be
Figure BDA0002306999920000102
The above 011 can be expressed by the following formula (1): g1＝S1*I (1)。
Wherein G is1Representing a first gradient image, representing a convolution operation, I representing an image to be detected, S1Representing a first filter operator.
Accordingly, the above 012 can be expressed by the following formula (2): g2＝S2*I (2)。
Wherein G is2Representing the second gradient image, representing the convolution operation, I representing the image to be detected, S2Representing a second filter operator.
The above 013 can be expressed by the following formula (3):
Figure BDA0002306999920000111
wherein G represents the gradient image to be detected corresponding to the image to be detected.
S103: determining whether the image to be detected contains the license plate or not by using the image to be detected, the gradient image to be detected and a pre-trained license plate detection model, and determining the position information of the license plate contained in the image to be detected under the condition that the image to be detected contains the license plate.
The pre-trained license plate detection model comprises the following steps: and training the obtained model based on the sample original image marked with the position of the sample vehicle and the sample edge image corresponding to the sample original image. For the sake of clarity of layout, the specific training process of the pre-trained license plate detection model is described later.
In the step, the image to be detected and the gradient image to be detected can form a multi-channel matrix; inputting a pre-trained license plate detection model; performing image feature extraction on an image to be detected and a gradient image to be detected through a feature extraction layer in a pre-trained license plate detection model to obtain image features corresponding to the image to be detected and image features corresponding to the gradient image to be detected; and then, through a feature regression and recognition layer in a pre-trained license plate detection model, performing regression on image features corresponding to the image to be detected and image features corresponding to the gradient image to be detected, determining whether the image to be detected contains a license plate, and determining the position information of the license plate contained in the image to be detected under the condition that the image to be detected contains the license plate. For the sake of clarity of layout, the specific training process of the pre-trained license plate detection model is described later.
Wherein, it is to determine whether to contain the license plate in waiting to detect the image: and determining based on the image characteristics corresponding to the image to be detected and the image characteristics corresponding to the gradient image to be detected. If the image to be detected contains the license plate through the image characteristics corresponding to the image to be detected and/or the image to be detected contains the license plate through the image characteristics corresponding to the gradient image to be detected, the license plate in the image to be detected can be determined.
If the image to be detected is an RGB image, the image to be detected and the gradient image to be detected can form a 4-channel matrix.
By applying the embodiment of the invention, the detection of the image to be detected and the gradient image to be detected thereof can be realized based on the sample original image marked with the position of the sample vehicle and the sample edge image corresponding to the sample original image which is less influenced by the change of the illumination condition, and the pre-trained license plate detection model is trained, so that the position information of the license plate contained in the image to be detected is determined under the condition that the image to be detected contains the license plate. The method comprises the steps of training to obtain edge characteristics based on images through a sample original image and a sample edge image corresponding to the sample original image, detecting a pre-trained license plate detection model of the position where a license plate is located, wherein the sample edge image is less influenced by the change of illumination conditions, and even if the sample edge image is located in a complex illumination environment, namely a scene with large illumination change, the license plate contained in the sample edge image and the position where the license plate is located can be detected through the pre-trained license plate detection model, so that the detection rate of license plate detection in the complex illumination environment, such as the environment scene with large illumination change, is improved.
In another embodiment of the present invention, the electronic device may be pre-trained to obtain the pre-trained license plate detection model. Before the S103, the method may further include:
a process of training a pre-trained license plate detection model, where as shown in fig. 2, the process may include:
s201: and acquiring original images of a plurality of samples and calibration information thereof.
Wherein, every sample original image contains the license plate, and every calibration information includes: and calibrating position information, calibrating type information and calibrating confidence information of the position of the sample license plate in the corresponding sample original image. The calibration position information, the calibration category information and the calibration confidence information have a corresponding relation, the calibration category information comprises information indicating whether the corresponding calibration position information comprises a license plate, and the calibration confidence information is that the corresponding calibration position information comprises the confidence of the license plate.
S202: and performing edge detection on each sample original image by using a preset edge detection algorithm, and determining a sample gradient image corresponding to each sample original image.
S203: and training an initial license plate detection model by using the sample original image, the sample gradient image corresponding to the sample original image and calibration position information, calibration type information and calibration confidence information which are included in calibration information corresponding to the sample original image to obtain a pre-trained license plate detection model.
The sample original image may be a color image, such as an RGB image, the sample original image may include one or more license plates, and the license plates included in the sample original image may have various sizes. For clarity of description, the license plate included in the sample original image may be referred to as a sample license plate.
Each calibration information includes: and calibrating position information of the position of the sample license plate in the corresponding sample original image, wherein the calibrating position information can comprise position information of the central point of the sample license plate and width and height information of the sample license plate.
After the electronic equipment obtains a plurality of sample original images and calibration information thereof, edge detection can be performed on each sample original image by using a preset edge detection algorithm to determine a sample gradient image corresponding to each sample original image; the process of determining the sample gradient image corresponding to each sample original image is similar to the process of determining the edge image to be detected corresponding to the image to be detected, and is not repeated herein.
The electronic equipment forms each sample original image and a sample gradient image corresponding to the sample original image into a multi-channel matrix, inputs an initial license plate detection model, and inputs calibration position information, calibration category information and calibration confidence information included in calibration information corresponding to each sample original image into the initial license plate detection model to train the initial license plate detection model until the initial license plate detection model converges to obtain a pre-trained license plate detection model.
In one case, after the electronic device obtains the sample original image and the calibration information thereof, based on a preset clustering algorithm and size information of each sample license plate included in the sample original image, clustering is performed to obtain at least one group of vehicle size information as at least one group of anchor point size information, where each group of anchor point size information includes at least one pair of width and height information, and each pair of width and height information is: and obtaining the size information of each sample license plate included in the original sample image according to the fungus. The number of anchor size information groups and the number of width and height information included in each group may be set in advance. For example: the number of anchor point size information groups is set to be 3 in advance, the number of width and height information included in each group is 3, at this time, 3 groups of anchor point size information can be obtained through clustering based on a preset clustering algorithm and size information of each sample license plate included in a sample original image, and each anchor point size information comprises at least 3 pairs of width and height information. The width and height information included in the 3 groups of anchor point size information may be different levels of width and height information, for example: the first set of anchor size information may include 3 pairs of width and height information having a smaller size, the second set of anchor size information may include 3 pairs of width and height information having a medium size, and the third set of anchor size information may include 3 pairs of width and height information having a larger size.
The initial license plate detection model may be a network model based on a deep learning technique, for example, a YOLO (young Look Only Once) network model, a convolutional neural network model, or the like.
In another embodiment of the present invention, the initial license plate detection model comprises: a feature extraction layer and a feature regression and recognition layer;
the step S203 may include the following steps 021-:
021: and inputting each sample image into the feature extraction layer to obtain the sample image features corresponding to each sample image.
Wherein each sample image comprises: a sample original image and its corresponding sample gradient image.
022: and inputting the sample image characteristics corresponding to each sample image into a characteristic regression and identification layer to obtain the detection information of the position of the license plate in each sample image.
Wherein the detection information includes: detection position information, detection category information, and detection confidence information.
023: and determining a target loss value corresponding to each sample image by using a preset loss function, the detection position information, the detection category information and the detection confidence coefficient information of the position of the license plate in each sample image, and the calibration position information, the calibration category information and the calibration confidence coefficient information included in the calibration information corresponding to each sample image.
Wherein, the calibration information corresponding to each sample image is as follows: and the calibration information corresponding to the original sample image included in the sample image.
024: and adjusting network parameters of the initial license plate detection model by using the target loss value corresponding to each sample image and a preset loss threshold value until the initial license plate detection model converges to obtain a pre-trained license plate detection model comprising a feature extraction layer and a feature regression and recognition layer.
In one case, before the sample original image and the sample edge image corresponding to the sample original image are input to the feature extraction layer, the sample original image and the sample edge image corresponding to the sample original image may be scaled to a preset size, for example 416 × 416, and then input to the feature extraction layer.
In one case, in order to avoid the phenomenon of local extremum during network model training, sample images, i.e., sample original images and corresponding sample edge images, and calibration information corresponding to the sample images can be randomly input into the feature extraction layer of the initial license plate detection model. In one case, the sample images may be input to the images of the initial license plate detection models in batches, that is, the number of input images in each batch may be set, for example, the number of input images in each batch is set to be 16, that is, the number of images of the initial license plate detection models input in each batch is set to be 16, that is, the images of the initial license plate detection models input in each batch include 8 sample original images and 8 corresponding sample edge images.
Subsequently, inputting each batch of sample images into a feature extraction layer aiming at each batch of sample images to obtain sample image features corresponding to each sample image; further, inputting the sample image characteristics corresponding to each sample image into a characteristic regression and recognition layer to obtain detection position information, detection category information and detection confidence information of the position of the license plate in each sample image, and determining a target loss value corresponding to each sample image by using a preset loss function, the detection position information, the detection category information and the detection confidence information of the position of the license plate in each sample image, and calibration position information, calibration category information and calibration confidence information included in the calibration information corresponding to each sample image; further, determining a loss value corresponding to each batch by using a target loss value corresponding to the sample images included in each batch, comparing the loss value corresponding to the batch with a preset loss threshold value, and judging whether the loss value corresponding to the batch is smaller than the preset loss threshold value or not; if the judgment is smaller than the preset judgment threshold, determining that the initial license plate detection model is converged to obtain a pre-trained license plate detection model; if the judgment result is not less than the preset value, adjusting network parameters of a feature extraction layer and a feature regression and recognition layer of the initial license plate detection model, returning to execute sample images for each batch, and inputting each sample image for each batch into the feature extraction layer to obtain a sample image feature corresponding to each sample image; and determining the convergence of the initial license plate detection model until the loss value corresponding to the batch is judged to be smaller than a preset loss threshold value, so as to obtain a pre-trained license plate detection model.
The above process of inputting the sample image features corresponding to each sample image into the feature regression and recognition layer to obtain the detection position information, the detection category information, and the detection confidence information of the position of the license plate in each sample image may be: inputting the sample image characteristics corresponding to each sample image into a characteristic regression and recognition layer, detecting the positions of sample license plates in a sample original image and a sample edge image included in the sample image by utilizing at least one pair of width and height information in at least one group of anchor point size information obtained by pre-clustering, and obtaining the detection position information, the detection category information and the detection confidence information of the positions of the license plates in each sample image.
As described above, the target loss values corresponding to the sample images included in each batch may be used to determine the loss value corresponding to the batch, an average value of the target loss values corresponding to the sample images included in each batch may be calculated as the loss value corresponding to the batch, or a sum of the target loss values corresponding to the sample images included in each batch may be calculated as the loss value corresponding to the batch. The loss values corresponding to the batch are determined in different ways, and correspondingly, the preset loss thresholds are different.
In another embodiment of the present invention, the detection information of the position of the license plate in each sample image includes: the sample image comprises first detection position information, first detection category information and first detection confidence information of the position of the license plate in the sample original image, and second detection position information, second detection category information and second detection confidence information of the position of the license plate in the sample gradient image corresponding to the sample original image;
023, it can include the following steps:
0231: and for each sample image, determining a first loss value corresponding to the sample original image included in the sample image by using first detection position information, first detection category information and first detection confidence information of the position of the sample license plate in the sample original image included in the sample image, and calibration position information, calibration category information and calibration confidence information included in calibration information corresponding to the sample image.
0232: and determining a second loss value corresponding to the sample gradient image corresponding to the sample original image by using second detection position information, second detection category information and second detection confidence information of the position of the sample license plate in the sample gradient image corresponding to the sample original image included in the sample image, and calibration position information, calibration category information and calibration confidence information included in the calibration information corresponding to the sample image.
0233: and determining a target loss value corresponding to the sample image by using a first loss value corresponding to the sample original image included in the sample image and a second loss value corresponding to the sample gradient image corresponding to the sample original image.
In this implementation manner, the target loss value corresponding to the sample image may be determined according to the loss value corresponding to the sample original image included in the sample image and the corresponding loss value of the sample edge image included in the sample original image. Specifically, it can be expressed by the following formula (4):
L＝L1+(1-)L2； (4)
wherein L represents a target loss value corresponding to the sample image, L2A first loss value, L, corresponding to a sample original image included in the sample image1And indicating a second loss value corresponding to a sample edge image included in the sample image, and indicating a preset balance weight.
The process of determining the first loss value corresponding to the original sample image included in the sample image and the process of determining the second loss value corresponding to the gradient sample image corresponding to the original sample image can both be represented by the following formula (5):
Figure BDA0002306999920000151
wherein L isiRepresenting a first loss value corresponding to a sample original image included in the sample image or a second loss value corresponding to a sample gradient image corresponding to the sample original image included in the sample image; truthrIndicating calibration position information included in calibration information corresponding to the sample image, namely position information (x, y) of a center point of the calibrated sample license plate and width and height information w and h of the sample license plate; truthclassThe calibration type information of the sample license plate in the sample original image included in the sample image is represented; truthconfRepresenting the calibration confidence information of the sample license plate in the sample original image included in the sample image; question mark "? "is a conditional expression, which means that if the judgment condition in front of the question mark is true, the result is 1, otherwise it is 0;
wherein L isiIndicating a first loss value corresponding to a sample original image included in the sample imagerThe method comprises the steps that first detection position information representing the position of a sample license plate in a sample original image included in a sample image is obtained, namely position information of the center point of the detected sample license plate and width and height information of the sample license plate; predictclassFirst detection category information representing a sample license plate in a sample original image included in the sample image; predictconfRepresenting first detection confidence information of a sample license plate in a sample original image included in the sample image;
wherein L isiIndicating a second loss value corresponding to a sample gradient image corresponding to a sample original image included in the sample imagerSecond detection position information representing the position of the sample license plate in the sample edge image corresponding to the sample original image included in the sample image, namely position information of the center point of the detected sample license plate and width and height information of the sample license plate; predictclassSecond detection category information of a sample license plate in a sample edge image corresponding to a sample original image included in the sample image is represented; predictconfAnd second detection confidence information of the sample license plate in the sample edge image corresponding to the sample original image included in the sample image is represented.
In one case, if the sample images are input in batches, the target loss value corresponding to each batch of sample images can be calculated by the above formula.
In another embodiment of the present invention, the pre-trained license plate detection model corresponds to at least one group of anchor point size information, each group of anchor point size information includes at least one pair of width and height information, wherein each group of anchor point size information includes at least one pair of width and height information: obtaining the license plate size information of a sample license plate in a sample original image of a pre-trained license plate detection model through clustering according to training;
the step S103 may include the following steps 031-:
031: inputting the image to be detected and the gradient image to be detected into a feature extraction layer of a pre-trained license plate detection model, and determining a first feature map of a first size corresponding to the image to be detected and a second feature map of the first size corresponding to the gradient image to be detected.
The first characteristic diagram comprises the image characteristics of the image to be detected, and the second characteristic diagram comprises the image characteristics of the gradient image to be detected.
032: determining whether the image to be detected contains the license plate or not by utilizing the first characteristic diagram, the second characteristic diagram, the characteristic regression and recognition layer of the pre-trained license plate detection model and each pair of width and height information in at least one pair of width and height information included by at least one group of anchor point size information;
033: and if the image to be detected contains the license plate, determining the position information of the license plate contained in the image to be detected.
In this embodiment, at least one group of anchor point size information may be obtained by clustering in advance using the size information of the sample license plate in the sample original image, where each group of anchor point size information includes at least one pair of width and height information. Subsequently, in the process of determining the position information of the license plate contained in the image to be detected by the electronic equipment by using the image to be detected, the gradient image to be detected and the pre-trained license plate detection model, the image to be detected and the gradient image to be detected can be input into a feature extraction layer of the pre-trained license plate detection model to obtain a first feature map of a first size corresponding to the image to be detected and a second feature map of the first size corresponding to the gradient image to be detected, so that the image features corresponding to the image to be detected and the image features corresponding to the gradient image to be detected are obtained; inputting the first characteristic diagram and the second characteristic diagram into a characteristic regression and recognition layer of a pre-trained license plate detection model, determining whether an image to be detected contains a license plate or not by combining each pair of width and height information in at least one pair of width and height information included by at least one group of anchor point size information, and determining the position information of the license plate contained in the image to be detected under the condition that the image to be detected contains the license plate, wherein the determined position information of the license plate contained in the image to be detected, the corresponding category information represents that the license plate exists in the position information, and the corresponding confidence coefficient information exceeds a preset confidence coefficient threshold.
In one case, the electronic device can determine confidence information corresponding to the position information and output the confidence information when determining the position information of the license plate contained in the image to be detected.
In the process of utilizing a pre-trained license plate detection model to carry out intelligent identification on the license plate of a vehicle suitable for a complex illumination environment, the suspected license plate areas in an image to be detected and an edge image to be detected are extracted from the image to be detected and the edge image to be detected mainly through at least one group of anchor point size information corresponding to the pre-trained license plate detection model and image characteristics corresponding to the image to be detected and gradient images to be detected, and then whether the extracted suspected license plate areas comprise license plates or not is identified. After the original sample image corresponding to the pre-trained license plate detection model is fixed, the size information of the at least one group of anchor points is fixed, correspondingly, the size information of the license plate in the image, which can be detected by the pre-trained license plate detection model, is also fixed correspondingly, and in the actual detection process, when the size information of the license plate in the image has a larger difference with at least one pair of width and height information in the size information of the at least one group of anchor points, the condition that the license plate is missed to be detected is easy to occur.
In order to solve the above problem, in another embodiment of the present invention, the step S103 may further include the following steps 034-037:
034: if the image to be detected does not contain the license plate by utilizing the first feature diagram, the second feature diagram, the feature regression and recognition layer of the pre-trained license plate detection model and each pair of width and height information in at least one pair of width and height information included by at least one group of anchor point size information, obtaining the at least one pair of width and height information included by the first anchor point size information, and obtaining the first width and height information after scaling based on a first scaling coefficient corresponding to the first anchor point size information.
Wherein, the first anchor point size information is: the largest set of anchor size information, or the smallest set of anchor size information, of the at least one set of anchor size information. The first aspect information may be: the electronic equipment is stored in a preset storage space after being determined in advance based on at least one pair of width and height information and a first scaling coefficient which are included in the first anchor point size information. The electronic equipment directly obtains the image to be detected from a preset storage space after determining that the image to be detected does not contain the license plate by using each pair of width and height information in at least one pair of width and height information included by the image to be detected, the gradient image to be detected, the pre-trained license plate detection model and at least one group of anchor point size information; it can also be: after the electronic equipment determines that the image to be detected does not contain the license plate by using each pair of width and height information in at least one pair of width and height information included by the image to be detected, the gradient image to be detected, the pre-trained license plate detection model and at least one set of anchor point size information, the electronic equipment performs scaling by using at least one pair of width and height information included by the first anchor point size information and a first scaling coefficient corresponding to the first anchor point size information, and calculates the scaling coefficient in real time.
035: and obtaining a third feature map of the second size corresponding to the image to be detected and a fourth feature map of the second size corresponding to the gradient image to be detected.
Wherein, the third characteristic diagram and the fourth characteristic diagram are both: and inputting the image to be detected and the gradient image to be detected into a feature extraction layer of a pre-trained license plate detection model to obtain a feature map, wherein the third feature map comprises the image features of the image to be detected, and the fourth feature map comprises the image features of the gradient image to be detected.
S36: and determining whether the image to be detected contains the license plate or not by utilizing the third feature map, the fourth feature map, the feature regression and recognition layer of the pre-trained license plate detection model and the first width and height information.
037: and if the image to be detected contains the license plate, determining the position information of the license plate contained in the image to be detected.
In one case, if the electronic device determines that the image to be detected does not contain the license plate by using each pair of width and height information in at least one pair of width and height information included in the image to be detected, the gradient image to be detected, the pre-trained license plate detection model and at least one set of anchor point size information, the electronic device can take the anchor point size information with the largest grade, namely the anchor point size information with the larger size, or the anchor point size information with the smallest grade, namely the anchor point size information with the smaller size, as the first anchor point size information, obtain the width and height information included in the first anchor point size information, and obtain the first width and height information after scaling based on a first scaling coefficient corresponding to the first anchor point size information; further, a third feature map of a second size corresponding to the image to be detected and a fourth feature map of the second size corresponding to the gradient image to be detected are obtained, the third feature map, the fourth feature map and the first width and height information are input into a feature regression and recognition layer of a pre-trained license plate detection model, by utilizing the image characteristics corresponding to the image to be detected in the third characteristic diagram and the image characteristics corresponding to the gradient image to be detected in the fourth characteristic diagram, and first width and height information, determining a suspected license plate area from the image to be detected, performing position regression on the suspected license plate area to obtain position information of the regressed area, determining whether the regressed area comprises a license plate or not, and determining that the regressed region comprises the license plate and the corresponding confidence information exceeds a preset confidence threshold value, so as to obtain the position information of the regressed region, namely the position information of the position of the license plate contained in the image to be detected. Correspondingly, the confidence information corresponding to the suspected license plate area can be obtained.
If a group of anchor point size information with the maximum grade, namely a group of anchor point size information with a larger size, is taken as first anchor point size information, correspondingly, a first scaling coefficient corresponding to the first anchor point size information is a scaling coefficient corresponding to the group of anchor point size information with the maximum grade, and the second size is larger than the first size. In order to enlarge the range of size information of the license plate that can be detected by the pre-trained license plate detection model, the first anchor point size information needs to be enlarged, that is, at least one pair of width and height information included in the first anchor point size information is enlarged. The scaling factor corresponding to the set of anchor point size information with the highest level is a value greater than 1. In one case, the set range of the scaling factor corresponding to the set of anchor point size information with the highest level may be [1.1, 1.3 ].
If a group of anchor point size information with the minimum grade, namely a group of anchor point size information with a smaller size, is taken as first anchor point size information, correspondingly, a first scaling coefficient corresponding to the first anchor point size information is a scaling coefficient corresponding to the group of anchor point size information with the minimum grade, and the second size is smaller than the first size. In order to expand the range of size information of the license plate that can be detected by the pre-trained license plate detection model, the first anchor point size information needs to be reduced, that is, at least one pair of width and height information included in the first anchor point size information needs to be reduced. The scaling factor corresponding to the set of anchor point size information with the smallest level is a value smaller than 1 and larger than 0. In one case, the set range of scaling factors corresponding to the set of anchor point size information with the smallest level may be [0.8, 0.9 ].
In the embodiment, by fixing the anchor point size information and the adaptive anchor point size information, that is, adjusting at least one pair of width and height information included in at least one group of anchor point size information, the license plate and the position information of the license plate included in the image to be detected are jointly detected, the size of the sample vehicle included in the sample original image is weakened to a certain extent, the limitation on the size of the license plate which can be detected by a pre-trained license plate detection model is realized, the detection rate of the license plate is improved when the size of the license plate is different from that of the license plate in the sample original image in the actual detection process, and the condition that the license plate is missed to be detected when the size of the license plate is different from that of the license plate in the.
The self-adaptive anchor point detection method comprises the steps of adjusting first anchor point size information and determining whether an image to be detected contains a license plate or not based on the first anchor point size information, wherein the process of extracting image features of the image to be detected and an edge image to be detected by a pre-trained license plate detection model is carried out, the related classification and regression operation calculation speed is high, namely the process of extracting a suspected vehicle region from the image to be detected and classifying is high in calculation speed, and the consumed calculation amount can be ignored relative to the calculation amount consumed by image feature extraction of the pre-trained license plate detection model.
In another embodiment of the present invention, the step S103 may further include the following steps 038-:
038: and if the third feature map, the fourth feature map, the feature regression and recognition layer of the pre-trained license plate detection model and the first width and height information are utilized, determining that the image to be detected does not contain the license plate, obtaining at least one pair of width and height information included by the second anchor point size information, and obtaining second width and height information after scaling based on a second scaling coefficient corresponding to the second anchor point size information.
Wherein, the second anchor size information is: the minimum set of anchor size information, or the maximum set of anchor size information, of the at least one set of anchor size information. The second aspect information may be: the electronic equipment is stored in a preset storage space after being determined in advance based on at least one pair of width and height information and a second scaling factor included in the second anchor point size information. The electronic equipment directly obtains the image to be detected from a preset storage space after determining that the image to be detected does not contain the license plate by using each pair of width and height information in at least one pair of width and height information included by the image to be detected, the gradient image to be detected, the pre-trained license plate detection model and at least one group of anchor point size information; it can also be: after the electronic equipment determines that the image to be detected does not contain the license plate by using each pair of width and height information in at least one pair of width and height information included by the image to be detected, the gradient image to be detected, the pre-trained license plate detection model and at least one set of anchor point size information, the electronic equipment performs scaling by using at least one pair of width and height information included by the second anchor point size information and a second scaling coefficient corresponding to the second anchor point size information, and calculates the scaling coefficient in real time.
039: and acquiring a fifth feature map of a third size corresponding to the image to be detected and a sixth feature map of a third size corresponding to the gradient image to be detected.
Wherein, the fifth characteristic diagram and the sixth characteristic diagram are both: inputting the image to be detected and the gradient image to be detected into a feature extraction layer of a pre-trained license plate detection model to obtain feature maps, wherein the fifth feature map comprises image features corresponding to the image to be detected, and the sixth feature map comprises image features corresponding to the edge image to be detected;
0310: determining whether the image to be detected contains the license plate or not by utilizing the fifth feature map, the sixth feature map, the feature regression and recognition layer of the pre-trained license plate detection model and the second width and height information;
0311: and if the image to be detected contains the license plate, determining the position information of the license plate contained in the image to be detected.
In this embodiment, after adjusting at least one pair of widths and heights included in a group of anchor point size information, a license plate is not detected from an image to be detected, that is, if the electronic device utilizes the first feature map, the second feature map, the feature extraction layer of the pre-trained license plate detection model, and the first width and height information, it is determined that the image to be detected does not include a license plate; the electronic equipment can continue to use the group of anchor point size information with the minimum grade or the group of anchor point size information with the maximum grade as second anchor point size information, obtain at least one pair of width and height information included in the second anchor point size information, and obtain second width and height information obtained after scaling is performed on the basis of a second scaling coefficient corresponding to the second anchor point size information; further, a fifth feature map of a third size corresponding to the image to be detected and a sixth feature map of the third size corresponding to the gradient image to be detected are obtained, the fifth feature map, the sixth feature map and second width and height information are input into a feature regression and recognition layer of a pre-trained license plate detection model, by utilizing the image characteristics corresponding to the image to be detected in the fifth characteristic diagram and the image characteristics corresponding to the gradient image to be detected in the sixth characteristic diagram, and second width and height information, determining a suspected license plate region from the image to be detected, performing position regression on the suspected license plate region to obtain position information of the regressed region, determining whether the regressed region includes a license plate, and determining that the regressed region comprises the license plate and the corresponding confidence information exceeds a preset confidence threshold value, so as to obtain the position information of the regressed region, namely the position information of the position of the license plate contained in the image to be detected. Correspondingly, the confidence information corresponding to the suspected license plate area can be obtained.
If the electronic device determines a group of anchor point size information with the largest grade as first anchor point size information, and correspondingly, the electronic device determines a group of anchor point size information with the smallest grade as second anchor point size information, and correspondingly, a second scaling coefficient corresponding to the second anchor point size information is a scaling coefficient corresponding to the group of anchor point size information with the smallest grade, and the third size is smaller than the first size. If the electronic device determines the group of anchor point size information with the minimum grade as the first anchor point size information, and correspondingly, the electronic device determines the group of anchor point size information with the maximum grade as the second anchor point size information, and correspondingly, the second scaling coefficient corresponding to the second anchor point size information is the scaling coefficient corresponding to the group of anchor point size information with the maximum grade, and the third size is larger than the first size.
Under the condition that the electronic equipment firstly determines a group of anchor point size information with the minimum grade as first anchor point size information and then determines a group of anchor point size information with the maximum grade as second anchor point size information, the second size is smaller than the first size, and the first size is smaller than the third size. In one case, a pre-trained license plate detection model is input
As shown in fig. 3, the data interaction diagram is a schematic diagram for jointly detecting the license plate and the position information thereof included in the image to be detected through the fixed anchor point size information and the adaptive anchor point size information. As shown in fig. 3, after the electronic device obtains the image to be detected and the corresponding edge image to be detected, the image to be detected and the corresponding edge image to be detected are adjusted to a preset size, such as 416 × 416, and are input to the feature extraction layer of the pre-trained license plate detection model, so as to obtain a first feature map of a first size corresponding to the image to be detected and a second feature map of a first size corresponding to the gradient image to be detected, such as "output 1" shown in fig. 3, where the first size may be 26 × 26, where the pre-trained license plate detection model may be a CNN (Convolutional Neural Network) model, and the feature extraction layer of the pre-trained license plate detection model may be a CNN backbone Network for feature extraction; inputting each pair of width and height information in at least one pair of width and height information included by the first feature map, the second feature map and the at least one group of anchor point size information into a feature regression and recognition layer of a pre-trained license plate detection model, determining a suspected license plate region from an image to be detected by using each pair of width and height information in at least one pair of width and height information included by the at least one group of anchor point size information, namely 'anchor 1' shown in fig. 3, the first feature map and the second feature map, performing position regression on the suspected license plate region, performing 'target detection regression' shown in fig. 3 to obtain position information of the regressed region, determining whether the regressed region includes a license plate, determining the position information of the region where the license plate is included in the image to be detected after determining that the regressed region includes the license plate, and outputting the position information.
In one case, the specific structure of the CNN backbone network may refer to a network structure part of YOLO (young Only Look Once) V3 for feature extraction in the related art that adopts the network structure of Darknet-53.
If the regressed region does not include the license plate, determining that the image to be detected does not include the license plate; adaptive anchor size information, obtaining at least one pair of width and height information included in the first anchor size information, obtaining first width and height information obtained after scaling based on a first scaling coefficient corresponding to the first anchor size information, such as "anchor 2" shown in fig. 3, obtaining a third feature map of a second size corresponding to the image to be detected and a fourth feature map of the second size corresponding to the gradient image to be detected, such as "output 2" shown in fig. 3, where the second size may be 13 × 13 or 52 × 52, and is related to the first width and height information; inputting the third feature map, the fourth feature map and the first width and height information into a feature regression and recognition layer of a pre-trained license plate detection model, determining a suspected license plate region from the image to be detected by using the third feature map, the fourth feature map and the first width and height information, performing position regression on the suspected license plate region, obtaining position information of the regressed region as shown in the target detection regression in fig. 3, determining whether the regressed region includes a license plate, determining the position information of the region where the license plate is located included in the image to be detected after determining that the regressed region includes the license plate, and outputting the position information.
If the regressed region does not include the license plate, determining that the image to be detected does not include the license plate; adaptive anchor point size information, obtaining at least one pair of width and height information included in the second anchor point size information, and obtaining second width and height information obtained after scaling based on a second scaling coefficient corresponding to the second anchor point size information, such as "anchor 3" shown in fig. 3, obtaining a fifth feature map of a third size corresponding to the image to be detected and a sixth feature map of the third size corresponding to the gradient image to be detected, such as "output 3" shown in fig. 3, where the third size may be 52 x 52 or 13 x 13 and is related to the second width and height information; inputting the fifth feature map and the sixth feature map into a feature regression and recognition layer of a pre-trained license plate detection model, determining a suspected license plate region from an image to be detected by using the fifth feature map, the sixth feature map and second width and height information, performing position regression on the suspected license plate region, obtaining position information of the regressed region as shown in a target detection regression in fig. 3, determining whether the regressed region comprises a license plate, determining the position information of the region where the license plate is located contained in the image to be detected after determining that the regressed region comprises the license plate, and outputting the position information; and after the regression area is determined not to contain the license plate, determining that the image to be detected does not contain the license plate, and ending the process.
Corresponding to the above method embodiment, an embodiment of the present invention provides an intelligent vehicle license plate recognition apparatus suitable for a complex lighting environment, and as shown in fig. 4, the intelligent vehicle license plate recognition apparatus may include:
a first obtaining module 410 configured to obtain an image to be detected;
a first determining module 420, configured to perform edge detection on the image to be detected by using a preset edge detection algorithm, and determine a gradient image to be detected corresponding to the image to be detected;
a second determining module 430, configured to determine whether the image to be detected contains a license plate or not by using the image to be detected, the gradient image to be detected, and a pre-trained license plate detection model, and determine position information of the license plate contained in the image to be detected under the condition that it is determined that the image to be detected contains the license plate, where the pre-trained license plate detection model is: and training the obtained model based on the sample original image marked with the position of the sample vehicle and the sample edge image corresponding to the sample original image.
By applying the embodiment of the invention, the detection of the image to be detected and the gradient image to be detected thereof can be realized based on the sample original image marked with the position of the sample vehicle and the sample edge image corresponding to the sample original image which is less influenced by the change of the illumination condition, and the pre-trained license plate detection model is trained, so that the position information of the license plate contained in the image to be detected is determined under the condition that the image to be detected contains the license plate. The method comprises the steps of training to obtain edge characteristics based on images through a sample original image and a sample edge image corresponding to the sample original image, detecting a pre-trained license plate detection model of the position where a license plate is located, wherein the sample edge image is less influenced by the change of illumination conditions, and even if the sample edge image is located in a complex illumination environment, namely a scene with large illumination change, the license plate contained in the sample edge image and the position where the license plate is located can be detected through the pre-trained license plate detection model, so that the detection rate of license plate detection in the complex illumination environment, such as the environment scene with large illumination change, is improved.
In another embodiment of the present invention, the first determining module 420 is specifically configured to perform edge detection on the image to be detected in the horizontal direction by using a first filter operator, and determine a first gradient image corresponding to the image to be detected;
utilizing a second filter operator to carry out edge detection on the image to be detected in the vertical direction, and determining a second gradient image corresponding to the image to be detected;
and determining the gradient image to be detected corresponding to the image to be detected by utilizing the first gradient image and the second gradient image.
In another embodiment of the present invention, the apparatus further comprises:
a training module (not shown in the figure), configured to determine whether the image to be detected contains a license plate or not by using the image to be detected, the gradient image to be detected, and a pre-trained license plate detection model, and train to obtain the pre-trained license plate detection model before determining position information of the license plate contained in the image to be detected under the condition that the image to be detected contains the license plate, wherein the training module includes:
an obtaining unit (not shown in the figure) configured to obtain a plurality of sample original images and calibration information thereof, wherein each sample original image contains a license plate, and each calibration information includes: calibrating position information, calibrating type information and calibrating confidence information of the position of the sample license plate in the corresponding sample original image;
a determining unit (not shown in the figure), configured to perform edge detection on each sample original image by using the preset edge detection algorithm, and determine a sample gradient image corresponding to each sample original image;
and a training unit (not shown in the figure) configured to train an initial license plate detection model by using the calibration position information, the calibration category information and the calibration confidence information included in the calibration information corresponding to each sample original image, the sample gradient image corresponding to each sample original image and the calibration information corresponding to each sample original image, so as to obtain the pre-trained license plate detection model.
In another embodiment of the present invention, the initial license plate detection model includes: a feature extraction layer and a feature regression and recognition layer;
the training unit includes:
a first input sub-module (not shown in the figure), configured to input each sample image into the feature extraction layer, and obtain a sample image feature corresponding to each sample image, where each sample image includes: a sample original image and a sample gradient image corresponding to the sample original image;
a second input sub-module (not shown in the figure), configured to input the sample image features corresponding to each sample image into the feature regression and recognition layer, so as to obtain detection information of a position of a license plate of the sample in each sample image, where the detection information includes: detecting position information, detection category information and detection confidence information;
a determining sub-module (not shown in the figure), configured to determine a target loss value corresponding to each sample image by using a preset loss function, the detection position information, the detection category information, and the detection confidence information of the position of the license plate of the sample in each sample image, and the calibration position information, the calibration category information, and the calibration confidence information included in the calibration information corresponding to each sample image, where the calibration information corresponding to each sample image is: the calibration information corresponding to the original sample image included in the sample image;
and an adjusting submodule (not shown in the figure) configured to adjust network parameters of an initial license plate detection model by using a target loss value and a preset loss threshold value corresponding to each sample image until the initial license plate detection model converges, so as to obtain a pre-trained license plate detection model including the feature extraction layer and the feature regression and recognition layer.
In another embodiment of the present invention, the detection information of the position of the license plate in each sample image includes: the sample image comprises first detection position information, first detection category information and first detection confidence information of the position of the license plate in the sample original image, and second detection position information, second detection category information and second detection confidence information of the position of the license plate in the sample gradient image corresponding to the sample original image;
the determining sub-module is specifically configured to determine, for each sample image, a first loss value corresponding to the sample original image included in the sample image by using first detection position information, first detection category information, and first detection confidence information of a position where a sample license plate is located in the sample original image included in the sample image, and calibration position information, calibration category information, and calibration confidence information included in calibration information corresponding to the sample image;
determining a second loss value corresponding to the sample gradient image corresponding to the sample original image by using second detection position information, second detection category information and second detection confidence information of the position of the sample license plate in the sample gradient image corresponding to the sample original image included in the sample image, and calibration position information, calibration category information and calibration confidence information included in the calibration information corresponding to the sample image;
and determining a target loss value corresponding to the sample image by using a first loss value corresponding to the sample original image included in the sample image and a second loss value corresponding to the sample gradient image corresponding to the sample original image.
In another embodiment of the present invention, the pre-trained license plate detection model corresponds to at least one group of anchor point size information, each group of anchor point size information includes at least one pair of width and height information, wherein each group of anchor point size information includes at least one pair of width and height information: clustering the license plate size information of the sample license plate in the sample original image of the pre-trained license plate detection model according to the training;
the second determining module 430 is specifically configured to
Inputting the image to be detected and the gradient image to be detected into a feature extraction layer of a pre-trained license plate detection model to obtain a first feature map of a first size corresponding to the image to be detected and a second feature map of the first size corresponding to the gradient image to be detected;
determining whether the image to be detected contains the license plate or not by utilizing the first feature map, the second feature map, a feature regression and recognition layer of a pre-trained license plate detection model and at least one pair of width and height information included by the at least one group of anchor point size information;
and if the image to be detected contains the license plate, determining the position information of the license plate contained in the image to be detected.
In another embodiment of the present invention, the second determining module 430 is specifically configured to determine that the image to be detected does not include the license plate if the first feature map, the second feature map, the feature regression and recognition layer of the pre-trained license plate detection model, and at least one pair of width and height information included in the at least one group of anchor point size information are utilized, obtain at least one pair of width and height information included in the first anchor point size information, and obtain the first width and height information obtained after scaling based on a first scaling coefficient corresponding to the first anchor point size information, where the first anchor point size information is: a largest group of anchor size information or a smallest group of anchor size information in the at least one group of anchor size information;
obtaining a third feature map of a second size corresponding to the image to be detected and a fourth feature map of the second size corresponding to the gradient image to be detected, wherein the third feature map and the fourth feature map are both: inputting the image to be detected and the gradient image to be detected into a feature extraction layer of a pre-trained license plate detection model to obtain a feature map;
determining whether the image to be detected contains the license plate or not by utilizing the third feature map, the fourth feature map, a feature regression and recognition layer of a pre-trained license plate detection model and the first width and height information;
and if the image to be detected contains the license plate, determining the position information of the license plate contained in the image to be detected.
In another embodiment of the present invention, the second determining module 430 is specifically configured to, if the third feature map, the fourth feature map, a feature regression and recognition layer of a pre-trained license plate detection model, and the first width and height information are utilized to determine that the image to be detected does not include a license plate, obtain at least one pair of width and height information included in the second anchor point size information, and obtain the second width and height information obtained after scaling based on a second scaling coefficient corresponding to the second anchor point size information, where the second anchor point size information is: a minimum set of anchor size information or a maximum set of anchor size information of the at least one set of anchor size information;
obtaining a fifth feature map of a third size corresponding to the image to be detected and a sixth feature map of the third size corresponding to the gradient image to be detected, wherein the fifth feature map and the sixth feature map are both: inputting the image to be detected and the gradient image to be detected into a feature extraction layer of a pre-trained license plate detection model to obtain a feature map;
determining whether the image to be detected contains the license plate or not by utilizing the fifth feature map, the sixth feature map, a feature regression and recognition layer of a pre-trained license plate detection model and the second width and height information;
and if the image to be detected contains the license plate, determining the position information of the license plate contained in the image to be detected.
The device and system embodiments correspond to the method embodiments, and have the same technical effects as the method embodiments, and specific descriptions refer to the method embodiments. The device embodiment is obtained based on the method embodiment, and for specific description, reference may be made to the method embodiment section, which is not described herein again.
Those of ordinary skill in the art will understand that: the figures are merely schematic representations of one embodiment, and the blocks or flow diagrams in the figures are not necessarily required to practice the present invention.
Those of ordinary skill in the art will understand that: modules in the devices in the embodiments may be distributed in the devices in the embodiments according to the description of the embodiments, or may be located in one or more devices different from the embodiments with corresponding changes. The modules of the above embodiments may be combined into one module, or further split into multiple sub-modules.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims (9)

1. A vehicle license plate intelligent identification method suitable for a complex illumination environment is characterized by comprising the following steps:
obtaining an image to be detected;
performing edge detection on the image to be detected by using a preset edge detection algorithm, and determining a gradient image to be detected corresponding to the image to be detected;
determining whether the image to be detected contains a license plate or not by using the image to be detected, the gradient image to be detected and a pre-trained license plate detection model, and determining the position information of the license plate contained in the image to be detected under the condition that the image to be detected contains the license plate, wherein the pre-trained license plate detection model is as follows: training the obtained model based on the sample original image marked with the position of the sample vehicle and the sample edge image corresponding to the sample original image; the pre-trained license plate detection model corresponds to at least one group of anchor point size information, each group of anchor point size information comprises at least one pair of width and height information, wherein each group of anchor point size information comprises at least one pair of width and height information: clustering the license plate size information of the sample license plate in the sample original image of the pre-trained license plate detection model according to the training;
the method comprises the following steps of utilizing the image to be detected, the gradient image to be detected and a pre-trained license plate detection model to determine whether the image to be detected contains a license plate or not, and determining the position information of the license plate contained in the image to be detected under the condition that the image to be detected contains the license plate, wherein the steps comprise:
inputting the image to be detected and the gradient image to be detected into a feature extraction layer of a pre-trained license plate detection model to obtain a first feature map of a first size corresponding to the image to be detected and a second feature map of the first size corresponding to the gradient image to be detected;
determining whether the image to be detected contains the license plate or not by utilizing the first feature map, the second feature map, a feature regression and recognition layer of a pre-trained license plate detection model and each pair of width and height information in at least one pair of width and height information included by the at least one group of anchor point size information;
and if the image to be detected contains the license plate, determining the position information of the license plate contained in the image to be detected.
2. The method according to claim 1, wherein the step of performing edge detection on the image to be detected by using a preset edge detection algorithm to determine the gradient image to be detected corresponding to the image to be detected comprises:
utilizing a first filter operator to carry out edge detection on the image to be detected in the horizontal direction, and determining a first gradient image corresponding to the image to be detected;
utilizing a second filter operator to carry out edge detection on the image to be detected in the vertical direction, and determining a second gradient image corresponding to the image to be detected;
and determining the gradient image to be detected corresponding to the image to be detected by utilizing the first gradient image and the second gradient image.
3. The method of claim 1, wherein before the step of determining whether the image to be detected contains a license plate by using the image to be detected, the gradient image to be detected and a pre-trained license plate detection model, and determining the position information of the license plate contained in the image to be detected in case that the image to be detected contains a license plate, the method further comprises:
a process of training to obtain the pre-trained license plate detection model, wherein the process comprises:
obtaining a plurality of sample original images and calibration information thereof, wherein each sample original image contains a license plate, and each calibration information comprises: calibrating position information, calibrating type information and calibrating confidence information of the position of the sample license plate in the corresponding sample original image;
performing edge detection on each sample original image by using the preset edge detection algorithm, and determining a sample gradient image corresponding to each sample original image;
and training an initial license plate detection model by using each sample original image, the sample gradient image corresponding to each sample original image and calibration position information, calibration category information and calibration confidence information which are included in calibration information corresponding to each sample original image to obtain the pre-trained license plate detection model.
4. The method of claim 3, wherein the initial license plate detection model comprises: a feature extraction layer and a feature regression and recognition layer;
the method for obtaining the pre-trained license plate detection model by training an initial license plate detection model by using the sample original image, the sample gradient image corresponding to the sample original image and the calibration position information included in the calibration information corresponding to the sample original image comprises the following steps:
inputting each sample image into the feature extraction layer to obtain a sample image feature corresponding to each sample image, wherein each sample image comprises: a sample original image and a sample gradient image corresponding to the sample original image;
inputting the sample image characteristics corresponding to each sample image into the characteristic regression and identification layer to obtain the detection information of the position of the license plate of the sample in each sample image, wherein the detection information comprises: detecting position information, detection category information and detection confidence information;
determining a target loss value corresponding to each sample image by using a preset loss function, the detection position information, the detection category information and the detection confidence information of the position of the sample license plate in each sample image, and the calibration position information, the calibration category information and the calibration confidence information included in the calibration information corresponding to each sample image, wherein the calibration information corresponding to each sample image is as follows: the calibration information corresponding to the original sample image included in the sample image;
and adjusting network parameters of an initial license plate detection model by using a target loss value corresponding to each sample image and a preset loss threshold value until the initial license plate detection model converges to obtain a pre-trained license plate detection model comprising the feature extraction layer and the feature regression and recognition layer.
5. The method of claim 4, wherein the detection information of the position of the license plate in each sample image comprises: the sample image comprises first detection position information, first detection category information and first detection confidence information of the position of the license plate in the sample original image, and second detection position information, second detection category information and second detection confidence information of the position of the license plate in the sample gradient image corresponding to the sample original image;
the step of determining the target loss value corresponding to each sample image by using the preset loss function, the detection position information, the detection category information and the detection confidence information of the position of the sample license plate in each sample image, and the calibration position information, the calibration category information and the calibration confidence information included in the calibration information corresponding to each sample image comprises:
for each sample image, determining a first loss value corresponding to the sample original image included in the sample image by using first detection position information, first detection category information and first detection confidence information of the position of the sample license plate in the sample original image included in the sample image, and calibration position information, calibration category information and calibration confidence information included in calibration information corresponding to the sample image;
determining a second loss value corresponding to the sample gradient image corresponding to the sample original image by using second detection position information, second detection category information and second detection confidence information of the position of the sample license plate in the sample gradient image corresponding to the sample original image included in the sample image, and calibration position information, calibration category information and calibration confidence information included in the calibration information corresponding to the sample image;
and determining a target loss value corresponding to the sample image by using a first loss value corresponding to the sample original image included in the sample image and a second loss value corresponding to the sample gradient image corresponding to the sample original image.
6. The method of claim 1, wherein the method further comprises:
if the image to be detected does not contain the license plate by utilizing the first feature diagram, the second feature diagram, the feature regression and recognition layer of the pre-trained license plate detection model and each pair of width and height information in at least one pair of width and height information included by the at least one group of anchor point size information, obtaining the at least one pair of width and height information included by the first anchor point size information, and obtaining the first width and height information obtained after scaling based on a first scaling coefficient corresponding to the first anchor point size information, wherein the first anchor point size information is: a largest group of anchor size information or a smallest group of anchor size information in the at least one group of anchor size information;
obtaining a third feature map of a second size corresponding to the image to be detected and a fourth feature map of the second size corresponding to the gradient image to be detected, wherein the third feature map and the fourth feature map are both: inputting the image to be detected and the gradient image to be detected into a feature extraction layer of a pre-trained license plate detection model to obtain a feature map;
determining whether the image to be detected contains the license plate or not by utilizing the third feature map, the fourth feature map, a feature regression and recognition layer of a pre-trained license plate detection model and the first width and height information;
and if the image to be detected contains the license plate, determining the position information of the license plate contained in the image to be detected.
7. The method of claim 6, wherein the method further comprises:
if the third feature map, the fourth feature map, a feature regression and recognition layer of a pre-trained license plate detection model and the first width and height information are utilized to determine that the image to be detected does not contain a license plate, at least one pair of width and height information included in the second anchor point size information is obtained, and the second width and height information is obtained after scaling is performed on the basis of a second scaling coefficient corresponding to the second anchor point size information, wherein the second anchor point size information is as follows: a minimum set of anchor size information or a maximum set of anchor size information of the at least one set of anchor size information;
obtaining a fifth feature map of a third size corresponding to the image to be detected and a sixth feature map of the third size corresponding to the gradient image to be detected, wherein the fifth feature map and the sixth feature map are both: inputting the image to be detected and the gradient image to be detected into a feature extraction layer of a pre-trained license plate detection model to obtain a feature map;
determining whether the image to be detected contains the license plate or not by utilizing the fifth feature map, the sixth feature map, a feature regression and recognition layer of a pre-trained license plate detection model and the second width and height information;
and if the image to be detected contains the license plate, determining the position information of the license plate contained in the image to be detected.
8. An intelligent vehicle license plate recognition device suitable for complex lighting environments, the device comprising:
a first obtaining module configured to obtain an image to be detected;
the first determining module is configured to perform edge detection on the image to be detected by using a preset edge detection algorithm, and determine a gradient image to be detected corresponding to the image to be detected;
the second determining module is configured to determine whether the image to be detected contains a license plate or not by using the image to be detected, the gradient image to be detected and a pre-trained license plate detection model, and determine position information of the license plate contained in the image to be detected under the condition that the image to be detected contains the license plate, wherein the pre-trained license plate detection model is as follows: training the obtained model based on the sample original image marked with the position of the sample vehicle and the sample edge image corresponding to the sample original image; the pre-trained license plate detection model corresponds to at least one group of anchor point size information, each group of anchor point size information comprises at least one pair of width and height information, wherein each group of anchor point size information comprises at least one pair of width and height information: clustering the license plate size information of the sample license plate in the sample original image of the pre-trained license plate detection model according to the training;
the second determining module is specifically configured to input the image to be detected and the gradient image to be detected into a feature extraction layer of a pre-trained license plate detection model, so as to obtain a first feature map of a first size corresponding to the image to be detected and a second feature map of the first size corresponding to the gradient image to be detected;
determining whether the image to be detected contains the license plate or not by utilizing the first feature map, the second feature map, a feature regression and recognition layer of a pre-trained license plate detection model and each pair of width and height information in at least one pair of width and height information included by the at least one group of anchor point size information;
and if the image to be detected contains the license plate, determining the position information of the license plate contained in the image to be detected.
9. The apparatus of claim 8, wherein the first determining module is specifically configured to perform edge detection on the image to be detected in a horizontal direction by using a first filter operator, and determine a first gradient image corresponding to the image to be detected;
utilizing a second filter operator to carry out edge detection on the image to be detected in the vertical direction, and determining a second gradient image corresponding to the image to be detected;
and determining the gradient image to be detected corresponding to the image to be detected by utilizing the first gradient image and the second gradient image.
CN201911243936.6A 2019-12-06 2019-12-06 Intelligent vehicle license plate identification method and device suitable for complex illumination environment Active CN111079744B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911243936.6A CN111079744B (en) 2019-12-06 2019-12-06 Intelligent vehicle license plate identification method and device suitable for complex illumination environment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911243936.6A CN111079744B (en) 2019-12-06 2019-12-06 Intelligent vehicle license plate identification method and device suitable for complex illumination environment

Publications (2)

Publication Number Publication Date
CN111079744A CN111079744A (en) 2020-04-28
CN111079744B true CN111079744B (en) 2020-09-01

Family

ID=70313210

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911243936.6A Active CN111079744B (en) 2019-12-06 2019-12-06 Intelligent vehicle license plate identification method and device suitable for complex illumination environment

Country Status (1)

Country Link
CN (1) CN111079744B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111915899A (en) * 2020-08-14 2020-11-10 中国工商银行股份有限公司 Parking space monitoring management method and system
CN113723408B (en) * 2021-11-02 2022-02-25 上海仙工智能科技有限公司 License plate recognition method and system and readable storage medium

Citations (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7734097B1 (en) * 2006-08-01 2010-06-08 Mitsubishi Electric Research Laboratories, Inc. Detecting objects in images with covariance matrices
CN102214290A (en) * 2010-04-12 2011-10-12 无锡科利德斯科技有限公司 License plate positioning method and license plate positioning template training method
CN102968646A (en) * 2012-10-25 2013-03-13 华中科技大学 Plate number detecting method based on machine learning
CN103530640A (en) * 2013-11-07 2014-01-22 沈阳聚德视频技术有限公司 Unlicensed vehicle detection method based on AdaBoost and SVM (support vector machine)
CN104820831A (en) * 2015-05-13 2015-08-05 沈阳聚德视频技术有限公司 Front vehicle face identification method based on AdaBoost license plate location
CN106355180A (en) * 2016-09-07 2017-01-25 武汉安可威视科技有限公司 Method for positioning license plates on basis of combination of color and edge features
CN106355179A (en) * 2016-08-30 2017-01-25 韦嘉志 License plate location method and system
KR101715273B1 (en) * 2016-10-05 2017-03-10 한밭대학교 산학협력단 360 IP Multi license plate recognition system and method using production high-resolution 360all-round IP camera
CN107067015A (en) * 2016-10-27 2017-08-18 武汉烽火众智数字技术有限责任公司 A kind of vehicle checking method and device based on multiple features deep learning
CN107203754A (en) * 2017-05-26 2017-09-26 北京邮电大学 A kind of license plate locating method and device based on deep learning
CN107609555A (en) * 2017-09-15 2018-01-19 北京文安智能技术股份有限公司 Detection method of license plate, model recognizing method and relevant apparatus using it
CN108268867A (en) * 2016-12-30 2018-07-10 杭州海康威视数字技术股份有限公司 A kind of license plate locating method and device
CN109670498A (en) * 2018-11-10 2019-04-23 江苏网进科技股份有限公司 A kind of license plate locating method
CN109685723A (en) * 2018-11-05 2019-04-26 建湖云飞数据科技有限公司 A kind of multimedia data information processing method
CN109711413A (en) * 2018-12-30 2019-05-03 陕西师范大学 Image, semantic dividing method based on deep learning
CN109840557A (en) * 2019-01-24 2019-06-04 深圳爱莫科技有限公司 Image-recognizing method and device
CN110020651A (en) * 2019-04-19 2019-07-16 福州大学 Car plate detection localization method based on deep learning network
WO2019174405A1 (en) * 2018-03-14 2019-09-19 台达电子工业股份有限公司 License plate identification method and system thereof

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101999084B1 (en) * 2019-02-15 2019-07-11 주식회사 넥스파시스템 Determining small car and recognizing vehicle number system using vehicle feature vectors classification
CN110210475B (en) * 2019-05-06 2021-05-18 浙江大学 License plate character image segmentation method based on non-binarization and edge detection
CN110414507B (en) * 2019-07-11 2022-07-26 深圳智优停科技有限公司 License plate recognition method and device, computer equipment and storage medium
CN110443245B (en) * 2019-08-14 2022-02-15 上海世茂物联网科技有限公司 License plate region positioning method, device and equipment in non-limited scene
CN110458168A (en) * 2019-08-29 2019-11-15 上海眼控科技股份有限公司 Processing method, device, computer equipment and the storage medium of vehicle detection report

Patent Citations (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7734097B1 (en) * 2006-08-01 2010-06-08 Mitsubishi Electric Research Laboratories, Inc. Detecting objects in images with covariance matrices
CN102214290A (en) * 2010-04-12 2011-10-12 无锡科利德斯科技有限公司 License plate positioning method and license plate positioning template training method
CN102968646A (en) * 2012-10-25 2013-03-13 华中科技大学 Plate number detecting method based on machine learning
CN103530640A (en) * 2013-11-07 2014-01-22 沈阳聚德视频技术有限公司 Unlicensed vehicle detection method based on AdaBoost and SVM (support vector machine)
CN104820831A (en) * 2015-05-13 2015-08-05 沈阳聚德视频技术有限公司 Front vehicle face identification method based on AdaBoost license plate location
CN106355179A (en) * 2016-08-30 2017-01-25 韦嘉志 License plate location method and system
CN106355180A (en) * 2016-09-07 2017-01-25 武汉安可威视科技有限公司 Method for positioning license plates on basis of combination of color and edge features
KR101715273B1 (en) * 2016-10-05 2017-03-10 한밭대학교 산학협력단 360 IP Multi license plate recognition system and method using production high-resolution 360all-round IP camera
CN107067015A (en) * 2016-10-27 2017-08-18 武汉烽火众智数字技术有限责任公司 A kind of vehicle checking method and device based on multiple features deep learning
CN108268867A (en) * 2016-12-30 2018-07-10 杭州海康威视数字技术股份有限公司 A kind of license plate locating method and device
CN107203754A (en) * 2017-05-26 2017-09-26 北京邮电大学 A kind of license plate locating method and device based on deep learning
CN107609555A (en) * 2017-09-15 2018-01-19 北京文安智能技术股份有限公司 Detection method of license plate, model recognizing method and relevant apparatus using it
WO2019174405A1 (en) * 2018-03-14 2019-09-19 台达电子工业股份有限公司 License plate identification method and system thereof
CN109685723A (en) * 2018-11-05 2019-04-26 建湖云飞数据科技有限公司 A kind of multimedia data information processing method
CN109670498A (en) * 2018-11-10 2019-04-23 江苏网进科技股份有限公司 A kind of license plate locating method
CN109711413A (en) * 2018-12-30 2019-05-03 陕西师范大学 Image, semantic dividing method based on deep learning
CN109840557A (en) * 2019-01-24 2019-06-04 深圳爱莫科技有限公司 Image-recognizing method and device
CN110020651A (en) * 2019-04-19 2019-07-16 福州大学 Car plate detection localization method based on deep learning network

Also Published As

Publication number Publication date
CN111079744A (en) 2020-04-28

Similar Documents

Publication Publication Date Title
CN112380952B (en) Power equipment infrared image real-time detection and identification method based on artificial intelligence
CN106960195B (en) Crowd counting method and device based on deep learning
CN108898047B (en) Pedestrian detection method and system based on blocking and shielding perception
US20190188528A1 (en) Text detection method and apparatus, and storage medium
CN108121991B (en) Deep learning ship target detection method based on edge candidate region extraction
CN107657249A (en) Method, apparatus, storage medium and the processor that Analysis On Multi-scale Features pedestrian identifies again
CN106599789A (en) Video class identification method and device, data processing device and electronic device
CN107038416B (en) Pedestrian detection method based on binary image improved HOG characteristics
CN108805016B (en) Head and shoulder area detection method and device
CN104834933A (en) Method and device for detecting salient region of image
CN111008961B (en) Transmission line equipment defect detection method and system, equipment and medium thereof
CN110659550A (en) Traffic sign recognition method, traffic sign recognition device, computer equipment and storage medium
CN111079744B (en) Intelligent vehicle license plate identification method and device suitable for complex illumination environment
CN107609555A (en) Detection method of license plate, model recognizing method and relevant apparatus using it
CN110689043A (en) Vehicle fine granularity identification method and device based on multiple attention mechanism
CN111091023A (en) Vehicle detection method and device and electronic equipment
CN106780727A (en) A kind of headstock detection model method for reconstructing and device
CN109389105A (en) A kind of iris detection and viewpoint classification method based on multitask
CN111882620A (en) Road drivable area segmentation method based on multi-scale information
CN109993134A (en) A kind of intersection vehicle checking method based on HOG and SVM classifier
CN115082922A (en) Water meter digital picture processing method and system based on deep learning
CN114926791A (en) Method and device for detecting abnormal lane change of vehicles at intersection, storage medium and electronic equipment
CN112597995B (en) License plate detection model training method, device, equipment and medium
CN106778777A (en) A kind of vehicle match method and system
CN106446929B (en) Type of vehicle detection method based on edge gradient potential energy

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
PE01 Entry into force of the registration of the contract for pledge of patent right
PE01 Entry into force of the registration of the contract for pledge of patent right

Denomination of invention: Intelligent vehicle license plate recognition method and device suitable for complex lighting environment

Effective date of registration: 20211216

Granted publication date: 20200901

Pledgee: Yantai financing guarantee Group Co.,Ltd.

Pledgor: LUDONG University

Registration number: Y2021980015152

PC01 Cancellation of the registration of the contract for pledge of patent right
PC01 Cancellation of the registration of the contract for pledge of patent right

Date of cancellation: 20220317

Granted publication date: 20200901

Pledgee: Yantai financing guarantee Group Co.,Ltd.

Pledgor: LUDONG University

Registration number: Y2021980015152