CN116403200A

CN116403200A - License plate real-time identification system based on hardware acceleration

Info

Publication number: CN116403200A
Application number: CN202310221631.5A
Authority: CN
Inventors: 李静磊; 鲁玥; 丛日原; 杨清海
Original assignee: Xidian University
Current assignee: Xidian University
Priority date: 2023-03-08
Filing date: 2023-03-08
Publication date: 2023-07-07

Abstract

The invention relates to a license plate real-time identification system based on hardware acceleration, belongs to the technical field of computer vision, and is used for solving the problems that in a complex environment such as a foggy weather, etc., for a lens with a large view field, the data volume required to be processed in the license plate image identification process is extremely huge, so that the detection time is long, and the real-time detection of high-frame-rate and high-resolution images is difficult to realize. After image defogging processing and license plate number region positioning are carried out by adopting the FPGA processing board, the system provided by the invention carries out number recognition by the license plate recognition module in the upper computer software, so that not only is the license plate number recognition precision improved, but also the real-time processing speed of image data is improved.

Description

License plate real-time identification system based on hardware acceleration

Technical Field

The invention relates to the technical field of computer vision, in particular to a license plate real-time identification system based on hardware acceleration.

Background

License plate recognition and detection are important research subjects in the field of intelligent transportation. Today, vehicles become a part of our lives, and many problems occur while greatly facilitating travel. License plate recognition plays an important role in traffic detection, congestion control, illegal monitoring, suspicious vehicle detection and the like. However, in a foggy weather, the visibility is very low, the loss of vehicle detail information is serious, which brings great difficulty to license plate recognition, and for a large-view lens, a license plate image only occupies a small part of the whole image, the license plate recognition is performed by using the whole image, the recognition precision is difficult to ensure, and the data volume to be processed is extremely huge in the recognition process, so that the detection is directly long, and the real-time detection of high-frame-rate and high-resolution images is difficult to realize. Aiming at the development trend of the computer vision field in the new era, a method which can adapt to the complex environment, large visual field, high frame rate and high resolution video license plate real-time detection is needed.

Disclosure of Invention

In complex environments such as foggy weather, for a lens with a large view field, a license plate image only occupies a small part of the whole image, and license plate recognition is carried out by using the whole image, so that the problem that the detection time is long and the real-time detection of high-frame-rate and high-resolution images is difficult to realize due to the fact that the processed data volume is extremely huge in the recognition process. In order to solve the problems, the invention provides a license plate real-time identification system based on hardware acceleration, which adopts an FPGA processing board to carry out image defogging processing and license plate number region positioning, and then carries out number identification by a license plate identification module in upper computer software, so that not only the license plate number identification precision is improved, but also the real-time processing speed of image data is improved.

The technical scheme of the invention is as follows: a license plate real-time identification system based on hardware acceleration comprises an FPGA processing board and a license plate identification module;

the FPGA processing board performs hardware defogging processing on the image by using the defogging module according to the video image input source, and transmits the defogged image to the license plate positioning module for license plate position preselection;

transmitting the position coordinates and the region image with the coordinates as the center to a license plate recognition module in the upper computer software, and recognizing license plate characters through the license plate recognition module.

In the technical scheme, the FPGA processing board comprises 1 HDMI input interface, 2 optical fiber input/output interfaces, 1 HDMI output interface, 1 gigabit Ethernet output interface, 1 FPGA processing chip and 4 DDR3 particles;

wherein:

the HDMI input interface is used for connecting an HDMI wire and inputting computer equipment projection;

the optical fiber input/output interface is connected with the optical fiber line to the camera end, the camera video image is input by the input interface, and the output interface sends instructions to the camera, for example: adjusting exposure, resolution and other camera parameters;

the HDMI output interface is used for outputting the defogged video image to a display to check defogging effects;

the Ethernet output interface is used for uploading the position coordinates and the specified area images to the upper computer software, and identifying license plate characters by utilizing a license plate identification module in the upper computer software;

the FPGA processing chip is used for completing defogging processing and calculation required by license plate position preselection;

DDR3 particles, which are used to buffer images and computation data.

In the above technical solution, the specified area image specifically includes: and the geometric center of the positioned license plate region is taken as a coordinate origin, a Cartesian coordinate system is taken as a reference coordinate system, rectangular regions surrounded by X1 and Y1 pixels are respectively expanded towards the positive and negative directions of the X axis and the positive and negative directions of the Y axis, and the set values of the X1 and Y1 are related to the computing capacity of the FPGA processing chip.

In the above technical solution, the defogging module includes the following processes:

splitting each pixel of an input image into 3 pieces of 8-bit data, comparing the 3 pieces of data by using a comparator, taking the minimum value as a pixel value of a dark channel image of the input video image, and outputting according to an original image time sequence;

constructing a 9 multiplied by 9 filter window by using BRAM resources as a basic template, traversing the basic template by using a 5 multiplied by 5 sub-window, calculating the pixel mean value of each sub-region, calculating the difference between the mean value and the pixel value of a central point, and selecting the sub-region with the minimum difference value for minimum value filtering to obtain a processed dark channel diagram for calculating the transmissivity;

accumulating the dark channel pixel values, storing the accumulated result in each address of the BRAM, wherein each address corresponds to a histogram statistical result of one gray value, estimating the mean value of about the first 0.1% pixel of the dark channel map of the previous frame as a threshold value through the statistical histogram, buffering the data of the previous frame and the current frame in the DDR unit, and carrying out linear adjustment on three channels of the current frame according to the difference value between the pixel and the threshold value to estimate the atmospheric light value;

and obtaining defogging images by using a foggy imaging model based on the transmissivity and the atmospheric light value.

In the above technical scheme, the license plate positioning module comprises the following processes:

performing color space conversion on the defogging RGB image, and converting the RGB image into a YCbCr color space;

extracting gray images from Cb and Cr components of the target information by utilizing FPGA parallel processing, so as to obtain a gray image with enhanced target information;

and carrying out binarization segmentation on the gray level image enhanced by the target information, and then carrying out morphological treatment to complete the reduction of the geometric dimension of the target area and the filtering of speckle interference.

In the above technical solution, the target information enhanced gray image obtaining step includes:

traversing the Cr component gray level image pixel by pixel point according to a line field time sequence, and respectively setting two registers to store a target pixel range value and a pixel mean value;

after traversing, taking the difference value between the pixel mean value and the target pixel range value in the Cr component as a threshold C, comparing the threshold C with the Cr component to obtain a processed Cr component gray scale map, and storing the processed Cr component gray scale map in the DDR;

and reading the Cb component gray level image and the processed Cr component gray level image in the same frame from the DDR buffer unit, and performing difference operation pixel by using DSP resources to obtain the gray level image with enhanced target information.

In the above technical solution, the morphological processing includes:

firstly, performing expansion processing on a binary-segmented gray level image, communicating holes of a license plate region, and constructing a line cache structure by using BRAM; after three data caching is completed, reading the image to a register array with a required size to construct a convolution unit, wherein the size of a convolution kernel constructed by the register is 9 multiplied by 9;

then, two corrosions are carried out, the corrosions are respectively 9×9 and 3×3, and the reduction of the geometric dimension of the target area and the filtering of speckle interference are completed.

In the technical scheme, after binarization segmentation is carried out, an improved bounding box target detection algorithm is adopted to identify a binarized image containing a license plate region:

firstly, defining a BRAM memory, carrying out area accumulation on lines with overlapping areas in DDR read-out images in a progressive scanning mode, sequentially storing accumulation results in a minimum storage unit in BRAM addresses, marking a closed area in a searching process, placing the marking results in a register, recording the area of the closed area and coordinate information of vertexes at two sides, and sending the areas of all the closed areas into a comparator for comparison at the end of image searching, wherein the area range and the aspect ratio are set as filtering conditions, and the conditions refer to the aspect ratio of actual license plates, so that the closed area with the largest probability is screened out as a target license plate;

and then, carrying out output smoothing on the coordinates of the target license plate, wherein the smoothing operation is to set a fixed limit difference value between the detected two results, carry out subtraction operation on the two results through a DSP, send the subtraction result and the limit difference value to a comparator, adopt the original result if the result is within a set limit range, and assign the limit difference value to the final result if the result exceeds the range.

In the above technical solution, the filtering condition is stored in the ROM resource of the FPGA.

In the above technical scheme, the license plate recognition module comprises the following processes:

performing coordinate offset processing on license plate coordinates based on the received position coordinates to obtain actual position information;

constructing rectangular structural elements for the license plate image, sequentially corroding and expanding the binarized image, and subtracting the constructed structural elements from the processed binarized image to obtain an image with redundant parts removed;

traversing each pixel point of the image with the redundant part removed to obtain the number of target pixels of each column of the image;

step S: comparing the number of target pixels in each column with a set threshold A, and judging whether the column is a character or not;

if the character is not used, comparing the number of continuous columns with the number of target pixels being larger than the set threshold A with the set threshold B to update character position information;

if the character is the character, accumulating the character width, and when the character width is not accumulated any more, ending the positioning of the current character position and starting to position the next character;

returning to the step S until all 8 character positions on the license plate are positioned;

and performing character recognition by using template matching, creating an affine transformation target image, calculating an affine transformation matrix, performing affine transformation operation on the loaded graph, comparing with a character library, and outputting the recognized license plate number.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort to a person skilled in the art.

Fig. 1 is a schematic diagram of a license plate real-time recognition system in a complex environment based on hardware acceleration according to an embodiment of the present invention;

fig. 2 is a schematic workflow diagram of a license plate real-time recognition system in a complex environment based on hardware acceleration according to an embodiment of the present invention;

fig. 3 is a schematic flow chart of a license plate region positioning algorithm under a complex environment by using an FPGA processing board provided by an embodiment of the present invention;

fig. 4 is a schematic diagram of network data transmission content according to an embodiment of the present invention;

fig. 5 is a schematic diagram of an effect of upper computer software according to an embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application.

The license plate real-time identification system based on the hardware acceleration in the complex environment is shown in figure 1, and mainly comprises a video image source, an FPGA processing board, computer equipment, a display and upper computer software. The video image source is divided into a camera image picture, a computer equipment projection picture and a test image generated in the FPGA processing board, and mainly provides video input for the FPGA processing board.

As shown in fig. 2, a plurality of modules and input/output interfaces are integrated on the FPGA processing board, in the working process, firstly, the input state of the video source is judged, and the input video source is selectively input in a mode that the priorities of the camera input, the HDMI projection and the test image are sequentially decreased. The camera image is transmitted to the FPGA processing board from the optical fiber interface through the optical fiber, the optical fiber transmission bandwidth is 5Gbps, the bidirectional data transmission can be carried out, and the camera parameters of the camera end can be configured remotely while receiving the video picture. HDMI projection of the computer equipment is connected with an HDMI input interface through an HDMI line and is transmitted to the FPGA processing board. The test images are connected through the internal wiring of the FPGA without external physical connection. The video image source has an image size of 4096×2160 and a frame rate of about 100 frames/second.

After the FPGA finishes the selection of an image input source, the image is subjected to hardware defogging algorithm realization, the defogged clear image is subjected to HDMI output display and is sent to a next-stage algorithm module, the license plate positioning module realizes the preselection of the license plate position on the FPGA, the position coordinates and the area image with the coordinates as the center are uploaded to a computer through a network, and the software is used for completing the subsequent license plate character recognition and user operation interface work.

The FPGA processing board is connected with the video image source and used for defogging an input image or video image in real time. In order to improve the recognition speed, the method adopts a dark channel image defogging algorithm and combines an FPGA processing board to carry out targeted algorithm improvement design, and comprises the following steps:

(1) Obtaining dark channel images from input video images

Each pixel point of the input image is split into 3 8bit data, which respectively represent R, G, B three channels. Comparing the three data by using a comparator to obtain a minimum value; and the minimum value is used as the pixel value of the dark channel image and is output according to the original image time sequence.

(2) Estimating a transmittance value t (x) and an atmospheric light value A from a dark channel image

(2.1) transmittance value t (x)

And filtering the input video image by using a double-scale minimum difference filtering algorithm, and constructing a 9 multiplied by 9 filtering window by using BRAM resources. And traversing the basic template by using the filtering window as the basic template and using a sub-window of 5 multiplied by 5.

The traversing method mainly completes basic mathematical operation by calling DSP resources in the FPGA. Firstly, calculating the pixel mean value of each sub-area, then calculating the difference between the mean value and the pixel value of the central point, and then selecting the sub-area with the smallest difference value for minimum value filtering to obtain a refined dark channel diagram, wherein the refined dark channel is used for calculating the transmissivity, and the transmissivity can be obtained according to a general foggy day imaging model.

(2.2) estimating the atmospheric light value A

And performing accumulation operation on the pixel values of the dark channels, wherein the accumulation results are stored in each address of the BRAM, and each address corresponds to a histogram statistical result of one gray value. Taking the mean value of about the previous 0.1% pixel of the dark channel diagram of the previous frame as a threshold value through a statistical histogram, and caching the previous frame and the current frame data in a DDR unit; and then, carrying out linear adjustment on three channels of the current frame according to the difference value between the pixel and the threshold value to estimate the atmospheric light value A.

(3) Substituting the estimated parameters into a foggy day imaging model to obtain defogged images.

The FPGA processing board searches the defogged image and determines the coordinate position of the license plate in the whole image in the frame of image.

The license plate positioning algorithm workflow is shown in fig. 3, and is implemented as follows:

first, the RGB image processed by the defogging algorithm is converted into YCbCr color space by color space conversion.

And secondly, the parallel processing capability of the FPGA is utilized, and meanwhile, the Cb and Cr components of the FPGA are extracted to obtain gray images, so that the processing speed can be doubled compared with software implementation.

Then, traversing the Cr component gray level image pixel by pixel according to the row field time sequence, and respectively setting two registers to store the target pixel range value and the pixel mean value. The range of the image maximum gradation value and the 0.85 times the gradation value is defined as a target pixel range value since the range in which the target information is located contains the pixel minimum value. After the traversal is finished, the design idea of background suppression is adopted, and nonlinear compensation is carried out on the minimum value obtained from the Cr component gray level image. And taking the difference value between the pixel mean value and the target pixel range value as a threshold value C, comparing the threshold value C with Cr components to obtain a processed Cr component gray map, and storing the processed Cr component gray map in the DDR. If the Cr component is larger than or equal to the threshold C, taking the threshold C as a processed Cr component gray scale map; and if the Cr component is smaller than the predetermined value, taking the Cr component as a processed Cr component gray pattern. And then reading the Cb component gray level image and the processed Cr component gray level image in the same frame from the DDR buffer unit, and performing difference operation pixel by using DSP resources to obtain a gray level image with enhanced target information, thereby realizing the purpose of inhibiting background interference caused by the vehicle body color and achieving the purpose of inhibiting redundant noise.

And performing binarization segmentation on the gray level image through a self-adaptive binarization module, and performing morphological processing. The method comprises the following steps: firstly, carrying out expansion treatment on the binary segmented gray level image, communicating holes of a license plate area, and constructing a line cache structure by using BRAM; after three-line data caching is completed, reading the image to a register array with a required size to construct a convolution unit, wherein the size of a convolution kernel constructed by the register is 9 multiplied by 9, then, performing corrosion twice, and the sizes of the corrosion convolution kernels are 9 multiplied by 9 and 3 multiplied by 3 respectively, thereby completing the restoration of the geometric size of a target area and the filtering of speckle interference.

Next, the binarized image containing the license plate region is identified, and an improved bounding box target detection algorithm is adopted. Specifically, a BRAM memory is defined first, then area accumulation is carried out on lines with overlapping areas in DDR read-out images in a progressive scanning mode, and accumulation results are sequentially stored in minimum memory cells in BRAM addresses. In the searching process, the closed area is marked, the marking result is placed in a register, and the area of the closed area and the coordinate information of vertexes at two sides are recorded. At the end of the image search, all the closed area areas are fed into a comparator for comparison, and the area range and aspect ratio are set as filtering conditions. The condition refers to the actual license plate aspect ratio, and specific data is stored in on-chip ROM resources. And taking the closed region with the highest probability as a target license plate, and carrying out smoothing treatment on the coordinates of the target license plate, wherein the smoothing step length is 50 pixel points. The smoothing operation is to set a fixed limit difference between the detected 2 results, perform subtraction operation on the two results by the DSP, and send the subtraction result and the limit difference to the comparator. If the comparison result is within the set limit range, adopting the detected coordinate result of the license plate; if the coordinate result exceeds the range, the limit difference value is assigned to the coordinate result. And then, reading out RGB images of corresponding frames from the DDR3 cache region, taking the license plate coordinate information and the geometric center of the license plate region as the origin of coordinates through a network port, taking a Cartesian coordinate system as a reference coordinate system, expanding a rectangular region surrounded by 370 and 300 pixels towards the positive and negative directions of the X axis and the positive and negative directions of the Y axis respectively, and uploading the rectangular region to upper computer software of computer equipment. The transmission format adopts a custom coding mode, an FPGA logic unit is used for constructing an Ethernet frame format, effective data is filled into a data segment, and after the data segment is sent to an MAC layer with an RGMII interface, the on-board PHY chip completes subsequent network communication, as shown in figure 4.

After receiving video stream data sent by the FPGA processing board through the Ethernet interface, the upper computer software simultaneously performs the following two works:

1. and displaying the received video pictures in the video display area of the upper computer software in real time.

2. After receiving the license plate coordinate information, the upper computer software performs coordinate offset processing on the license plate coordinate according to the difference of the license plate coordinate in the FPGA processing board and the uploading picture reference coordinate system, so as to obtain the actual position information of the license plate coordinate in the uploading video image.

After license plate coordinate information is obtained, binarizing the license plate image, and expanding the license plate image by the following operations: and constructing structural elements in rectangular shapes, sequentially corroding and expanding the binary images, and subtracting the constructed structural elements from the processed binary images to obtain a graph with redundant parts removed. And further positioning the characters, traversing each pixel point of the license plate number image, and obtaining the number of target pixels of each column of the image. Comparing the number of the target pixels in each column with a set threshold A to judge whether the column is a character, and if not, comparing the number of the continuous columns with the number of the target pixels being greater than the threshold A with a threshold B to update character position information; if the character is the character, accumulating the character width, and when the character width is not accumulated any more, finishing the positioning of the current character position, starting to position the next character, and repeating the operation steps of positioning the character until all the 8 character positions on the license plate are positioned. Finally, character recognition is carried out by using template matching, an affine transformation target image is created, an affine transformation matrix is calculated, affine transformation operation is carried out on the loaded graph, and then the affine transformation target image is compared with a character library, and the recognized license plate characters are output to a license plate character display area of upper computer software, as shown in fig. 5.

Although the embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to the above-described specific embodiments and application fields, and the above-described specific embodiments are merely illustrative, and not restrictive. Those skilled in the art, having the benefit of this disclosure, may effect numerous forms of the invention without departing from the scope of the invention as claimed.

Claims

1. The license plate real-time identification system based on hardware acceleration is characterized by comprising an FPGA processing board and a license plate identification module;

2. The method of claim 1, wherein the FPGA processing board comprises 1 HDMI input interface, 2 fiber input/output interfaces, 1 HDMI output interface, 1 gigabit ethernet output interface, 1 FPGA processing chip, 4 DDR3 particles;

wherein:

an optical fiber input/output interface connected to the camera end through an optical fiber line;

the Ethernet output interface is used for uploading the position coordinates and the specified area images to the upper computer, and identifying license plate characters by utilizing a license plate identification module in the upper computer;

DDR3 particles, which are used to buffer images and computation data.

3. The system according to claim 2, wherein the specified area image is specifically: and the geometric center of the positioned license plate region is taken as a coordinate origin, a Cartesian coordinate system is taken as a reference coordinate system, rectangular regions surrounded by X1 and Y1 pixels are respectively expanded towards the positive and negative directions of the X axis and the positive and negative directions of the Y axis, and the set values of the X1 and Y1 are related to the computing capacity of the FPGA processing chip.

4. The system of claim 1, wherein the defogging module comprises the following:

5. The system of claim 1, wherein the license plate location module comprises the following:

6. The system of claim 5, wherein the target information enhanced gray scale image acquisition step comprises:

traversing pixel points of the Cr component gray level image according to a line field time sequence, and respectively setting two registers to store a target pixel range value and a pixel mean value;

7. The system of claim 5, wherein the morphological processing comprises:

8. The system of claim 5, wherein the improved bounding box object detection algorithm is used to identify the binary image containing license plate regions after the binary segmentation:

9. The system of claim 8, wherein the filter criteria is stored in a ROM resource of the FPGA.

10. The system of claim 1, wherein the license plate recognition module comprises the following: