CN116403200A - License plate real-time identification system based on hardware acceleration - Google Patents
License plate real-time identification system based on hardware acceleration Download PDFInfo
- Publication number
- CN116403200A CN116403200A CN202310221631.5A CN202310221631A CN116403200A CN 116403200 A CN116403200 A CN 116403200A CN 202310221631 A CN202310221631 A CN 202310221631A CN 116403200 A CN116403200 A CN 116403200A
- Authority
- CN
- China
- Prior art keywords
- image
- license plate
- pixel
- value
- character
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000001133 acceleration Effects 0.000 title claims abstract description 9
- 238000012545 processing Methods 0.000 claims abstract description 47
- 238000000034 method Methods 0.000 claims abstract description 11
- 238000001514 detection method Methods 0.000 claims abstract description 10
- 238000001914 filtration Methods 0.000 claims description 11
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 9
- 230000009466 transformation Effects 0.000 claims description 9
- 239000013307 optical fiber Substances 0.000 claims description 8
- 238000009825 accumulation Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 claims description 5
- 238000009499 grossing Methods 0.000 claims description 5
- 238000003384 imaging method Methods 0.000 claims description 4
- 230000000877 morphologic effect Effects 0.000 claims description 4
- 239000002245 particle Substances 0.000 claims description 4
- 230000011218 segmentation Effects 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000005260 corrosion Methods 0.000 claims description 3
- 230000007797 corrosion Effects 0.000 claims description 3
- 230000000694 effects Effects 0.000 claims description 3
- 239000011159 matrix material Substances 0.000 claims description 3
- 230000000750 progressive effect Effects 0.000 claims description 2
- 230000003139 buffering effect Effects 0.000 claims 1
- 238000006243 chemical reaction Methods 0.000 claims 1
- 239000000835 fiber Substances 0.000 claims 1
- 238000011897 real-time detection Methods 0.000 abstract description 3
- 238000002834 transmittance Methods 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 4
- 238000012360 testing method Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 2
- 230000003628 erosive effect Effects 0.000 description 2
- 230000001629 suppression Effects 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/62—Text, e.g. of license plates, overlay texts or captions on TV images
- G06V20/625—License plates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/20—Image enhancement or restoration using local operators
- G06T5/30—Erosion or dilatation, e.g. thinning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/187—Segmentation; Edge detection involving region growing; involving region merging; involving connected component labelling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Multimedia (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Image Processing (AREA)
Abstract
Description
技术领域technical field
本发明涉及计算机视觉技术领域,尤其涉及一种基于硬件加速的车牌实时识别系统。The invention relates to the technical field of computer vision, in particular to a real-time license plate recognition system based on hardware acceleration.
背景技术Background technique
车牌识别检测是智能交通领域中的重要研究课题。现如今,车辆已成为我们生活的一部分,在极大得方便出行的同时,也出现了很多问题。车牌识别在交通检测,拥堵控制,违规监控,可疑车辆检测等方面有着重要的作用。但是,在大雾天气下,能见度很低,车辆细节信息损失严重,这给车牌识别带来极大的困难,而且对于大视场的镜头,车牌图像只占整幅图像的一小部分,使用一整幅图像进行车牌识别,识别精度很难保证,在识别的过程中,需要处理的数据量也极为庞大,这就直接导致检测耗时长,难以实现高帧率、高分辨率图像的实时检测。着眼新时代计算机视觉领域的发展趋势,亟需一种能适应复杂环境、大视场、高帧率、高分辨率视频车牌实时检测的方法。License plate recognition and detection is an important research topic in the field of intelligent transportation. Nowadays, vehicles have become a part of our lives. While it is very convenient to travel, there are also many problems. License plate recognition plays an important role in traffic detection, congestion control, violation monitoring, suspicious vehicle detection, etc. However, in foggy weather, the visibility is very low, and the detailed information of the vehicle is seriously lost, which brings great difficulties to the license plate recognition. Moreover, for a lens with a large field of view, the license plate image only occupies a small part of the entire image. Using It is difficult to guarantee the recognition accuracy of a whole image for license plate recognition. In the process of recognition, the amount of data that needs to be processed is also extremely large, which directly leads to long detection time, and it is difficult to realize real-time detection of high frame rate and high resolution images. . Focusing on the development trend of the field of computer vision in the new era, there is an urgent need for a real-time license plate detection method that can adapt to complex environments, large field of view, high frame rate, and high-resolution video.
发明内容Contents of the invention
在大雾天气等复杂环境,对于大视场的镜头,车牌图像只占整幅图像的一小部分,使用一整幅图像进行车牌识别,在识别的过程中,存在处理的数据量也极为庞大,导致检测耗时长,难以实现高帧率、高分辨率图像实时检测的问题。针对前述问题,本发明提出一种基于硬件加速的车牌实时识别系统,采用FPGA处理板进行图像去雾处理和车牌号码区域定位后,在由上位机软件中的车牌识别模块进行号码识别,不仅提高车牌号码识别精度,而且提高图像数据实时处理速度。In complex environments such as foggy weather, for a lens with a large field of view, the license plate image only accounts for a small part of the entire image, and the entire image is used for license plate recognition. In the process of recognition, the amount of data processed is also extremely large. , resulting in a long time-consuming detection, and it is difficult to realize real-time detection of high frame rate and high-resolution images. For the foregoing problems, the present invention proposes a real-time license plate recognition system based on hardware acceleration. After using an FPGA processing board to perform image defogging processing and license plate number area positioning, the license plate recognition module in the upper computer software performs number recognition, which not only improves The recognition accuracy of the license plate number is improved, and the real-time processing speed of image data is improved.
本发明的技术方案为:一种基于硬件加速的车牌实时识别系统,所述系统包括FPGA处理板、车牌识别模块;The technical scheme of the present invention is: a kind of license plate real-time recognition system based on hardware acceleration, described system comprises FPGA processing board, license plate recognition module;
所述FPGA处理板根据视频图像输入源,使用去雾模块对图像做硬件去雾处理,将去雾后的图像传输给车牌定位模块进行车牌位置预选;According to the video image input source, the FPGA processing board uses the defogging module to perform hardware defogging processing on the image, and transmits the defogged image to the license plate positioning module to pre-select the license plate position;
将位置坐标与坐标为中心的区域图像传输给上位机软件中的车牌识别模块,通过车牌识别模块对车牌字符进行识别。The location coordinates and the area image centered on the coordinates are transmitted to the license plate recognition module in the host computer software, and the license plate characters are recognized by the license plate recognition module.
在上述技术方案中,FPGA处理板包括1个HDMI输入接口、2路光纤输入/输出接口、1路HDMI输出接口、1路千兆以太网输出接口、1颗FPGA处理芯片、4颗DDR3颗粒;In the above technical solution, the FPGA processing board includes 1 HDMI input interface, 2 optical fiber input/output interfaces, 1 HDMI output interface, 1 Gigabit Ethernet output interface, 1 FPGA processing chip, and 4 DDR3 particles;
其中:in:
HDMI输入接口,用于接HDMI线,用于输入电脑设备投影;HDMI input interface, used to connect HDMI cable, used to input computer equipment projection;
光纤输入/输出接口,通过接光纤线至相机端,由输入接口输入相机视频图像,由输出接口对相机发送指令,例如:调整曝光、分辨率等相机参数;Optical fiber input/output interface, connect the optical fiber cable to the camera end, input the video image of the camera through the input interface, and send instructions to the camera through the output interface, for example: adjust camera parameters such as exposure and resolution;
HDMI输出接口,将去雾后的视频图像输出至显示器以查看去雾效果;HDMI output interface, output the defogged video image to the monitor to view the defogged effect;
以太网输出接口,用于将位置坐标与指定区域图像上传至上位机软件,利用上位机软件中的车牌识别模块对车牌字符进行识别;The Ethernet output interface is used to upload the position coordinates and the image of the designated area to the host computer software, and use the license plate recognition module in the host computer software to recognize the license plate characters;
FPGA处理芯片,用于完成去雾处理和车牌位置预选所需的计算;FPGA processing chip, used to complete the calculation required for fog removal processing and license plate position preselection;
DDR3颗粒,用于缓存图像和计算数据。DDR3 particles, used to cache images and calculate data.
在上述技术方案中,指定区域图像具体为:以定位的车牌区域几何中心为坐标原点,以笛卡尔坐标系为参考坐标系,向X轴正负方向、Y轴正负方向各扩展X1、Y1个像素所包围成的矩形区域,X1、Y1的设定值与FPGA处理芯片的计算能力相关。In the above technical solution, the specified area image is specifically: take the geometric center of the license plate area as the origin of the coordinates, take the Cartesian coordinate system as the reference coordinate system, and expand X1 and Y1 in the positive and negative directions of the X axis and the positive and negative directions of the Y axis. A rectangular area surrounded by pixels, the setting values of X1 and Y1 are related to the computing power of the FPGA processing chip.
在上述技术方案中,去雾模块包括下述处理:In the above technical solution, the defogging module includes the following processing:
通过将输入图像的每一个像素拆分为3个8bit数据,使用比较器对3个数据作比较,将最小值作为输入视频图像的暗通道图像的像素值,并按照原图像时序输出;By splitting each pixel of the input image into three 8-bit data, using a comparator to compare the three data, the minimum value is used as the pixel value of the dark channel image of the input video image, and output according to the original image timing;
将输出的暗通道图像的像素值,采用BRAM资源构建9×9的滤波窗口作为基础模板,以5×5的子窗口对基础模板进行遍历,先计算每个子区域像素均值,再计算均值与中心点像素值之差,之后选择差值最小的子区域进行最小值滤波得到处理后的暗通道图,用来计算透射率;The pixel value of the output dark channel image is used to construct a 9×9 filter window as the basic template using BRAM resources, and the basic template is traversed with a 5×5 sub-window, and the pixel mean value of each sub-region is calculated first, and then the mean value and the center are calculated. Then select the sub-area with the smallest difference to perform minimum value filtering to obtain the processed dark channel map, which is used to calculate the transmittance;
对暗通道像素值做累加操作,累加结果存储在BRAM的各个地址中,每一地址对应一个灰度值的直方图统计结果,通过统计直方图来估计前一帧暗通道图约前0.1%像素的均值作为阈值,将前一帧与当前帧的数据缓存在DDR单元中,再在当前帧的三通道分别根据像素与阈值的差值进行线性调整来估计大气光值;Perform an accumulation operation on the dark channel pixel values, and the accumulation results are stored in each address of the BRAM. Each address corresponds to a histogram statistical result of a gray value, and the statistical histogram is used to estimate the top 0.1% pixels of the dark channel image of the previous frame. The average value of the pixel is used as the threshold, and the data of the previous frame and the current frame are cached in the DDR unit, and then the three channels of the current frame are linearly adjusted according to the difference between the pixel and the threshold to estimate the atmospheric light value;
基于透射率和大气光值,利用雾天成像模型,获得去雾图像。Based on the transmittance and atmospheric light value, the foggy imaging model is used to obtain the dehazed image.
在上述技术方案中,车牌定位模块包括下述处理:In the above technical solution, the license plate positioning module includes the following processing:
将去雾处理后的RGB图像进行色彩空间转换,转换到YCbCr彩色空间;Convert the RGB image after dehazing to the YCbCr color space;
利用FPGA并行处理,同时对其Cb和Cr分量提取灰度图像,进而得到目标信息增强的灰度图像;Use FPGA parallel processing to extract the grayscale image of its Cb and Cr components at the same time, and then obtain the grayscale image with enhanced target information;
对目标信息增强的灰度图像进行二值化分割,再进行形态学处理以完成目标区域几何尺寸的还原以及散斑状干扰的滤除。The grayscale image enhanced with target information is binarized and segmented, and then morphologically processed to restore the geometric size of the target area and filter out speckle interference.
在上述技术方案中,目标信息增强的灰度图像获取步骤包括:In the above technical solution, the grayscale image acquisition step of target information enhancement includes:
对Cr分量灰度图像逐像素点按照行场时序进行遍历,分别设置两个寄存器来存放目标像素范围值和像素均值;The Cr component grayscale image is traversed pixel by pixel according to the line and field timing, and two registers are respectively set to store the target pixel range value and pixel mean value;
在遍历结束后,在Cr分量中将像素均值和目标像素范围值的差值作为阈值C,将阈值C与Cr分量比较后得到处理后的Cr分量灰度图,并将处理后的Cr分量灰度图存放在DDR中;After the traversal, in the Cr component, the difference between the pixel mean value and the target pixel range value is used as the threshold C, and the threshold C is compared with the Cr component to obtain the processed Cr component grayscale image, and the processed Cr component grayscale The degree map is stored in DDR;
将同一帧下的Cb分量灰度图像与处理后的Cr分量灰度图从DDR缓存单元读出,使用DSP资源逐像素做差值运算,得到目标信息增强的灰度图像。Read out the Cb component grayscale image and the processed Cr component grayscale image in the same frame from the DDR cache unit, and use DSP resources to perform difference operation pixel by pixel to obtain a grayscale image with enhanced target information.
在上述技术方案中,形态学处理包括:In the above technical solution, the morphological treatment includes:
首先,对二值化分割的灰度图像进行膨胀处理,对车牌区域的孔洞进行连通,使用BRAM构建行缓存结构;在完成三行数据缓存后,将图像读出至所需尺度的寄存器阵列中构建卷积单元,由寄存器构建的卷积核大小为固定的9×9;First, expand the binary segmented grayscale image, connect the holes in the license plate area, and use BRAM to build a line cache structure; after completing the three-line data cache, read the image into the register array of the required scale Build a convolution unit, the size of the convolution kernel constructed by the register is fixed 9×9;
然后,进行两次腐蚀,腐蚀卷积核分别为9×9、3×3大小,完成目标区域几何尺寸的还原以及散斑状干扰的滤除。Then, two erosions are performed, and the erosion convolution kernels are respectively 9×9 and 3×3 in size to complete the reduction of the geometric size of the target area and the filtering of speckle interference.
在上述技术方案中,在进行二值化分割后,采用改进的包围盒目标检测算法对含有车牌区域的二值化图像进行识别:In the above technical solution, after the binary segmentation, the improved bounding box target detection algorithm is used to identify the binary image containing the license plate area:
首先,定义一个BRAM存储器,以逐行扫描的方式对DDR读出图像中有重叠区域的线条进行面积累加,累加结果依次存储在BRAM地址中的最小存储单元中,在搜索过程中对闭合区域进行标记,标记结果放置在寄存器中,并记录该闭合区域的面积、两侧顶点的坐标信息,在图像搜索结束时对所有闭合区域面积送进比较器中进行比较,并设置面积范围和长宽比例作为过滤条件,该条件参考实际车牌长宽比例,以此筛选出概率最大的闭合区域作为目标车牌;First, a BRAM memory is defined, and the lines with overlapping areas in the DDR readout image are accumulated in a row-by-row manner, and the accumulation results are stored in the smallest storage unit in the BRAM address in turn, and the closed area is searched during the search process. Marking, the marking result is placed in the register, and the area of the closed area and the coordinate information of the vertices on both sides are recorded. At the end of the image search, the area of all closed areas is sent to the comparator for comparison, and the area range and aspect ratio are set. As a filter condition, this condition refers to the actual license plate length and width ratio, so as to filter out the closed area with the highest probability as the target license plate;
然后,对目标车牌坐标进行输出平滑,平滑操作是对检测出的两次结果之间设定一固定极限差值,通过DSP对两次结果做减法运算,并将减法结果与极限差值送比较器,若结果在设定的限定范围内则采用原有结果,若超出范围则将极限差值赋值给最终结果。Then, smooth the output of the target license plate coordinates. The smoothing operation is to set a fixed limit difference between the two detected results, perform subtraction on the two results through DSP, and compare the subtraction result with the limit difference. If the result is within the set limit range, the original result will be used, and if it exceeds the range, the limit difference will be assigned to the final result.
在上述技术方案中,过滤条件存储在FPGA的ROM资源中。In the above technical solution, the filter condition is stored in the ROM resource of the FPGA.
在上述技术方案中,车牌识别模块包括下述处理:In the above technical solution, the license plate recognition module includes the following processing:
基于接受的位置坐标,对车牌坐标进行坐标偏移处理,得到实际位置信息;Based on the accepted position coordinates, the coordinate offset processing is performed on the license plate coordinates to obtain the actual position information;
对车牌图像构建矩形形状的结构元素,对二值化图像依次进行腐蚀、膨胀,将构建的结构元素和处理后的二值化图像相减,以此得到去除冗余部分的图像;Construct a rectangular structural element on the license plate image, corrode and expand the binarized image in turn, and subtract the constructed structural element from the processed binarized image to obtain an image with redundant parts removed;
对去除冗余部分的图像,遍历每一个像素点,得到该图像的每一列的目标像素个数;For the image from which the redundant part is removed, traverse each pixel to obtain the number of target pixels in each column of the image;
步骤S:将每一列的目标像素个数和设定阈值A作比较,判断该列是否为字符;Step S: compare the number of target pixels in each column with the set threshold A to determine whether the column is a character;
若不为字符,将目标像素个数大于设定阈值A的连续列数和设定阈值B作比较,用来更新字符位置信息;If it is not a character, compare the number of consecutive columns with the number of target pixels greater than the set threshold A with the set threshold B to update the character position information;
若为字符,将字符宽度进行累加,当字符宽度不再累加,则当前字符位置定位结束,开始定位下一个字符;If it is a character, the character width is accumulated. When the character width is no longer accumulated, the positioning of the current character position ends and the positioning of the next character begins;
返回步骤S,直到车牌上的8个字符位置全部定位结束;Return to step S until all 8 character positions on the license plate are positioned;
利用模板匹配进行字符识别,创建仿射变换目标图像,计算仿射变换矩阵,对加载图形进行仿射变换操作,再与字符库作对比,输出识别的车牌号码。Use template matching for character recognition, create an affine transformation target image, calculate the affine transformation matrix, perform affine transformation operation on the loaded image, compare it with the character library, and output the recognized license plate number.
附图说明Description of drawings
为了更清楚地说明本申请实施例中的技术方案,下面将对实施例描述中所需要使用的附图作简单地介绍,显而易见地,下面描述中的附图仅仅是本申请的一些实施例,对于本领域普通技术人员来讲,在不付出创造性劳动性的前提下,还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.
图1、为本发明实施例提供的基于硬件加速的复杂环境下车牌实时识别系统示意图;Fig. 1, the schematic diagram of license plate real-time recognition system under the complex environment based on hardware acceleration provided by the embodiment of the present invention;
图2、为本发明实施例提供的基于硬件加速的复杂环境下车牌实时识别系统工作流程示意图;Fig. 2 is a schematic diagram of the workflow of the license plate real-time recognition system under the complex environment based on hardware acceleration provided by the embodiment of the present invention;
图3、为本发明实施例提供的FPGA处理板对复杂环境下车牌区域定位算法流程示意图;Fig. 3, FPGA processing board provided for the embodiment of the present invention is to the schematic flow chart of license plate area location algorithm under complex environment;
图4、为本发明实施例提供的网络数据传输内容的示意图;FIG. 4 is a schematic diagram of network data transmission content provided by an embodiment of the present invention;
图5、为本发明实施例提供的上位机软件效果示意图。Fig. 5 is a schematic diagram of the effect of the host computer software provided by the embodiment of the present invention.
具体实施方式Detailed ways
下面将结合本申请实施例中的附图,对本申请实施例中的技术方案进行清楚、完整地描述,显然,所描述的实施例仅仅是本申请一部分实施例,而不是全部的实施例。The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them.
基于硬件加速的复杂环境下车牌实时识别系统组成如图1,主要包括视频图像源、FPGA处理板、电脑设备、显示器和上位机软件。所述视频图像源,分为相机图像画面、电脑设备投影画面以及FPGA处理板内部生成的测试图像,主要为FPGA处理板提供视频输入。The composition of the real-time license plate recognition system based on hardware acceleration in a complex environment is shown in Figure 1, which mainly includes video image source, FPGA processing board, computer equipment, display and host computer software. The video image sources are divided into camera image screens, computer equipment projection screens and test images generated inside the FPGA processing board, mainly providing video input for the FPGA processing board.
如图2所示,FPGA处理板上集成多个模块与输入/输出接口,在工作过程中,首先对视频图像源的输入状态进行判断,按照相机输入、HDMI投影和测试图像的优先级依次递减的方式对输入的视频源进行选择性输入。相机图像通过光纤从光纤接口传输至FPGA处理板,光纤传输带宽为5Gbps,可进行双向数据传输,在接收视频画面的同时也可对相机端相机参数进行远程配置。电脑设备的HDMI投影,通过HDMI线连接HDMI输入接口传输至FPGA处理板。测试图像通过FPGA内部走线连接,无需外部物理连接。上述视频图像源的图像尺寸为4096×2160、帧率约为100帧/秒。As shown in Figure 2, the FPGA processing board integrates multiple modules and input/output interfaces. During the work process, the input status of the video image source is first judged, and the priorities of camera input, HDMI projection and test images are decremented. Selectively input the input video source in the same way. The camera image is transmitted from the optical fiber interface to the FPGA processing board through the optical fiber. The optical fiber transmission bandwidth is 5Gbps, which can carry out two-way data transmission. While receiving the video picture, the camera parameters of the camera can also be remotely configured. The HDMI projection of the computer equipment is connected to the HDMI input interface through the HDMI cable and transmitted to the FPGA processing board. The test image is connected through FPGA internal traces, no external physical connection is required. The image size of the above video image source is 4096×2160, and the frame rate is about 100 frames per second.
FPGA完成图像输入源的选择后,对图像做硬件去雾算法实现,将去雾后的清晰图像进行HDMI输出显示和送下一级算法模块,车牌定位模块在FPGA上实现了车牌位置的预选,并将位置坐标与坐标为中心的区域图像通过网络上传至计算机中,由软件完成后续车牌字符识别和用户操作界面工作。After the FPGA completes the selection of the image input source, it implements the hardware defogging algorithm for the image. The clear image after defogging is displayed on the HDMI output and sent to the next-level algorithm module. The license plate positioning module realizes the pre-selection of the license plate position on the FPGA. And the location coordinates and the area image centered on the coordinates are uploaded to the computer through the network, and the software completes the subsequent license plate character recognition and user interface work.
所述FPGA处理板,连接所述视频图像源,用于对输入的图像或视频图像进行实时去雾。为提高识别速度,本案采用暗通道图像去雾算法,并结合FPGA处理板进行针对性算法改进设计,步骤包括:The FPGA processing board is connected to the video image source for real-time defogging of the input image or video image. In order to improve the recognition speed, this case adopts the dark channel image defogging algorithm, and combines the FPGA processing board to carry out targeted algorithm improvement design. The steps include:
(1)对输入视频图像求取暗通道图像(1) Find the dark channel image for the input video image
对输入图像的每一像素点进行拆分,拆分为3个8bit数据,分别代表R、G、B三个通道。使用比较器对三个数据做比较,求出最小值;并将最小值作为暗通道图像的像素值,按照原有图像时序进行输出。Split each pixel of the input image into three 8-bit data, representing the three channels of R, G, and B respectively. Use a comparator to compare the three data to find the minimum value; and use the minimum value as the pixel value of the dark channel image, and output it according to the original image timing.
(2)根据暗通道图像估计透射率值t(x)和大气光值A(2) Estimate the transmittance value t(x) and atmospheric light value A according to the dark channel image
(2.1)透射率值t(x)(2.1) Transmittance value t(x)
使用双尺度最小差值滤波算法对输入视频图像进行滤波,采用BRAM资源构建9×9的滤波窗口。将所述滤波窗口作为基础模板,以5×5的子窗口对所述基础模板进行遍历。The input video image is filtered using the dual-scale minimum difference filtering algorithm, and a 9×9 filtering window is constructed using BRAM resources. The filtering window is used as a basic template, and the basic template is traversed with 5×5 sub-windows.
遍历方法主要通过调用FPGA内部的DSP资源以完成基础数学运算。首先,计算每个子区域像素均值,再计算均值与中心点像素值之差,之后选择差值最小的子区域进行最小值滤波,得到细化后的暗通道图,该细化后的暗通道用来计算透射率,透射率可根据通用雾天成像模型得到。The traversal method mainly completes basic mathematical operations by calling the DSP resources inside the FPGA. First, calculate the mean value of each sub-region pixel, and then calculate the difference between the mean value and the pixel value of the center point, and then select the sub-region with the smallest difference to perform minimum value filtering to obtain a refined dark channel map. The refined dark channel is used To calculate the transmittance, the transmittance can be obtained according to the general fog imaging model.
(2.2)估计大气光值A(2.2) Estimated atmospheric light value A
对暗通道的像素值做累加操作,累加结果存储在BRAM的各个地址中,每一地址对应一个灰度值的直方图统计结果。通过统计直方图,将前一帧暗通道图约前0.1%像素的均值作为阈值,并将前一帧与当前帧数据缓存在DDR单元中;再在当前帧的三通道根据像素与阈值的差值,进行线性调整来估计大气光值A。The pixel value of the dark channel is accumulated, and the accumulated result is stored in each address of the BRAM, and each address corresponds to a histogram statistical result of a gray value. Through the statistical histogram, the average value of the first 0.1% pixels of the dark channel image in the previous frame is used as the threshold, and the data of the previous frame and the current frame are cached in the DDR unit; value, a linear adjustment is made to estimate the atmospheric light value A.
(3)将估计所得的参数代入雾天成像模型,求得去雾后的图像。(3) Substitute the estimated parameters into the fog imaging model to obtain the image after fog removal.
FPGA处理板对去雾后的图像进行搜索,确定该帧图像中车牌在整幅图像的坐标位置。The FPGA processing board searches the image after defogging, and determines the coordinate position of the license plate in the whole image in the frame image.
车牌定位算法工作流程如图3所示,实施如下:The workflow of the license plate location algorithm is shown in Figure 3, and the implementation is as follows:
首先,对去雾算法处理后的RGB图像进行色彩空间转换,转换到YCbCr彩色空间。First, the RGB image processed by the dehazing algorithm is converted into the YCbCr color space.
其次,利用FPGA并行处理能力,同时对其Cb和Cr分量提取灰度图像,相较于软件实现能够将处理速度提升一倍。Secondly, using the parallel processing capability of FPGA to simultaneously extract grayscale images for its Cb and Cr components, compared with software implementation, the processing speed can be doubled.
接着,对Cr分量灰度图像逐像素点按照行场时序进行遍历,分别设置两个寄存器来存放目标像素范围值和像素均值。将图像最大灰度值与该灰度值的0.85倍值作为范围边界值,由于目标信息所在范围包含像素最小值,所以把此范围定义为目标像素范围值。在遍历结束后,采取背景抑制的设计思想,对从Cr分量灰度图像中得到的最小值进行非线性补偿。将像素均值和目标像素范围值的差值作为阈值C,将阈值C和Cr分量进行比较,得到处理后的Cr分量灰度图,并将处理后的Cr分量灰度图存放在DDR中。若Cr分量大于等于阈值C,将阈值C作为处理后的Cr分量灰度图;若小于,则将Cr分量作为处理后的Cr分量灰度图。再将同一帧下的Cb分量灰度图像与处理后的Cr分量灰度图从DDR缓存单元读出,使用DSP资源逐像素做差值运算,得到目标信息增强的灰度图像,从而实现抑制车身颜色带来的背景干扰,达到抑制多余噪声的目的。Next, the Cr component grayscale image is traversed pixel by pixel according to the line and field timing, and two registers are respectively set to store the target pixel range value and the pixel average value. The maximum gray value of the image and 0.85 times the gray value are used as the range boundary value. Since the range of the target information contains the minimum value of the pixel, this range is defined as the target pixel range value. After the traversal, the design idea of background suppression is adopted, and the minimum value obtained from the Cr component gray image is nonlinearly compensated. The difference between the pixel mean value and the target pixel range value is used as the threshold value C, and the threshold value C is compared with the Cr component to obtain the processed Cr component grayscale image, and the processed Cr component grayscale image is stored in the DDR. If the Cr component is greater than or equal to the threshold C, the threshold C is used as the processed Cr component grayscale image; if it is smaller, the Cr component is used as the processed Cr component grayscale image. Then read the grayscale image of Cb component and the processed grayscale image of Cr component in the same frame from the DDR cache unit, and use DSP resources to perform difference calculation pixel by pixel to obtain a grayscale image with enhanced target information, so as to realize the suppression of vehicle body The background interference brought by the color achieves the purpose of suppressing redundant noise.
后续通过自适应二值化模块,对上述灰度图像进行二值化分割,再进行形态学处理。具体为:先对二值化分割后的灰度图像进行膨胀处理,对车牌区域的孔洞进行连通,使用BRAM构建行缓存结构;在完成三行数据缓存后,将图像读出至所需尺度的寄存器阵列中构建卷积单元,由寄存器构建的卷积核大小为固定的9×9,然后再进行两次腐蚀,腐蚀卷积核分别为9×9、3×3大小,完成目标区域几何尺寸的还原以及散斑状干扰的滤除。Subsequently, through the adaptive binarization module, the above-mentioned grayscale image is binarized and segmented, and then morphological processing is performed. Specifically: first, expand the grayscale image after binary segmentation, connect the holes in the license plate area, and use BRAM to build a line cache structure; after completing the three-line data cache, read the image to the desired scale. The convolution unit is built in the register array. The size of the convolution kernel constructed by the register is fixed 9×9, and then two corrosions are performed. The corroded convolution kernels are respectively 9×9 and 3×3 in size, and the geometric size of the target area is completed. reduction and filtering of speckle interference.
接下来,对含有车牌区域的二值化图像进行识别,采用改进的包围盒目标检测算法。具体为,先定义一个BRAM存储器,然后以逐行扫描的方式对DDR读出图像中有重叠区域的线条进行面积累加,累加结果依次存储在BRAM地址中的最小存储单元中。在搜索过程中,对闭合区域进行标记,标记结果放置在寄存器中,并记录该闭合区域的面积、两侧顶点的坐标信息。在图像搜索结束时,对所有闭合区域面积送进比较器中进行比较,并设置面积范围和长宽比例作为过滤条件。该条件参考实际车牌长宽比例,具体数据被存储在片上ROM资源中。以此筛选出概率最大的闭合区域作为目标车牌,再对目标车牌坐标进行平滑处理,平滑步长为50个像素点。平滑操作是在检测出的2次结果之间设定一固定极限差值,通过DSP对两次结果做减法运算,并将减法结果与极限差值送比较器。若比较结果在设定的限定范围内,则采用检测出的车牌的坐标结果;若超出范围,则将极限差值赋值给坐标结果。随后将对应帧的RGB图像从DDR3缓存区读出,通过网口将车牌坐标信息和以车牌区域几何中心为坐标原点,以笛卡尔坐标系为参考坐标系,向X轴正负方向、Y轴正负方向各扩展370、300个像素所包围成的矩形区域,上传给电脑设备的上位机软件中。传输格式采用自定义编码方式,使用FPGA逻辑单元构建以太网帧格式并将有效数据填充至数据段,送至接口为RGMII的MAC层后,由板载PHY芯片完成后续网络通信,如图4。Next, the binarized image containing the license plate area is recognized, and an improved bounding box target detection algorithm is used. Specifically, a BRAM memory is first defined, and then the lines in the DDR readout image with overlapping areas are accumulated and accumulated in a progressive scanning manner, and the accumulation results are sequentially stored in the smallest storage unit in the BRAM address. During the search process, the closed area is marked, and the marked result is placed in the register, and the area of the closed area and the coordinate information of the vertices on both sides are recorded. At the end of the image search, the areas of all closed areas are sent to the comparator for comparison, and the area range and aspect ratio are set as filter conditions. This condition refers to the actual license plate aspect ratio, and the specific data is stored in the on-chip ROM resource. Based on this, the closed area with the highest probability is selected as the target license plate, and then the coordinates of the target license plate are smoothed with a smoothing step of 50 pixels. The smoothing operation is to set a fixed limit difference between the two detected results, perform subtraction on the two results through DSP, and send the subtraction result and the limit difference to the comparator. If the comparison result is within the set limit range, the coordinate result of the detected license plate is used; if it exceeds the range, the limit difference is assigned to the coordinate result. Then the RGB image of the corresponding frame is read from the DDR3 buffer area, and the coordinate information of the license plate and the geometric center of the license plate area as the coordinate origin and the Cartesian coordinate system as the reference coordinate system are transferred to the positive and negative directions of the X axis and the Y axis through the network port. The rectangular area surrounded by 370 and 300 pixels expanded in the positive and negative directions is uploaded to the host computer software of the computer equipment. The transmission format adopts a custom encoding method, uses the FPGA logic unit to construct the Ethernet frame format and fills the valid data into the data segment, and after sending it to the MAC layer with the RGMII interface, the onboard PHY chip completes the subsequent network communication, as shown in Figure 4.
上位机软件在接收到FPGA处理板通过以太网接口发送的视频流数据后,同时做以下两项工作:After the host computer software receives the video stream data sent by the FPGA processing board through the Ethernet interface, it will do the following two tasks at the same time:
一、将接收到的视频画面实时显示在上位机软件视频显示区域中。1. Display the received video screen in the video display area of the upper computer software in real time.
二、上位机软件在接收到车牌坐标信息后,因车牌坐标在FPGA处理板与上传画面参考坐标系的不同,对车牌坐标作坐标偏移处理,得到上传视频图像中车牌坐标的实际位置信息。2. After receiving the license plate coordinate information, the upper computer software performs coordinate offset processing on the license plate coordinates due to the difference between the license plate coordinates on the FPGA processing board and the reference coordinate system of the uploaded screen, and obtains the actual position information of the license plate coordinates in the uploaded video image.
得到车牌坐标信息后,对车牌图像进行二值化处理,并对其展开以下操作:构建矩形形状的结构元素,对二值化图像依次进行腐蚀、膨胀,将构建的结构元素和处理后的二值化图像相减,以此得到去除冗余部分的图形。进一步定位字符,遍历车牌号码图像的每一个像素点,得到该图像的每一列的目标像素个数。将每一列的目标像素个数和设定的阈值A作比较,来判断该列是否为字符,若不为字符,将目标像素个数大于阈值A的连续列数和阈值B作比较,用来更新字符位置信息;若为字符,则将字符宽度进行累加,当字符宽度不再累加,则当前字符位置定位结束,开始定位下一个字符,重复定位字符的操作步骤,直到车牌上的8个字符位置全部定位结束。最后利用模板匹配进行字符识别,创建仿射变换目标图像,计算仿射变换矩阵,对加载图形进行仿射变换操作,再与字符库作对比,输出识别的车牌字符到上位机软件车牌字符显示区域中,如图5。After the license plate coordinate information is obtained, the license plate image is binarized, and the following operations are performed on it: construct a rectangular structural element, corrode and expand the binarized image in turn, combine the constructed structural element with the processed binary The valued images are subtracted to obtain a graph with redundant parts removed. Further locate the characters, traverse each pixel of the license plate number image, and obtain the target pixel number of each column of the image. Compare the number of target pixels in each column with the set threshold A to determine whether the column is a character. If it is not a character, compare the number of consecutive columns with the number of target pixels greater than the threshold A with the threshold B for Update the character position information; if it is a character, the character width is accumulated. When the character width is no longer accumulated, the current character position positioning is completed, and the next character is positioned, and the operation steps of character positioning are repeated until there are 8 characters on the license plate All positions are located. Finally, use template matching for character recognition, create an affine transformation target image, calculate the affine transformation matrix, perform an affine transformation operation on the loaded image, compare it with the character library, and output the recognized license plate characters to the upper computer software license plate character display area , as shown in Figure 5.
尽管以上结合附图对本发明的实施方案进行了描述,但本发明并不局限于上述的具体实施方案和应用领域,上述的具体实施方案仅仅是示意性的、指导性的,而不是限制性的。本领域的普通技术人员在本说明书的启示下和在不脱离本发明权利要求所保护的范围的情况下,还可以做出很多种的形式,这些均属于本发明保护之列。Although the embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields, and the above-mentioned specific embodiments are only illustrative, instructive, and not restrictive . Under the enlightenment of this specification and without departing from the protection scope of the claims of the present invention, those skilled in the art can also make many forms, which all belong to the protection of the present invention.
Claims (10)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202310221631.5A CN116403200A (en) | 2023-03-08 | 2023-03-08 | License plate real-time identification system based on hardware acceleration |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202310221631.5A CN116403200A (en) | 2023-03-08 | 2023-03-08 | License plate real-time identification system based on hardware acceleration |
Publications (1)
Publication Number | Publication Date |
---|---|
CN116403200A true CN116403200A (en) | 2023-07-07 |
Family
ID=87016907
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202310221631.5A Pending CN116403200A (en) | 2023-03-08 | 2023-03-08 | License plate real-time identification system based on hardware acceleration |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN116403200A (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN117689732A (en) * | 2024-01-31 | 2024-03-12 | 西南技术物理研究所 | Image gray level rectangularity statistics and target searching method and system based on FPGA |
-
2023
- 2023-03-08 CN CN202310221631.5A patent/CN116403200A/en active Pending
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN117689732A (en) * | 2024-01-31 | 2024-03-12 | 西南技术物理研究所 | Image gray level rectangularity statistics and target searching method and system based on FPGA |
CN117689732B (en) * | 2024-01-31 | 2024-04-23 | 西南技术物理研究所 | Image gray level rectangularity statistics and target searching method and system based on FPGA |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP3853034B2 (en) | Object boundary determination method and apparatus, and recording medium recording object boundary determination program | |
CN112669344A (en) | Method and device for positioning moving object, electronic equipment and storage medium | |
CN112364865A (en) | Method for detecting small moving target in complex scene | |
CN112053383A (en) | Method and device for real-time positioning of robot | |
CN114025089B (en) | A video image acquisition jitter processing method and system | |
CN111951192A (en) | Shot image processing method and shooting equipment | |
CN117173070A (en) | Image processing fusion method and system based on FPGA | |
CN113421217A (en) | Method and device for detecting travelable area | |
CN110310305A (en) | A target tracking method and device based on BSSD detection and Kalman filter | |
CN111507340A (en) | Target point cloud data extraction method based on three-dimensional point cloud data | |
CN110197467A (en) | A kind of optimization system based on FPGA image Penetrating Fog | |
CN116403200A (en) | License plate real-time identification system based on hardware acceleration | |
CN117115415A (en) | Image labeling processing method and system based on big data analysis | |
CN116523959A (en) | A method and system for detecting moving objects based on artificial intelligence | |
CN111767752A (en) | Two-dimensional code identification method and device | |
CN113793349B (en) | Target detection method and device, computer readable storage medium and electronic equipment | |
CN119007137A (en) | Traffic flow monitoring method and system based on multi-source data fusion | |
CN117575966B (en) | A video stabilization method for high-altitude hovering shooting scenes of unmanned aerial vehicles | |
CN117496160B (en) | Indoor scene-oriented semantic segmentation method for low-illumination image shot by unmanned aerial vehicle | |
CN119027839A (en) | Bridge crack identification method, device and storage medium based on drone video stream | |
CN110264488B (en) | Binary image edge extraction device | |
CN117237386A (en) | Method, device and computer equipment for carrying out structuring processing on target object | |
CN115393746A (en) | Multi-scale target fault detection method and application of transmission line based on DenseNet | |
CN113721240A (en) | Target association method and device, electronic equipment and storage medium | |
CN114202520B (en) | Chromatographic processing method, device, computer equipment, and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |