WO2014193220A2

WO2014193220A2 - System and method for multiple license plates identification

Info

Publication number: WO2014193220A2
Application number: PCT/MY2014/000142
Authority: WO
Inventors: Hafez Bin Nawi AHMAD; Santosh Palai SMRUTI; Prasad DEVI
Original assignee: Mimos Berhad
Priority date: 2013-05-28
Filing date: 2014-05-21
Publication date: 2014-12-04
Also published as: MY182746A; WO2014193220A3

Abstract

The present invention provides a license plates recognition system (400) for detecting and extracting multiple license plates. The system (400) acquires video sequences of the moving vehicles and processes the same to extract license plate registration number for matching. The system (400) utilizes FPGA having an edge detector (406) and a ROI based extractor (408) for determining the possible license plates and it position (or coordinate) thereof. The edge detections and ROI based extractions are carried out to process multiple images in parallel. The extracted license plate images are then processed by an OCR (412) to extract the registration numbers therein, and a search and alert module (414) is provided for search and matching (510) the license plate of interest. A method of processing the images to extract license plate is also provided.

Description

System And Method for Multiple License Plates Identification

Field of the Invention

[0001] The present invention relates to image processing. More particularly, the present invention relates to system and method for multiple license plates identification. Background

[0002] With the advent of new generation of high-resolution surveillance cameras, it is now possible to apply video processing and recognition techniques on live video streams for the purpose of automatically detecting and identifying vehicle license plate. However, as the number of cameras increased, tough challenges have been posed on the license plate identification system to process multiple images efficiently.

[0003] License Plate Identification system, also widely known as License Plate

Recognition (LPR) system is required to possess sufficient real-time image processing capabilities. An LPR or Identification system includes a plurality of image processing processors, cameras, frame grabbers and other necessary elements in a closed network. However, conventional license plate identification system that relies on serial processing approach to process multiple license plate images, resulting in low reading rate of license plates, and thus slower response.

[0004] FIG. 1 shows a known LPR of US patent no. 7,068,185 issued to

Raytheon Company on 27 June 2006. The method disclosed uses a serial processing based approach whereby one process has to be completed before the next starts. This would affect system performance and limit the promptly response to the real-time data. [0005] FIG. 2 shows a diagram of Queues Analysis of US patent no. 8,224,028 issued to Verint System Ltd. on 17 July 2012. Although the application is used for analyzing the queues, the video analytics functions disclosed for image processing is similar to that of the method as shown in FIG. 1. [0006] FIG. 3 shows another LPR known in the art. The LPR provides a full software approach based on a serial processing method to recognize and extract license plate of interest. The process starts with acquiring video sequences of the motion vehicles. The video sequences are processed by a region-of-interest based position predictor. The position predictor processes the video sequences with an edge tracker follows by a MGET classifier then a perceptual region grouper and object identifier. A license plate model is used as a reference for producing segmented license plate. When the video sequences include multiple vehicles, the position predictor process is required to run through the edge tracker through objection identifier again to detect the next license plate. Summary

[0007] In accordance with one aspect of the present invention, there is provided a system for identifying multiple license plates from multiple video streams. The system comprises an edge detection based plate identification module adapted to operably process multiple images of the video streams in parallel for detecting license plate locations from the images of multiple video streams; a Region-of-Interest (ROI) based extraction module adapted to operably process in parallel the license plate location to extract data in relation to license plate area; an Optical Character Recognition (OCR) unit to recognise characters from the license plate area to extract license plate registration number; and a search and alert module adapted for matching the extracted plate registration number with multiple plate numbers of interest in parallel.

[0008] In one embodiment, the edge detection based plate identification module further comprises a noise filter for filtering noise from the images of the video streams; an image sharpening module for sharpening the images to emphasize features of the images; an edge detector for extracting boundaries of license plate images; and a plate candidate verification unit to filter out images without any license plate detected.

[0009] In another embodiment, the ROI based extraction module further comprises a line tracker for receiving memory pointer value of images and select the image that have height and width of plate candidate; and a plate evaluator to identify a correct plate candidates.

[0010] In a further embodiment, the OCR unit comprises an image segmentation unit for segmenting the images into individual character images; a character recognizer to recognize characters from the individual character images; and a syntax analyser to compare the form of chain of characters with the patterns of correct vehicle plate numbers.

[0011] In yet a further embodiment, the search and alert module may comprise an image buffer to store the images for later processing; a user input module to receive user input on plate search; a plate comparison and verification module to match the license plate registration number from the OCR module with the user input; and an alert unit to generate alert signal to user when a match is found. [0012] In another aspect of the present invention, there is provided a method for identifying multiple license plates from multiple video streams. The method comprises acquiring images of the multiple video streams; performing edge detection on the images of the multiple video streams in parallel to detect license plate locations from the images, wherein the images are processed in parallel; performing Region-of-Interest (ROI) based extraction for extracting data in relation to the license plate locations, wherein the extraction is carried out on the images in parallel; performing Optical Character Recognition (OCR) on the extracted license plate data to extract license plate registration number; and matching the extract license plate registration number with multiple plate numbers of interest in parallel.

[0013] In one embodiment the performing edge detection further comprises filtering noise from the images of the video streams; sharpening the images to emphasize features of the images; extracting boundaries of license plate images; and filtering out images without any license plate detected. [0014] In another embodiment, the performing ROI based extraction further comprises selecting the images that have height and width of plate candicate; and evaluating to identify a correct plate candicates.

[0015] In a further embodiment, the OCR further comprises segmenting the images into individual character images; recognising characters from the individual character images; and comparing the form of chain of characters with the patterns of correct vehicle plate numbers.

[0016] In yet a further embodiment, the matching the extract license plate registration number may further comprises storing the images for later processing; receiving user input on plate search; matching license plate registration number extracted by the OCR with the user input; and alerting users when a match is found.

Brief Description of the Drawings

[0017] Preferred embodiments according to the present invention will now be described with reference to the figures accompanied herein, in which like reference numerals denote like elements;

[0018] FIG. 1 shows a known license plate recognition system;

[0019] FIG. 2 shows a diagram of a known queues analysis;

[0020] FIG. 3 shows yet another known license plate recognition system;

[0021] FIG. 4 illustrates a schematic diagram of the License Plate Recognition

(LPR) system in accordance with one embodiment of the present invention;

[0022] FIG. 5 illustrates a process flow of identifying multiple license plates of vehicles through the LPR system in accordance with one embodiment of the present invention;

[0023] FIG. 6 illustrates a parallel processing process for extracting license plate registration number is accordance with another embodiment of the present invention;

[0024] FIG. 7 shows an Edge Detection Based Plate Identification module in accordance with one embodiment of the present invention; [0025] FIG. 8 shows a ROI Based Extraction module in accordance with one embodiment of the present invention;

[0026] FIG. 9 shows an OCR module in accordance with one embodiment of the present invention; and [0027] FIG. 10 shows a Search and Alert module in accordance with one embodiment of the present invention.

Detailed Description

[0028] Embodiments of the present invention shall now be described in detail, with reference to the attached drawings. It is to be understood that no limitation of the scope of the invention is thereby intended, such alterations and further modifications in the illustrated device, and such further applications of the principles of the invention as illustrated therein being contemplated as would normally occur to one skilled in the art to which the invention relates.

[0029] The present invention addresses the system performance issues described above by speeding up each process involved in processing the license plates' images. This can be achieved by implementing parallel processing on dedicated process blocks, where the rate of processing of license plates' images and the overall system performance can be increased. Further, the present invention proposes the application of an FPGA, which would improve performance and provide flexibility for future updates. In one embodiment, a system and method for multiple license plate identification with higher rate of checking license plates and less delay compared with typically used license plate identification system is provided. [0030] FIG. 4 illustrates a schematic diagram of the License Plate Recognition

(LPR) system 400 in accordance with one embodiment of the present invention. The LRP system 400 is running on a Field- Programmable Gate Array 401 deployed on a central server. The central server is adapted for controlling a plurality of video cameras 405 connected thereto. The LRP system 400 comprises two memory units 402, 404, an edge detection based plate identification module 406, a Region-of-Interest (ROI) Based Extraction Module 408, an optical character recognition (OCR) module 412, and a search and alert module 414.

[0031] The first memory unit 402 is adapted to stores incoming video streams captured by the plurality of video cameras 405. The second memory unit 404, on the other hand, is a temporary storage for the OCR module 412.

[0032] The Edge Detection Based Plate Identification Module 406 contains edge detector processing functions that work in parallel to process multiple video streams simultaneously to detect license plate locations. [0033] The ROI Based Extraction Module 408 provides region of interest processing functions that work in parallel to extract data related to plate area using the memory pointers indicated by the edge detector.

[0034] The OCR Module 412 offers optical character recognition processing functions working in serial to perform character recognition on the extracted license plate registration number from the detected license plate.

[0035] The detected license plate registrations numbers are being processed at the Search and Alert Module 414 provides a processing function which does license plate comparison in parallel with user input and finally gives an alert message if data match occurs. The search and alert module 414 may also compare the detected license plate numbers against a database of wanted license plate numbers, and similarly, when a match is found, the user is alerted accordingly. [0036] In this embodiment, the user is required to enter the license plate number of interest into the system for conducting the search and match. In another embodiment, there may be provided with a database with a list of license plate number of interest, and as the LPR system 400 is processing to recognize license plates from the images. The parallel processing allows the system to effectively process multiple license plates simultaneously.

[0037] FIG. 5 illustrates a process flow of identifying multiple license plates of vehicles through the LPR system 400 in accordance with one embodiment of the present invention. The process flow is initiated through the FPGA 401 on the central server receiving video streams from a number of video cameras 405 at step 501. Then at step 502, images of the video streams are stored on the first memory unit 402. At step 503, edge detection is performed on the images by the edge detection based plate identification module 406. The edge detection 503 on the images is looped until a license plate is identified at step 504. The edge detection 503 determines the location(s) of the license plate(s) from the stored images to output its coordinate(s). Once a license plate is detected, at step 505, its coordinate on the processed image is provided to the ROI based extraction module 408. At step 506, only images associated to these coordinates are fetched from first memory, i.e. images without any license plate detected are discarded or ignored. These coordinates are basically memory pointers. Operationally, the ROI based extraction module 408 accesses the memory and extracts the license plate number therefrom. At step 507, the extracted data related to the license plate area is stored on the second memory unit 404 before being further processed. Preferably, the steps 503 to 507 are carried out in parallel to process multiple images as they are received. In step 508, the extracted data are being further utilised by the OCR module 412 to extract readable texts or characters from the detected license plate images. The OCR processing is carried out in series. The extracted license plate registration numbers are then processed by the search and alert module 414 at step 509 to match with a database of wanted license plate registration at step 510. If a match is found, the results shall be displayed to alert the user at step 511. Similarly, it is desired that steps 509 to 511 are carried out in parallel.

[0038] FIG. 6 illustrates a parallel processing process for extracting license plate registration number is accordance with another embodiment of the present invention. As shown, upon receiving the video frames (or images) from the groups of video cameras, the video frames are stored in the first memory unit 402. Then the edge detection is performed on multiple video frames in parallel processing. The edge detection detect the possible license plates appear on the video frames, and it relevant coordinate or position information is being recorded and processed at the ROI based extraction module 408. The video frames are fetched from the first memory unit 402 for ROI based extractions. The ROI extraction results is then stored on the second memory unit 404 and thereafter, processed by the OCR module 412 to extract the license plate registration numbers. The extracted registration numbers are matched at the search and alert module 414 and user shall be alerted when a match is found. The entire license plate recognition is performed in a combined parallel and serial processing scheme to process multiple video frames at the same time. In contrast with the conventional method, the video frame is processed one at a time in a recursive manner from edge detection to OCR processing to registration number matching. Such serial processing is not efficient especially when the system is to handle a large number of video cameras. [0039] In the proposed invention, video image processing on multiple images is performed in the FPGA due to its ability to do pipelining and parallel operations whereby a software solution in a CPU may have to do hundreds of clock cycles to do the same thing.

[0040] FIG. 7 shows an Edge Detection Based Plate Identification module 406 in accordance with one embodiment of the present invention. The license plate images from first memory are processed to filter out the noise at step 702. The images are then sharpened to emphasize the images' details at step 704. Then, boundaries of the license plate images are detected at step 706. At step 708, only the images with detected license plate are selected for further processing. The select images of plate candidates are sent to ROI based processing. The images that do not contact any detected license plate are filtered out.

[0041] FIG. 8 shows a ROI Based Extraction module 408 in accordance with one embodiment of the present invention. At step 802, a Line Tracker receives memory pointer value, i.e. the coordinates, of plate candidate image from the edge detection based plate identification module 406 and selects the corresponding image from first memory unit 402. Then, a plate evaluator identifies the correct plate image among plate candidate images and forwards them to second memory unit 404. [0042] FIG. 9 shows an OCR module 412 in accordance with one embodiment of the present invention. The OCR module 412 receives the plate images from the second memory unit 404. At step 902, the plate images are segmented into separate character images. These character images are then processed to recognise the characters from the images at step 904. The extracted characters are further processed with a Syntax Analysis to compare the form of chain of characters created with the patterns of correct vehicle plate numbers at step 906. The results are transferred to Search and Alert module 414.

[0043] FIG. 10 shows a Search and Alert module 414 in accordance with one embodiment of the present invention. The results from the OCR module 412 are stored in an image buffer at step 1002. Meanwhile, the system also receives user input at step 1004. At step 1006, a plate comparison and verification unit receives the input from user and compares the results of OCR to determine for any matching. When a match is found, an alert message is generated at step 1008.

[0044] While specific embodiments have been described and illustrated, it is understood that many changes, modifications, variations, and combinations thereof could be made to the present invention without departing from the scope of the invention.

Claims

1. A system (400) for identifying multiple license plates from multiple video streams, the system (400) comprising:

an edge detection based plate identification module (406) adapted to operably process multiple images of the video streams in parallel for detecting license plate locations from the images of multiple video streams;

a Region-of-Interest (ROI) based extraction module (408) adapted to operably process in parallel the license plate location to extract data in relation to license plate area;

an Optical Character Recognition (OCR) unit (412) to recognise characters from the license plate area to extract license plate registration number; and

a search and alert module (414) adapted for matching the extracted plate registration number with multiple plate numbers of interest in parallel.

2. The system (400) of claim 1 , wherein edge detection based plate identification module (406) further comprising:

a noise filter (702) for filtering noise from the images of the video streams; an image sharpening module (704) for sharpening the images to emphasize features of the images;

an edge detector (706) for extracting boundaries of license plate images; and a plate candidate verification unit (708) to filter out images without any license plate detected.

3. The system (400) of claim 1, wherein the ROI based extraction module (408) further comprising:

a line tracker (802) for receiving memory pointer value of images and select the image that have height and width of plate candidate; and

a plate evaluator (804) to identify a correct plate candidates.

4. The system (400) of claim 1, wherein the OCR unit (412) further comprising: an image segmentation unit (902) for segmenting the images into individual character images; a character recognizer (904) to recognize characters from the individual character images; and

a syntax analyser (906) to compare the form of chain of characters with the patterns of correct vehicle plate numbers.

5. The system (400) of claim 1, wherein the search and alert module (414) further comprising:

an image buffer (1002) to store the images for later processing;

a user input module (1004) to receive user input on plate search;

a plate comparison and verification module (1006) to match the license plate registration number from the OCR module with the user input; and

an alert unit (1008) to generate alert signal to user when a match is found.

6. A method for identifying multiple license plates from multiple video streams, the method comprising:

acquiring (501) images of the multiple video streams;

performing edge detection (503) on the images of the multiple video streams in parallel to detect license plate locations from the images, wherein the images are processed in parallel;

performing Region-of-Interest (ROI) based extraction (506) for extracting data in relation to the license plate locations, wherein the extraction is carried out on the images in parallel;

performing Optical Character Recognition (OCR) (508) on the extracted license plate data to extract license plate registration number; and

matching (510) the extract license plate registration number with multiple plate numbers of interest in parallel.

7. The method according to claim 6, wherein performing edge detection (503) further comprising:

filtering noise (702) from the images of the video streams;

sharpening (704) the images to emphasize features of the images; extracting (706) boundaries of license plate images; and

filtering (708) out images without any license plate detected.

8. The method according to claim 6, wherein performing ROI based extraction further comprising:

selecting (802) the images that have height and width of plate candicate; and evaluating (804) to identify a correct plate candicates.

9. The method according to claim 6, wherein the OCR (508) further comprising: segmenting (902) the images into individual character images;

recognising (904) characters from the individual character images; and comparing (906) the form of chain of characters with the patterns of correct vehicle plate numbers.

10. The method of claim 6, wherein matching (510) the extract license plate registration number further comprising: storing (1002) the images for later processing; receiving (1004) user input on plate search; matching (1006) license plate registration number extracted by the OCR with the user input; and alerting (1008) users when a match is found.