KR102331453B1 - System for vehicle number plate recognition using neural network and its method - Google Patents

Abstract

The present invention relates to a license plate recognition system and method using a neural network, comprising: a) detecting a license plate image including a license plate area from the vehicle image when a vehicle image is obtained; b) generating a license plate composite image by synthesizing the detected license plate image into a preset font using an image conversion algorithm; c) learning the license plate recognition model using the learning data pairing the license plate composite image and the target vehicle number; and d) when a vehicle image composed of at least one or more letters and numbers is input, the license plate composite image generated through the image conversion algorithm is input to the learned license plate recognition model to recognize the vehicle number and then the recognized vehicle number It may be a method comprising the step of providing.

Description

License plate recognition system and method using neural network { SYSTEM FOR VEHICLE NUMBER PLATE RECOGNITION USING NEURAL NETWORK AND ITS METHOD }

The present invention relates to a text recognition system and method using a neural network capable of learning a license plate recognition model using a license plate composite image synthesized into a single font using an image conversion algorithm.

License plate recognition technology is being applied in places such as unmanned parking management system, illegally parked vehicle detection system, automatic speeding fine system on highway, and illegal license plate detection system of stolen vehicle, and constitutes a system that recognizes more quickly and accurately. This is an essential skill to do.

Recently, as an unmanned environment system for disaster and security is used to check, monitor, and track traffic information, a system capable of detecting and recognizing license plates in an unmanned environment is required.

The conventional license plate recognition system has a limitation in that the camera detects a license plate area within a limited angle in a fixed environment, and it is difficult to generate a large number of sample data for optimal accuracy even when using a deep learning algorithm.

In general, the vehicle license plate recognition system proceeds in three steps: image acquisition, license plate area detection, and character recognition. In this case, the license plate recognition technology may use a circular matching method, a syntactic method, a method using a neural network, and the like.

In particular, the license plate recognition method using a neural network includes a non-feature extraction method in which a pixel value of a text image is determined as an input pattern as it is, and a feature extraction method in which a feature is extracted from the text image and used. Compared to the feature extraction method, the non-feature extraction method takes less time to process because it does not consume the time to extract the feature, but in the Korean character recognition with a large number of image information constituting the text and the number of patterns to be recognized, the amount of calculation required to perform recognition increases. The number of input nodes increases.

On the other hand, as a prior art, there is a study on character recognition of a license plate using a grapheme separation method in order to increase the efficiency of character recognition. However, there is a problem in that it is practically difficult to separate Hangul characters from alphabets because a lot of noise is included in the license plate image due to damage to the actual license plate, camera performance, and various other conditions.

In addition, the license plate recognition method using a conventional neural network learns license plate images with various fonts for license plate recognition in various environments in the process of recognizing characters for the detected license plate area.

At this time, as a large amount of learning data can be generated for one license plate image through changes in illuminance, pixel movement, thickness and size changes of characters, and color changes of license plate backgrounds or characters, the required learning data increases rapidly. , there is a problem in that it takes a lot of money or time to learn a large amount of learning data, and therefore it is very difficult to add a new font.

Republic of Korea Patent Publication No. 10-2019-0095571 (Title of the invention: vehicle license plate automatic recognition method)

In order to solve the above problems, the present invention learns a license plate recognition model using a license plate composite image that synthesizes license plate images of various fonts into a single font according to an embodiment of the present invention, thereby reducing the number of plates compared to the existing license plate recognition system. The purpose is to improve recognition performance by using numerical training data.

However, the technical task to be achieved by the present embodiment is not limited to the technical task as described above, and other technical tasks may exist.

In the license plate recognition method using a neural network performed by the vehicle license plate recognition system according to an embodiment of the present invention as a technical means for achieving the above technical problem, a) when a vehicle image is obtained, the license plate area in the vehicle image Detecting a license plate image comprising a; b) generating a license plate composite image by synthesizing the detected license plate image into a preset font using an image conversion algorithm; c) learning the license plate recognition model using the learning data pairing the license plate composite image and the target vehicle number; and d) when a vehicle image composed of at least one or more letters and numbers is input, the license plate composite image generated through the image conversion algorithm is input to the learned license plate recognition model to recognize the vehicle number and then the recognized vehicle number It may be a method comprising the step of providing.

The image conversion algorithm may use an Image to Image Translation (Pix2Pix) algorithm.

The license plate image is converted to a font having a preset thickness and size through the image conversion algorithm, and Bidirectional Long Short-Term Memory (BLSTM) based on an attention algorithm is applied to position letters or numbers It may further include the step of generating a license plate composite image by determining the.

In the license plate image, a background area and a text area including numbers or letters are distinguished and detected, the casting area is converted to a preset color, and the text area is converted to have a color different from the background area.

In this case, each determined position of the letter or number may be displayed as a separator, and when the vehicle number in which at least one or more letters or numbers are combined is completed, it may be displayed as an end separator. The delimiter or the end delimiter may use any one or more of a special character or a special symbol.

The license plate recognition model may use a convolutional neural network-Long Short-Term Memory (CNN-LSTM) complex model that extracts features from the license plate composite image, and sequentially integrates and uses the extracted features.

According to the problem solving means of the present invention described above, using a license plate composite image synthesized to have a single font, a single background color and a character color through an image conversion algorithm of license plate images having various fonts, background colors, and character colors By learning the recognition model, the time and effort for generating learning data can be minimized compared to the existing license plate recognition system, and there is no need to learn by adding a new font, and labeling is possible using much less learning data, As the learning amount decreases, the cost required to construct the training data may decrease.

1 is a view showing the configuration of a vehicle license plate recognition system according to an embodiment of the present invention.
2 is a flowchart illustrating a license plate recognition method using a neural network according to an embodiment of the present invention.
3 is a view for explaining a learning process of the license plate recognition model according to an embodiment of the present invention.
4 is an exemplary diagram illustrating a text recognition process by a license plate recognition model according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . Also, when a part "includes" a component, it means that other components may be further included, rather than excluding other components, unless otherwise stated, and one or more other features However, it is to be understood that the existence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded in advance.

In the present specification, a 'terminal' may be a wireless communication device with guaranteed portability and mobility, for example, any type of handheld-based wireless communication device such as a smart phone, a tablet PC, or a notebook computer. In addition, the 'terminal' may be a wired communication device such as a PC that can connect to another terminal or server through a network. In addition, the network refers to a connection structure capable of exchanging information between each node, such as terminals and servers, and includes a local area network (LAN), a wide area network (WAN), and the Internet (WWW). : World Wide Web), wired and wireless data networks, telephone networks, and wired and wireless television networks.

Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi, Bluetooth communication, infrared communication, ultrasound Communication, Visible Light Communication (VLC), LiFi, etc. are included, but are not limited thereto.

The following examples are detailed descriptions to help the understanding of the present invention, and do not limit the scope of the present invention. Accordingly, an invention of the same scope performing the same function as the present invention will also fall within the scope of the present invention.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a view showing the configuration of a vehicle license plate recognition system according to an embodiment of the present invention.

Referring to FIG. 1 , the license plate recognition system 100 includes a communication module 110 , a memory 120 , a processor 130 , and a database 140 .

In detail, the communication module 110 provides a communication interface necessary to provide a signal transmitted and received to the license plate recognition system 100 in the form of packet data in conjunction with the communication network 300 . Here, the communication module 110 may be a device including hardware and software necessary for transmitting and receiving signals such as control signals or data signals through wired/wireless connection with other network devices.

In the memory 120, a program for performing a license plate recognition method using a neural network is recorded. In addition, the processor 130 performs a function of temporarily or permanently storing the processed data. Here, the memory 120 may include a volatile storage medium or a non-volatile storage medium, but the scope of the present invention is not limited thereto.

The processor 130 controls the entire process for executing the license plate recognition method, and converts the license plate image to a license plate composite image having a single font, and uses the learning data consisting of a pair of the license plate composite image and the target vehicle number. After learning the license plate recognition model, it is possible to recognize the vehicle number in the license plate area from the vehicle image through the learned license plate recognition model. Each step performed by the processor 130 will be described later with reference to FIG. 2 .

Here, the processor 130 may include all kinds of devices capable of processing data. Here, the 'processor' may refer to a data processing device embedded in hardware, for example, having a physically structured circuit to perform a function expressed as a code or an instruction included in a program. As an example of the data processing device embedded in the hardware as described above, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated (ASIC) circuit) and a processing device such as a field programmable gate array (FPGA), but the scope of the present invention is not limited thereto.

The database 140 stores data accumulated while executing a program for a license plate recognition method. For example, the database 140 may store a vehicle image, a license plate composite image, learning data, and newly recognized vehicle number information.

Figure 2 is a flowchart illustrating a license plate recognition method using a neural network according to an embodiment of the present invention, Figure 3 is a view for explaining a learning process of a license plate recognition model according to an embodiment of the present invention, Figure 4 is this It is an exemplary diagram illustrating a text recognition process by a license plate recognition model according to an embodiment of the present invention.

2 to 4 , the license plate recognition method using a neural network detects a license plate image including a license plate area from the vehicle image when a sample vehicle image is input from the processor 130 (S1). The processor 130 generates a license plate composite image by synthesizing the license plate image into a preset font using an image conversion algorithm (S2). In this case, the image conversion algorithm uses an Image to Image Translation (Pix2Pix) algorithm.

3, the processor 130 converts the sample vehicle image into a preset font through an image conversion algorithm to generate a license plate composite image, and an attention algorithm-based bidirectional Long Short- Term Memory, BLSTM ) is applied to determine the position of letters and numbers to recognize the license plate.

This attention algorithm-based BLSTM is usefully used to determine the position of a word in the classification or translation of a sentence, and in the license plate recognition model 200, A more improved bidirectional long short-term memory (BLSTM) is used to check the previous direction of the input from the existing LSTM that only faces one direction.

As shown in FIG. 4, the license plate composite image is displayed as a separator for each position of the determined letter or number, and when the vehicle number in which at least one or more letters and numbers are combined is completed, it is displayed as an end separator. The delimiter or end delimiter is a space character (SP (Space), EOS (End Of Space)), '*, #, ;, .' Any one or more of special characters or special symbols such as .

The license plate recognition model 200 is CNN-LSTM (convolutional neural network- Long Short-Term Memory) using a CNN model for extracting features from a license plate composite image, and an LSTM model that sequentially integrates and uses features extracted from the CNN model. It can be a composite model. The license plate recognition model 200 may enable license plate detection even in an unmanned aerial vehicle environment using the YOLO algorithm of the R-CNN (Region-Convolutional Neural Network) method as well as the CNN-LSTM complex model. That is, when a sample vehicle image entering from various angles is input, it is possible to generate a neural network for license plate detection by learning with the YOLO algorithm.

The processor 130 generates training data that pairs the license plate composite image and the target vehicle number (S3), and learns the license plate recognition model using the generated training data (S4). In this case, the target vehicle number may be various in Korean, numbers, special symbols, foreign languages including English, and the like.

The processor 130 recognizes the vehicle number corresponding to the vehicle image by inputting the vehicle image into the learned license plate recognition model input, and provides the recognized result to the user terminal (S5, S6). In this case, the processor 130 may learn the license plate recognition model by using the newly recognized vehicle number and vehicle image as learning data.

As shown in Figure 4, when the license plate image is 'YZ1547', the characters of the license plate have different fonts for each country or region, and sometimes they are written in handwriting. Accordingly, the processor 130 uses the Pix2Pix algorithm to synthesize a license plate image with a single font, a single background color, and a single character color, and provides it to the license plate recognition model 200 as a license plate composite image. At this time, the license plate composite image is displayed as a separator and an end separator between each letter and letter, letter and number, and number and number. can provide

As such, the license plate recognition model is composed of a neural network that converts a character image rather than character recognition, thereby making learning easier compared to a neural network for character recognition. That is, since the license plate recognition model 200 is learned using the license plate composite image synthesized into a single font by converting the license plate image, learning is relatively easy compared to the license plate recognition model learned with various fonts in the past, and the necessary learning data is Not only is it small, but there is no need to learn by adding new fonts.

The license plate recognition method using a neural network according to an embodiment of the present invention described above may be implemented in the form of a recording medium including instructions executable by a computer, such as a program module executed by a computer. Such recording media includes computer-readable media, and computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. Computer readable media also includes computer storage media, which include volatile and nonvolatile embodied in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, or other data. , both removable and non-removable media.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

100: license plate recognition system
110: communication module 120: memory
130: processor 140: database

Claims (9)

In the license plate recognition method using a neural network performed by the vehicle license plate recognition system,
a) when a vehicle image is obtained, detecting a license plate image including a license plate area from the vehicle image;
b) generating a license plate composite image by synthesizing the detected license plate image into a preset font using an image conversion algorithm;
c) learning the license plate recognition model using the learning data pairing the license plate composite image and the target vehicle number; and
d) When a vehicle image consisting of at least one or more letters and numbers is input, the license plate composite image generated through the image conversion algorithm is input to the learned license plate recognition model to recognize the vehicle number and then provide the recognized vehicle number comprising the steps of
The image conversion algorithm uses an Image to Image Translation (Pix2Pix) algorithm to synthesize a license plate image with a single font, a single background color, and a single character color, and provides a neural network to the license plate recognition model as a license plate composite image. How to recognize license plates. delete delete The method of claim 1,
Step b) is,
The license plate image is converted to a font having a preset thickness and size through the image conversion algorithm, and Bidirectional Long Short-Term Memory (BLSTM) based on an attention algorithm is applied to position letters or numbers Which further comprises the step of generating a license plate composite image by determining the, license plate recognition method using a neural network. 5. The method of claim 4,
Step b) is,
In the license plate image, a background area and a text area including numbers or letters are distinguished and detected, the background area is converted into a preset color, and the text area is converted to have a color different from the background area, a neural network license plate recognition method using 5. The method of claim 4,
A method for recognizing a license plate using a neural network, wherein each position of the determined letter or number is displayed as a separator, and when the vehicle number in which at least one or more letters or numbers are combined is completed, the vehicle number is displayed as an end separator. 7. The method of claim 6,
The delimiter or the end delimiter is to use any one or more of a special character or a special symbol, a license plate recognition method using a neural network. The method of claim 1,
The license plate recognition model is,
Extracting features from the license plate composite image, and using a CNN-LSTM (convolutional neural network-Long Short-Term Memory) complex model that sequentially integrates and uses the extracted features, a license plate recognition method using a neural network. A computer-readable recording medium in which a program for performing a license plate recognition method using a neural network according to any one of claims 1 to 8 is recorded.

Images