KR102194890B1 - Unmanned enforcement and road crime prevention method and system applied with vehicle license plate recognition technology using neural network - Google Patents

Abstract

Disclosed are an unmanned regulation and road security method employing vehicle number plate recognition technology using a neural network and a system thereof. The unmanned regulation and road security method employing vehicle number plate recognition technology using the neural network, which is performed by the vehicle number recognizing device including a processor, a memory, and a communication module for recognizing characters of the number plate of a vehicle, includes: a step of receiving, by the processor, an image including a classified character area; a step of outputting, by the processor, a first output feature map by performing a first convolution operation to the received image; a step of outputting, by the processor, a second output feature map by applying sampling to the outputted first output feature map; a step of outputting, by the processor, a third output feature map by performing a second convolution operation to the second output feature map; a step of outputting, by the processor, a fourth output feature map by applying sampling to the outputted third output feature map; and a step of recognizing, by the processor, characters from the classified character area by using a fully connected layer fully connecting the fourth output feature map with a preset feature map which can be defined by a numerical expression of features of the character area, and can be defined by the number of pixels of characters included in each area made by dividing the character area into predetermined areas, or the sum of white or black pixels located on a plurality of horizontal lines drawn in accordance with the height of the characters, or the sum of the white or black pixels located on a plurality of vertical lines drawn in accordance with the width of the characters.

Description

Unmanned enforcement and road crime prevention method and system applied with vehicle license plate recognition technology using neural network}

The present invention relates to an unmanned enforcement and road security method and system to which the vehicle license plate recognition technology using a neural network is applied, and in detail, the unmanned enforcement and road to which the vehicle license plate recognition technology is applied using a neural network that can improve the performance of vehicle license plate character recognition It relates to a security method and system.

Unmanned enforcement systems are used to control vehicle speeding and signal violations. The unmanned enforcement system captures a vehicle and recognizes the vehicle's license plate. Since the unmanned enforcement system is installed on an external road, the unmanned enforcement system is exposed to various external environments. For example, the amount or angle of light varies according to time, such as sunrise, sunset, and night. In bad weather conditions such as snow, rain, or fog, it may be difficult to recognize a vehicle's license plate. Damage to the license plate of the vehicle is also a factor that makes it difficult to recognize the license plate of the vehicle. In addition, since various license plate systems are used overseas, the license plate recognition algorithm applied in Korea cannot be applied as it is. As described above, there is a need for a new license plate recognition technology to solve the difficulty in recognizing the license plate of a vehicle speeding and violating the signal.

Korean Patent Publication No. 10-1999985 (2019.07.09.)

The technical problem to be achieved by the present invention is to provide a method and system for unmanned enforcement and road security using a vehicle license plate recognition technology by applying a neural network.

The unmanned enforcement and road security method to which the vehicle license plate recognition technology using a neural network performed by a vehicle number recognizer including a processor, a memory, and a communication module is applied to recognize characters on the vehicle license plate according to an embodiment of the present invention. Receiving, by a processor, an image including the classified character area, the processor performing a first convolution operation on the received image to output a first output feature map, and the processor is configured to perform a first convolution operation on the received image. Outputting a second output feature map by applying sampling to the output feature map, the processor outputting a third output feature map by performing a second convolution operation on the second output feature map, the processor Is defined as the step of applying sampling to the outputted third output feature map and outputting a fourth output feature map, and the processor expressing the outputted fourth output feature map and features of the character region as numerical values. The character area is divided into certain areas and is the number of pixels of characters included in each area, or is the sum of white pixels or black pixels located on horizontal lines drawn in several along the height of the character, or according to the width. A fully connected layer that completely connects white pixels located on several drawn vertical lines or a preset feature map that can be defined as the sum of black pixels It includes recognizing.

The processor is a second pulley for completely connecting a fifth output feature map output by performing convolution operations and sampling operations among CNN algorithms for data that may correspond to the character region and a second preset feature map. Classifying characters that may be confusing with each other by comparing the difference between the derived result values and a threshold value with respect to the result values derived by applying the connected layer, and the processor classifying the second characters according to the classified characters. Resetting a preset feature map to define a third preset feature map, and the processor includes a sixth output feature map output by applying the CNN algorithm to any one of the confusing characters, and the And recognizing a character by applying a third pulley connected layer that completely connects the third preset feature map.

The number of the fifth output feature map input to the second pulley connected layer and the number of the sixth output feature map input to the third pulley connected layer are different from each other, and input to the second pulley connected layer The number of second preset feature maps to be set and the number of third preset feature maps input to the third pulley connected layer are different from each other.

The fully connected layer includes an input layer, a hidden layer, and an output layer.

The hidden layer includes a plurality of hidden neurons.

)은 다음의 수학식으로 표현된다. When the number of features of the preset feature map is m (m is a natural number of 2 or more), and the number of features of the output fourth output feature is n (n is a natural number of 2 or more), the plurality of hidden neurons Of the jth (j is a natural number) hidden neuron input value (

) Is expressed by the following equation.

[Equation]

)은 다음의 수학식으로 표현된다. The output value of the j-th (j is a natural number) hidden neuron among the plurality of hidden neurons (

) Is expressed by the following equation.

[Equation]

는 상기 a번째 특징에 대한 j번째 가중치를, 상기

는 상기 미리 설정된 피처 맵의 특징들 중 a번째 특징을, 상기

는 상기 c번째 결과 값에 대한 j번째 가중치를, 상기

는 상기 출력된 제4출력 피처의 특징들 중 c번째 결과 값을, 상기

는 j번째 오프셋(offset)을, 상기 f( )는 시그모이드(sigmoid) 함수를 나타낸다. remind

Is the j-th weight for the a-th feature,

Is the a-th feature among the features of the preset feature map,

Is the j-th weight for the c-th result value,

Is the c-th result value among the features of the outputted fourth output feature,

Denotes a j-th offset, and f() denotes a sigmoid function.

The output layer includes a plurality of output neurons.

)은 다음의 수학식으로 표현된다. The input value of the k-th (k is a natural number) output neuron among the plurality of output neurons (

) Is expressed by the following equation.

[Equation]

)은 다음의 수학식으로 표현된다. The output value of the k-th (k is a natural number) output neuron among the plurality of output neurons (

) Is expressed by the following equation.

[Equation]

는 상기 j번째 은닉 뉴런의 출력 값에 대한 k번째 가중치를, 상기

는 상기 j번째 은닉 뉴런의 출력 값을, 상기

는 k번째 오프셋을, 상기 f( )는 시그모이드(sigmoid) 함수를 나타낸다.remind

Is the k-th weight for the output value of the j-th hidden neuron,

Is the output value of the j-th hidden neuron,

Denotes the k-th offset, and f () denotes a sigmoid function.

는 다음의 수학식으로 표현된다. The error of the j-th weight with respect to the output value of the i-th input neuron

Is expressed by the following equation.

[Equation]

는 상수로, 기존의 가중치를 줄이기 위해 가중치 감소 비율을, 상기

는 가중치를 어느 정도로 업데이트할 것인지 결정하는 학습율을 나타내며, 상기 E는

이며, 상기

는 복수의 출력 뉴런들 중 k번째 뉴런의 에러를 나타내며, 상기

는

으로 표현되며, 상기

는 입력 샘플 이미지에 대한 기대되는 출력 값이고, 상기

는 실제 출력 값을, 상기

는

으로 표현되며, 상기 t는 타임 스텝(time step)을 나타낸다. remind

Is a constant, the weight reduction ratio to reduce the existing weight,

Denotes the learning rate to determine how much to update the weights, and E is

Is, above

Represents the error of the k-th neuron among the plurality of output neurons,

Is

Expressed as, above

Is the expected output value for the input sample image, where

Is the actual output value,

Is

It is expressed as, and t represents a time step.

는 다음의 수학식으로 표현된다. The error of the k-th weight with respect to the output value of the j-th hidden neuron

Is expressed by the following equation.

[Equation]

는 상수로, 기존의 가중치를 줄이기 위해 가중치 감소 비율을, 상기

는 가중치를 어느 정도로 업데이트할 것인지 결정하는 학습율을, 상기 E는

이며, 상기

는 복수의 출력 뉴런들 중 k번째 뉴런의 에러를 나타내며, 상기

는

으로 표현되며, 상기

는 입력 샘플 이미지에 대한 기대되는 출력 값이고, 상기

는 실제 출력 값을, 상기

는

으로 표현되며, 상기 t는 타임 스텝(time step)을 나타낸다. remind

Is a constant, the weight reduction ratio to reduce the existing weight,

Is the learning rate that determines how much to update the weights, and E is

Is, above

Represents the error of the k-th neuron among the plurality of output neurons,

Is

Expressed as, above

Is the expected output value for the input sample image, where

Is the actual output value,

Is

It is expressed as, and t represents a time step.

An unmanned enforcement system according to an embodiment of the present invention includes a camera that photographs a license plate of a vehicle, and a vehicle number recognizer that photographs the license plate of the vehicle and recognizes the license plate of the vehicle from the photographed vehicle license plate image.

The vehicle number recognizer includes a communication module that transmits the recognized vehicle number to the unmanned enforcement system control server, a memory that stores instructions for vehicle license plate recognition technology using a neural network, and a processor that executes the instructions.

The commands receive an image including a classified character area processed in a license plate photographed by the camera, perform a first convolution operation on the received image, and output a first output feature map, and the output Sampling is applied to the first output feature map to output a second output feature map, a second convolution operation is performed on the second output feature map to output a third output feature map, and the output A sampling is applied to the 3 output feature map to output a fourth output feature map, and is defined as a numerical expression of the outputted fourth output feature map and the characteristics of the character area, and the character area is designated as predetermined areas. The number of pixels of the character included in each area after being divided, white pixels located on horizontal lines drawn in multiple lines along the height of the character, or the sum of black pixels, or white pixels located in vertical lines drawn in multiple lines depending on the width It is implemented to recognize a character in the classified character area by using a fully connected layer that completely connects a preset feature map that can be defined as a sum of fields or black pixels.

An unmanned enforcement and road security system according to an embodiment of the present invention includes an unmanned enforcement system and an unmanned enforcement system control server for communicating with the unmanned enforcement system through a network.

The unmanned enforcement system includes a camera that photographs the license plate of the vehicle, and a vehicle number recognizer that photographs the license plate of the vehicle and recognizes the license plate of the vehicle from the captured vehicle license plate image.

The vehicle number recognizer includes a communication module that transmits the recognized vehicle number to the unmanned enforcement system control server, a memory that stores instructions for vehicle license plate recognition technology using a neural network, and a processor that executes the instructions.

The commands receive an image including a classified character area processed in a license plate photographed by the camera, perform a first convolution operation on the received image, and output a first output feature map, and the output Sampling is applied to the first output feature map to output a second output feature map, a second convolution operation is performed on the second output feature map to output a third output feature map, and the output In the classified character area using a fully connected layer that completely connects the output 4 output feature map and the preset feature map by applying sampling to the 3 output feature map to output a fourth output feature map. It is implemented to recognize characters.

The unmanned enforcement and road crime prevention method and system to which the vehicle license plate recognition technology using a neural network according to an embodiment of the present invention is applied can recognize the vehicle license plate more accurately by using an algorithm combined with a neural network, and thus speeding in various environments. There is an effect that can increase the recognition rate of the license plate of the vehicle.

A detailed description of each drawing is provided in order to more fully understand the drawings cited in the detailed description of the present invention.
1 is a block diagram of an unmanned enforcement system according to an embodiment of the present invention.
FIG. 2 shows images for explaining the operation of the radar shown in FIG. 1.
3 shows a block diagram of an unmanned enforcement and road security system including the unmanned enforcement system shown in FIG. 1.
FIG. 4 is a flowchart illustrating a vehicle license plate recognition operation performed by the vehicle number recognizer shown in FIG. 1.
5 shows images processed by a processor of the vehicle number recognizer to describe the operation of the vehicle number recognizer shown in FIG. 1.
6 shows different images processed by the processor of the vehicle number recognizer to illustrate the operation of the vehicle number recognizer shown in FIG. 5.
7 shows a block diagram of a CNN algorithm processed by the processor shown in FIG. 1.
8 is a block diagram illustrating a relationship between an input feature map and an output feature map shown in FIG. 7.
9 shows an embodiment of a block diagram of a neural network algorithm processed by the processor shown in FIG. 1.
10 shows another embodiment of a block diagram of a neural network algorithm processed by the processor shown in FIG. 1.
11 shows another embodiment of a block diagram of a neural network algorithm processed by the processor illustrated in FIG. 1.
12 shows a graph of overfitting distortion according to learning of the neural network algorithm shown in FIG. 9.
13 shows a graph for solving the problem of overfit distortion shown in FIG. 12.
14 shows another embodiment of a block diagram of a neural network algorithm processed by the processor shown in FIG. 1.
15 are images showing differences between numbers that may be confusing.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed in the present specification are exemplified only for the purpose of describing the embodiments according to the concept of the present invention, and the embodiments according to the concept of the present invention are It may be implemented in various forms and is not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention can apply various changes and have various forms, the embodiments will be illustrated in the drawings and described in detail in the present specification. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all changes, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various elements, but the elements should not be limited by the terms. The terms are only for the purpose of distinguishing one component from other components, for example, without departing from the scope of the rights according to the concept of the present invention, the first component may be referred to as the second component, and similarly The second component may also be referred to as a first component.

When a component is referred to as being "connected" or "connected" to another component, it is understood that it may be directly connected or connected to the other component, but other components may exist in the middle. Should be. On the other hand, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there is no other component in the middle. Other expressions describing the relationship between components, such as "between" and "just between" or "adjacent to" and "directly adjacent to" should be interpreted as well.

The terms used in this specification are used only to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, action, component, part, or combination thereof is present, but one or more other features or numbers It is to be understood that the possibility of addition or presence of, steps, actions, components, parts, or combinations thereof is not preliminarily excluded.

differently

Unless defined, all terms, including technical or scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in this specification. Does not.

Hereinafter, the present invention will be described in detail by describing a preferred embodiment of the present invention with reference to the accompanying drawings.

1 is a block diagram of an unmanned enforcement system according to an embodiment of the present invention.

Referring to FIG. 1, the unmanned enforcement system 100 cracks down on a vehicle 103 running at speed, or cracks down on a vehicle 103 that violates a signal, or cracks down on a vehicle 103 running at speed in a section. System. The unmanned enforcement system 100 is a system to which number recognition technology using a neural network is applied. The unmanned enforcement system 100 can increase the license plate recognition rate of a vehicle by using a neural network, thereby increasing the efficiency of unmanned enforcement and road crime prevention.

The unmanned enforcement system 100 may include a radar 120, a camera 110, and a vehicle number recognizer 300.

The radar 120 and the camera 110 may be implemented at the end of another support implemented at the end of the support 101. The vehicle number recognizer 300 may be implemented in a separate box 301 located under the support 101. The radar 120, the camera 110, and the vehicle number recognizer 300 are connected to each other by wire or wirelessly. According to an embodiment, the unmanned enforcement system 100 may further include an auxiliary camera (not shown).

FIG. 2 shows images for explaining the operation of the radar shown in FIG. 1.

1 and 2, when the vehicle 103 travels toward the unmanned enforcement system 100, the radar 120 emits a radar signal and analyzes the radar signal reflected by the vehicle 103 and returned. To detect the vehicle 103. Radar 120 at different points of view

It is possible to determine the speed of the vehicle 103 by emitting a radar signal. The detection range of the vehicle 103 of the radar 120 is more than two lanes. Location information of the vehicle 103 may be detected through the virtual map image shown in FIG. 2. In the virtual map image shown in Fig. 2A, a white box indicates a virtual trigger area. When the vehicle enters the virtual trigger area illustrated in FIG. 2A, the radar 120 transmits lane information, trigger information, or speed information of the vehicle to the vehicle number recognizer 300. The lane information is information indicating whether the vehicle is traveling in one lane or two lanes, and the trigger information is information indicating a time when the vehicle enters the virtual trigger area.

The camera 110 continuously captures an image for a plurality of lanes. According to an embodiment, the camera 110 may further include an infrared LED for photographing at night.

The vehicle number recognizer 300 includes a central control unit (CCU) 310, an electrical control unit (ECU) 320, and a communication module 330. The CCU 310 continuously receives images captured from the camera 110. In addition, the CCU 310 receives the lane information and the trigger information from the radar 120. The CCU 310 uses the lane information and the trigger information to recognize a vehicle number using a neural network for each of the images received from the camera 110. The CCU 310 assigns a weight to the recognized vehicle number for a plurality of images, and derives a final vehicle number according to the assigned weight. According to an embodiment, the CCU 310 may be referred to as a processor. Hereinafter, the processor refers to the CCU 310. The vehicle number recognizer 300 may further include a memory (not shown) that stores commands for executing an operation of the CCU 310.

Vehicle license plate recognition technology using neural networks will be described in detail later. The ECU 320 detects a signal such as a traffic light and supplies power to the CCU 310, the camera 110, or the radar 120. The CCU 310, ECU 320, communication module 330, camera 110, and radar 120 are connected to each other by wire or wirelessly.

The CCU 310 includes the lane information, the trigger information, the speed information of the vehicle 103, the recognized number information of the vehicle 103, the enforcement date and time information, the image of the vehicle 103, the installation of the unmanned enforcement system 100 It generates data including location information and the like. The generated data is transmitted to an external server (not shown) through the communication module 330. The external server may be an unmanned enforcement system control server.

3 shows a block diagram of an unmanned enforcement and road security system including the unmanned enforcement system shown in FIG. 1.

1 and 3, the unmanned enforcement and road security system 400 includes at least one or more unmanned enforcement systems (100-1, 100-2, or 100-M; M is a natural number), and an unmanned enforcement system control It includes a server 410.

At least one or more unmanned enforcement systems 100-1, 100-2, or 100-M and the unmanned enforcement system control server 410 may communicate with each other through a wired or wireless network 401.

A plurality of unmanned enforcement systems 100-1 to 100-M may be installed at different locations. The unmanned enforcement system control server 410 includes the lane information, the trigger information, the vehicle 103 speed information, and the vehicle 103 from a plurality of unmanned enforcement systems 100-1 to 100-M installed at different locations. Receiving data including recognized number information, enforcement date and time information, image of the vehicle 103, installation location information of the unmanned enforcement system 100, etc., or a plurality of unmanned enforcement systems 100-1 to 100-M ) Each can be controlled. Controlling each of the plurality of unmanned enforcement systems 100-1 to 100-M means controlling the position of a camera included in each of the plurality of unmanned enforcement systems 100-1 to 100-M, or a vehicle number recognizer It is possible to control the power or operation of the 300. For example, the unmanned enforcement system control server 410 may control the unmanned enforcement system 100-1 to turn on the power of the unmanned enforcement system 100-1.

The unmanned enforcement system control server 410 may include a database 420. The database 420 may store the data transmitted through the network 401.

The unmanned enforcement system control server 410 is connected to a plurality of clients 430-1 to 430-N; N is a natural number to communicate with each other. The plurality of clients 430-1 to 430-N may monitor different unmanned enforcement systems 100-1 to 100-M in real time.

FIG. 4 is a flowchart illustrating a vehicle license plate recognition operation performed by the vehicle number recognizer shown in FIG. 1. 5 shows images processed by a processor of the vehicle number recognizer to describe the operation of the vehicle number recognizer shown in FIG. 1. Further, the operation of the processor refers to commands stored in a memory included in the vehicle number recognizer shown in FIG. 1.

1 to 5, the camera 110 photographs the license plate of the vehicle (S4001). Hereinafter, the processor 310 refers to the CCU 310.

The processor 310 binarizes the vehicle license plate image captured by the camera 110 (S4003). A specific algorithm for binarizing the vehicle license plate image is as follows.

The processor 310 extracts horizontal boundary points of text stroke candidates by applying a DoG (Difference of Gaussian) filter to the vehicle license plate image and generates a horizontal stroke boundary map (Horizontal Stroke Boundary Map). 5B shows an image of a horizontal stroke boundary point map. The processor 310 generates a segment image by grouping pixels between text strokes through vertical accumulation of horizontal boundary point connection lines. 5C shows a segment image. The processor 310 generates a color image so that each text has a different color in the segment image. 5D shows the color image. Depending on the embodiment, the color image generation operation may be omitted. The processor 310 converts the color image into a black and white image. 10A shows a black and white image.

Referring to FIG. 4, the processor 310 performs character blob analysis. The processor 310 indexes the pixel chunks, calculates location information and size information of the pixel chunks, analyzes the indexed pixel chunks, and extracts license plate candidate characters (S4004). The processor 310 extracts and cuts out the license plate area of the vehicle according to the analysis result (S4005). The processor 310 estimates the sharpness of the cut-out license plate area (S4006). The processor 310 estimates the sharpness, normalizes the size of the license plate and corrects the inclination (S4007). The processor 310 calculates the sharpness, and determines whether the calculated sharpness is greater than a predetermined value (S4008). When the calculated sharpness is greater than a predetermined value, the processor 310 decreases the sharpness (S4009). When the calculated sharpness is less than a predetermined value, the processor 310 improves the image of the license plate area of the vehicle (S4010).

The processor 310 selects a license plate type from the image including the license plate area of the vehicle (S4011). The license plate type may be divided into a single line license plate or a double line license plate. In addition, the license plate type may be classified by whether or not a place name is included. The processor 310 classifies the character area from the classified license plate area (S4012).

6 shows different images processed by the processor of the vehicle number recognizer to illustrate the operation of the vehicle number recognizer shown in FIG. 5. Specifically, FIG. 5 is an image showing the classification of an incorrect character area. In the classified license plate area, the classification operation of the character area is important. If the classification operation is wrong, it may be recognized as an incorrect character.

The processor 310 extracts a preset feature map from the classified character area (S4013). The preset feature map is defined as a numerical expression of character features. The preset feature map is used as an input in a neural network algorithm. For example, the preset feature map may divide a character area into predetermined areas and be defined as the number of pixels of characters included in each area. In addition, the preset feature map may be defined as a sum of white pixels or black pixels positioned on horizontal lines drawn in multiple numbers along the height of a character. In addition, the preset feature map may be defined as a sum of white pixels or black pixels positioned on vertical lines drawn in multiple numbers according to a width. The processor 310 outputs a CNN feature map by performing a CNN algorithm in the classified character region (S4014).

The processor 310 recognizes the vehicle license plate using a neural network algorithm (S4015). That is, the processor 310 recognizes the vehicle license plate by applying the ANN algorithm as inputs to the preset feature map and the CNN feature map output through the CNN algorithm.

The processor 310 recognizes the character string, determines whether or not it is misrecognized according to the type and arrangement rule of the characters, and outputs a final license plate character string recognition result (S4016).

The neural network algorithm will be described in detail later.

7 shows a block diagram of a CNN algorithm processed by the processor shown in FIG. 1.

1, 6, and 7, a convolutional neural network (CNN) is a deep learning algorithm and is mainly used to analyze an image.

The CNN algorithm includes a first convolution layer, a first max pooling layer, a second convolution layer, a second max pooling layer, and a fully connected layer. Includes (fully connected layer).

The processor 310 performs an operation of the first convolution layer. The operation in the first convolution layer is as follows.

The processor 310 receives an image including one recognized character area. The image including the one recognized character area includes one number, one English character, or one Hangul. The image is composed of pixel values having 0 or 1. The image is expressed as (horizontal size, vertical size, number of channels). When the image is binarized, the number of channels is 1. For example, the image shown in FIG. 6 may be expressed as (28, 28, 1).

The processor 310 outputs first convolution result values by performing a convolution operation on the received image and the first filter. The received image may be referred to as an input feature map. The first convolution result values may be referred to as a first output feature map. The first filter is implemented in a matrix form and is used to detect an edge in the received image or sharpen the received image. Depending on each purpose, the integer used in the matrix may vary. The first filter may be referred to as a kernel. The first filter may be expressed as (the horizontal size of the filter, the vertical size of the filter, the number of filter channels) x (the number of filters). For example, the first filter shown in FIG. 6 may be expressed as (5, 5, 1) x 16.

The convolution operation means convolution of the received image and the first filter. In this case, a weight may be multiplied by the first filter. The first output feature map resulting from the convolution operation may be expressed as (horizontal size, vertical size, number of filters).

When the input feature map that is the received image shown in FIG. 7 is (28, 28, 1) and the first filter is (5, 5, 1) x 16, the first output feature map is (24, 24, It can be expressed as 16).

8 is a block diagram illustrating a relationship between an input feature map and an output feature map shown in FIG. 7.

1 and 4 to 8, an input feature map is (l-1), an output feature map is (l), and a convolution operation using the filter bank w is as follows.

[Equation 1]

는 k번째 피처 맵의 c번째 필터의 u행, v열에 위치한 뉴런의 가중치를, 상기

는 입력 피처 맵(l-1)의 c 피처 맵의 (i+u)행, (j+v)열에 있는 뉴런의 출력을, 상기

는 l층의 k번째 피처 맵의 바이어스를, 상기

는 출력 피처 맵(l)의 k번째 출력 피처 맵에서 i행, j열에 위치한 뉴런의 출력을 의미한다. 상기 fc는 입력 피처 맵의 채널 개수, 상기 k는 필터 뱅크의 인덱스를, 상기 c는 입력 피처 맵의 인덱스를. 상기 u와 v는 필터의 가로, 세로 인덱스를 의미한다. remind

Is the weight of neurons located in row u and column v of the c-th filter of the k-th feature map,

Is the output of the neuron in the (i+u) row, (j+v) column of the c feature map of the input feature map (l-1),

Is the bias of the k-th feature map of the l layer,

Denotes the output of neurons located in row i and column j in the k-th output feature map of the output feature map (l). Wherein fc is the number of channels of the input feature map, k is the index of the filter bank, and c is the index of the input feature map. The u and v denote the horizontal and vertical indexes of the filter.

For example, the second output feature map (12, 12, 16) that is the output of the first max pooling layer is the input feature map (l-1), and the third output feature map (10, which is the output of the second convolution layer) 10, 24) may be the output feature map (l). In this case, the bank of the third filter (3, 3, 16) x 24 may be w.

Referring to FIG. 7, the processor 310 performs an operation of the first max pooling layer. The operation in the first max pooling layer refers to outputting a second output feature map by applying a second filter to a first output feature map that is a result value of the first convolution layer. The second filter may be expressed as (the horizontal size of the filter and the vertical size of the filter). For example, the second filter shown in FIG. 6 may be expressed as (2, 2).

When the first output feature map shown in FIG. 7 is (24, 24, 16) and the second filter is (2, 2), the second output feature map may be expressed as (12, 12, 16). .

The operation in the first max pooling layer may be expressed as the following equation.

[Equation 2]

Here, m and n denote the horizontal and vertical dimensions of the filter, respectively, and s and t denote the sampling area index of the input feature map.

The processor 310 performs an operation of the second convolution layer. The operation in the second convolution layer is similar to the operation in the first convolution layer. That is, the processor 310 outputs a third output feature map by convolving the second output feature map and the third filter, which are outputs of the first max pooling layer. The third filter may be a filter different from the first filter. When the second output feature map that is the received image shown in FIG. 6 is (12, 12, 16) and the third filter is (3, 3, 16) x 24, the third output feature map is (10, It can be expressed as 10, 24).

The processor 310 performs an operation of the second max pooling layer. The operation in the second max pooling layer is as follows. The operation in the second max pooling layer is similar to the operation in the first max pooling layer. That is, the processor 310 outputs the fourth output feature map by applying the fourth filter to the third output feature map, which is the output of the second convolution layer. The fourth filter may be a filter different from the second filter. When the third output feature map shown in FIG. 7 is (10, 10, 24) and the fourth filter is (2, 2), the fourth output feature map may be expressed as (5, 5, 24). .

)은 다음의 수학식과 같이 표현될 수 있다. The processor 310 performs an operation of a fully connected layer. The operation in the fully connected layer is as follows. The pulley connected layer includes a plurality of pulley connected layers. Each of the first pulley connected layer and the second pulley connected layer is expressed as (the number of input neurons and the number of output neurons). The number of input neurons in the first pulley-connected layer may be 600, and the number of output neurons in the first pulley-connected layer may be 200. The input neurons of the first pulley connected layer are derived from the fourth output feature maps 5, 5, and 24. J-th output neuron of the first pulley connected layer (

) Can be expressed as the following equation.

[Equation 3]

In addition, the number of input neurons in the second pulley-connected layer may be 200, which is the number of output neurons in the first pulley-connected layer, and 10, which is the number of output neurons in the second pulley-connected layer. An input value z and an output value of the sum of the weights of the k-th neuron of the second pulley connect layer may be expressed by the following equation.

[Equation 4]

The processor 310 receives a plurality of different images and repeats the operations of the CNN algorithm of the first convolution layer, the first max pooling layer, the second convolution layer, the second max pooling layer, and the fully connected layer. Train the CNN algorithm. According to learning, the weight of the fully connected layer and the filter weight of the convolutional layers are updated.

9 shows an embodiment of a block diagram of a neural network algorithm processed by the processor shown in FIG. 1. 10 shows another embodiment of a block diagram of a neural network algorithm processed by the processor shown in FIG. 1. 11 shows another embodiment of a block diagram of a neural network algorithm processed by the processor illustrated in FIG. 1.

1 and 9 to 11, the processor 310 receives an image including one recognized character area. The processor 310 outputs a first output feature map by performing a first convolution operation on the received image.

The processor 310 outputs a second output feature map by applying sampling to the first output feature map. That is, the processor 310 performs an operation of the first max pooling layer.

The processor 310 outputs a third output feature map by performing a second convolution operation on the output second output feature map.

The processor 310 outputs a fourth output feature map by applying sampling to the outputted third output feature map. That is, the processor 310 performs an operation of the second max pooling layer.

The processor 310 recognizes a character in the one recognized character area using a fully connected layer that completely connects the output fourth output feature map and a preset feature map.

The fully connected layer includes an input layer, a hidden layer, and an output layer.

The hidden layer includes a plurality of hidden neurons. The hidden layer may be expressed as (the number of input neurons of the hidden layer and the number of output neurons of the hidden layer). The output layer may be expressed as (the number of input neurons in the output layer and the number of output neurons in the output layer). For example, (840, 60) of the hidden layer shown in FIG. 9 means that the number of input neurons of the hidden layer is 840 and the number of output neurons of the hidden layer is 60. In addition, (60, 10) of the output layer shown in FIG. 9 means that the number of input neurons of the output layer is 60 and the number of output neurons of the output layer is 10.

)은 다음의 수학식으로 표현된다. When the number of features of the preset feature map is m (m is a natural number of 2 or more), and the number of features of the output fourth output feature is n (n is a natural number of 2 or more), the plurality of hidden neurons Of the jth (j is a natural number) hidden neuron input value (

) Is expressed by the following equation.

[Equation 5]

)은 다음의 수학식으로 표현된다. The output value of the j-th (j is a natural number) hidden neuron among the plurality of hidden neurons (

) Is expressed by the following equation.

[Equation 6]

는 상기 a번째 특징에 대한 j번째 가중치를, 상기

는 상기 미리 설정된 피처 맵의 특징들 중 a번째 특징을, 상기

는 상기 c번째 결과 값에 대한 j번째 가중치를, 상기

는 상기 출력된 제4출력 피처의 특징들 중 c번째 결과 값을, 상기

는 j번째 오프셋(offset)을 나타낸다. 상기 f( )는 시그모이드(sigmoid) 함수를 나타낸다. remind

Is the j-th weight for the a-th feature,

Is the a-th feature among the features of the preset feature map,

Is the j-th weight for the c-th result value,

Is the c-th result value among the features of the outputted fourth output feature,

Represents the j-th offset. The f() represents a sigmoid function.

Only features of the preset feature map are connected to each other, and only features of the output fourth output feature are connected to each other.

)은 다음의 수학식으로 표현된다. The output layer includes a plurality of output neurons, and an input value of the output neuron (k is a natural number) among the plurality of output neurons (

) Is expressed by the following equation.

[Equation 7]

)은 다음의 수학식으로 표현된다. The output value of the k-th (k is a natural number) output neuron among the plurality of output neurons (

) Is expressed by the following equation.

[Equation 8]

는 상기 j번째 은닉 뉴런의 출력 값에 대한 k번째 가중치를, 상기

는 상기 j번째 은닉 뉴런의 출력 값을, 상기

는 k번째 오프셋을 나타낸다. remind

Is the k-th weight for the output value of the j-th hidden neuron,

Is the output value of the j-th hidden neuron,

Represents the k-th offset.

와 10개의 뉴런들의 전체 에러 E는 다음의 수학식들로 표현된다. The error of the kth neuron among the plurality of output neurons

The total error E of and 10 neurons is expressed by the following equations.

[Equation 9]

는 입력 샘플 이미지에 대한 기대되는 출력 값이고, 상기

는 실제 출력 값을 나타낸다. remind

Is the expected output value for the input sample image, where

Represents the actual output value.

[Equation 10]

E represents the total error of 10 neurons.

The processor 310 updates the weight W in order to reduce the total error E even in a fully connected layer that completely connects the output fourth output feature map and the preset feature map.

An error of the k-th weight with respect to the output value of the j-th hidden neuron and the error of the j-th weight with respect to the output value of the i-th input neuron may be calculated by an error backpropagation algorithm. The error inversion plate algorithm is expressed by the following equations. In the conventional CNN algorithm, the error backpropagation algorithm was not used.

[Equation 11]

[Equation 12]

는 상기 j번째 은닉 뉴런의 출력 값에 대한 k번째 가중치의 오차를, 상기

는 상기 i번째 입력 뉴런의 출력 값에 대한 j번째 가중치의 오차를 나타낸다. 상기

은 학습율을 나타낸다. 상기 t는 타임 스텝(time step)을 나타낸다. remind

Is the error of the k-th weight with respect to the output value of the j-th hidden neuron,

Denotes an error of the j-th weight with respect to the output value of the i-th input neuron. remind

Represents the learning rate. The t denotes a time step.

The processor 310 calculates an error of the weights in the hidden layer and the output layer, and updates the existing weights using the following equation.

[Equation 13]

는 모멘템으로 가중치를 업데이트할 때, 기존의 업데이트 방향으로 관성이 유지되도록하는 상수이다. 상기 t는 타임 스텝(time step)을 나타낸다. remind

Is a constant that keeps the inertia in the existing update direction when updating the weight with the momentum. The t denotes a time step.

The unmanned enforcement and road security system 400 using a neural network recognizes the vehicle license plate using a neural network algorithm different from the conventional CNN, so that it can accurately recognize the vehicle license plate rather than recognizing the vehicle license plate using only the CNN algorithm. have.

12 shows a graph of overfitting distortion according to learning of the neural network algorithm shown in FIG. 9.

Referring to FIG. 12, when a neural network algorithm is repeatedly trained to update weights, a crystal plane may be distorted as shown in FIG. 12. In particular, the ability to classify samples in the vicinity of the crystal plane may deteriorate. A phenomenon in which the classification ability is deteriorated by excessively updating the weights is called over-fitting or overfitting.

13 shows a graph for solving the problem of overfit distortion shown in FIG. 12. In FIG. 13, the X-axis indicates the number of repetitive learning, and the Y-axis indicates accuracy.

Referring to FIGS. 1 and 13, the accuracy of object classification does not increase as the number of repetitive learning increases. When the number of repetitive learning is excessively performed, a problem of over-fitting may occur. When the number of iterative learning is performed too little, a problem of under-fitting, which is less accurate, may occur.

To solve the over-fitting problem, the processor 310 applies the following equations.

[Equation 14]

는 상기 i번째 입력 뉴런의 출력 값에 대한 j번째 가중치의 오차를 , 상기

는 상수로, 기존의 가중치를 줄이기 위해 가중치 감소 비율을, 상기

는 가중치를 어느 정도로 업데이트할 것인지 결정하는 학습율을 나타낸다. 상기

는 상기 i번째 입력 뉴런의 출력 값에 대한 j번째 가중치를 나타낸다. remind

Is the error of the j-th weight with respect to the output value of the i-th input neuron,

Is a constant, the weight reduction ratio to reduce the existing weight,

Denotes a learning rate that determines how much to update the weights. remind

Represents the j-th weight for the output value of the i-th input neuron.

[Equation 15]

는 상기 j번째 은닉 뉴런의 출력 값에 대한 k번째 가중치의 오차를, 상기

는 상수로, 기존의 가중치를 줄이기 위해 가중치 감소 비율을, 상기

는 가중치를 어느 정도로 업데이트할 것인지 결정하는 학습율을, 상기

는 복수의 출력 뉴런들 중 k번째 뉴런의 에러를 나타낸다. 상기

는 상기 j번째 은닉 뉴런의 출력 값에 대한 k번째 가중치를 나타낸다. remind

Is the error of the k-th weight with respect to the output value of the j-th hidden neuron,

Is a constant, the weight reduction ratio to reduce the existing weight,

Is the learning rate that determines how much to update the weights,

Represents the error of the k-th neuron among the plurality of output neurons. remind

Represents the k-th weight for the output value of the j-th hidden neuron.

Equations 14 and 15 are different from Equations 11 and 12. When updating the weights using Equations 14 and 15, the over-fitting problem can be solved by reducing the weight to some extent for a large weight.

14 shows another embodiment of a block diagram of a neural network algorithm processed by the processor shown in FIG. 1.

1 and 14, the processor 310 compares a difference and a threshold value between output result values using a fully connected layer that completely connects the output fourth output feature map and the preset feature map. Thus, it is determined whether there is a possibility of confusion in recognizing a character other than an expected character through the output result values. The possibility of confusion is determined by the classifier shown in FIG. 15. For example, numbers 3 and 8 can be confusing due to the influence of ambient light or the angle of the camera. In the image including the number 3, the character expected through recognition is 3, but it can be recognized as another character 8. Conversely, when the expected character is 8, it may be recognized as another character 3.

For the image including the number 3, result values derived by applying a fully connected layer that completely connects the output feature map and the preset feature map may have a probability of being number 3 being 0.7 and a probability of being number 8 being 0.6. . When the difference (0.1) between the derived result values (eg, 0.7 and 0.6) is less than the threshold value (eg, 0.2), it may be determined that there is a possibility of confusion with other characters. Accordingly, it may be determined that there is a possibility of confusion with the number 8 other than the expected number 3 through the result values derived by applying the fully connected layer.

When it is determined through the output result values that there is a possibility of confusion in recognizing a character other than the expected character, the processor 310 resets the preset feature map to define a second preset feature map. In this case, the preset feature map may be referred to as a first preset feature map.

The resetting means increasing the number of preset feature maps to clearly distinguish characters that may be confusing. Feature maps (e.g., one pixel on the left and one pixel on the right, based on the center of the character) that can indicate the characteristics of the characters (e.g., symmetry, or whether lines are connected) that can be distinguished from each other in potentially confusing characters (e.g., 3 and 8). Whether one pixel corresponds to each other, or whether black pixels are continuously connected) may be added.

The first preset feature map and the second preset feature map are used as inputs in an ANN structured neural network. For example, the first and second preset feature maps may divide a character area into predetermined areas and be defined as the number of pixels of characters included in each area. In addition, the first and second preset feature maps may be defined as the sum of white pixels or black pixels positioned on horizontal lines drawn in multiple numbers along the height of a character. In addition, it may be defined as a sum of white pixels or black pixels positioned on vertical lines drawn in multiple numbers according to the first and second preset feature map widths. However, the second preset feature map is different from the first preset feature map in that it is set to clearly distinguish characters that may be confusing.

15 are images showing differences between numbers that may be confusing.

Referring to FIG. 15, the number 8 is symmetric about the red line, and the number 3 is not symmetric about the red line. The second preset feature map may be set to determine the symmetry of these numbers.

Further, referring to FIG. 15, black pixels are connected to the left square of the number 8, but black pixels are not connected to the left square of the number 3. According to an embodiment, the second preset feature map may be set to determine the connectivity of such pixels.

The processor 310 provides a second fully connected layer that completely connects the fifth output feature map output by re-applying the CNN algorithm to the image including the classified character region and the second preset feature map. Apply to recognize characters. A pulley connected layer that completely connects the output fourth output feature map and the first preset feature map may be referred to as a first pulley connected layer.

In order to clearly recognize the potentially confusing characters, an operation of applying the second pulley connected layer is performed. The applied CNN algorithm is the same as the weights used in the process of outputting the fourth output feature map.

The number of the fourth output feature maps input to the first pulley-connected layer and the number of fifth output feature maps input to the second pulley-connected layer are different from each other. For example, the number of the fourth output feature map may be greater than the number of the fifth output feature map. When the number of the fourth output feature maps is n, the number of the fifth output feature maps may be q (q is a natural number less than n).

The number of first preset feature maps input to the first pulley connected layer and the number of second preset feature maps input to the second pulley connected layer are different from each other. For example, the number of first preset feature maps may be smaller than the number of second preset feature maps. When the number of first preset feature maps is m, the number of second preset feature maps may be p (p is a natural number greater than m). By making the number of the second preset feature maps greater than the number of the first preset feature maps, characters that may be confused can be clearly identified. That is, by increasing the number of the second preset feature maps, more character features than the first preset feature map may be extracted.

The sum of the number of the fourth output feature map and the number of the first preset feature map, the number of the fifth output feature map and the number of the second preset feature map are the same.

Weights in the hidden layer and the output layer of the first pulley-connected layer and the weights in the hidden layer and the output layer of the second pulley-connected layer are the same.

Although the present invention has been described with reference to the exemplary embodiment shown in the drawings, this is only exemplary, and those of ordinary skill in the art will appreciate that various modifications and other equivalent embodiments are possible therefrom. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the attached registration claims.

100: unmanned enforcement system;
110: camera;
120: radar;
300: vehicle number recognizer;
410: unmanned enforcement system control server;
420: database;
430-1-430-N: client;

Claims (5)

The unmanned enforcement and road security method to which the vehicle license plate recognition technology is applied using a neural network performed by a vehicle number recognizer including a processor, a memory, and a communication module in order to recognize characters on the vehicle license plate,
The processor receiving an image including the classified text area;
The processor performing a first convolution operation on the received image to output a first output feature map;
The processor outputting a second output feature map by applying sampling to the output first output feature map;
The processor performing a second convolution operation on the second output feature map to output a third output feature map;
The processor outputting a fourth output feature map by applying sampling to the outputted third output feature map; And
The processor is defined as a numerical expression of the outputted fourth output feature map and characteristics of the character area, and divides the character area into predetermined areas and is the number of pixels of characters included in each area, or the height of the character A preset feature map defined as the sum of white pixels or black pixels located on multiple horizontal lines along the line, or the sum of white pixels or black pixels located on multiple vertical lines depending on the width is completely completed. An unmanned enforcement and road security method to which a vehicle license plate recognition technology using a neural network is applied, comprising the step of recognizing a character in the classified character region using a fully connected layer to connect. The method of claim 1, wherein the fully connected layer,
Includes an input layer, a hidden layer, and an output layer,
The hidden layer includes a plurality of hidden neurons (neurons),
When the number of features of the preset feature map is m (m is a natural number of 2 or more), and the number of features of the output fourth output feature is n (n is a natural number of 2 or more),
The input value of the j-th (j is a natural number) hidden neuron among the plurality of hidden neurons (

) Is expressed by the following equation,
[Equation]

The output value of the j-th (j is a natural number) hidden neuron among the plurality of hidden neurons (

) Is expressed by the following equation,
[Equation]

remind

Is the j-th weight for the a-th feature,

Is the a-th feature among the features of the preset feature map,

Is the j-th weight for the c-th result value,

Is the c-th result value among the features of the outputted fourth output feature,

Denotes the j-th offset, and f() denotes a sigmoid function. A vehicle license plate recognition technology applied using a neural network is applied. The method of claim 2,
The output layer includes a plurality of output neurons,
The input value of the k-th (k is a natural number) output neuron among the plurality of output neurons (

) Is expressed by the following equation,
[Equation]

The output value of the k-th (k is a natural number) output neuron among the plurality of output neurons (

) Is expressed by the following equation,
[Equation]

remind

Is the k-th weight for the output value of the j-th hidden neuron,

Is the output value of the j-th hidden neuron,

Denotes the k-th offset, and f() denotes a sigmoid function. A vehicle license plate recognition technology is applied to the unmanned enforcement and road security method. The method of claim 3,
The error of the j-th weight with respect to the output value of the i-th input neuron

Is expressed by the following equation,
[Equation]

remind

Is a constant, the weight reduction ratio to reduce the existing weight,

Denotes the learning rate to determine how much to update the weights, and E is

Is, above

Represents the error of the k-th neuron among the plurality of output neurons,

Is

Expressed as, above

Is the expected output value for the input sample image, where

Is the actual output value,

Is

Is expressed as, wherein t represents a time step,
The error of the k-th weight with respect to the output value of the j-th hidden neuron

Is expressed by the following equation,
[Equation]

remind

Is a constant, the weight reduction ratio to reduce the existing weight,

Is the learning rate that determines how much to update the weights, and E is

Is, above

Represents the error of the k-th neuron among the plurality of output neurons,

Is

Expressed as, above

Is the expected output value for the input sample image, where

Is the actual output value,

Is

And t is an unmanned enforcement and road crime prevention method to which vehicle license plate recognition technology using a neural network representing a time step is applied. A camera for photographing the license plate of the vehicle; And
A vehicle number recognizer for photographing the license plate of the vehicle and recognizing the vehicle license plate from the photographed vehicle license plate image,
The vehicle number recognizer,
A communication module for transmitting the recognized vehicle number to the unmanned enforcement system control server;
A memory for storing instructions related to vehicle license plate recognition technology using a neural network; And
And a processor that executes the instructions,
The above commands are:
Receives an image including the classified text area processed by the license plate photographed by the camera, performs a first convolution operation on the received image to output a first output feature map, and outputs the first Sampling is applied to the output feature map to output a second output feature map, a second convolution operation is performed on the second output feature map to output a third output feature map, and the output third output feature Sampling is applied to the map to output a fourth output feature map, and is defined as a numerical expression of the outputted fourth output feature map and the characteristics of the character area, and the character area is divided into certain areas and each area The number of pixels included in the character, white pixels located on horizontal lines drawn in multiple numbers along the height of the character, or the sum of black pixels, white pixels located in vertical lines drawn in multiple lines depending on the width, or An unmanned enforcement system implemented to recognize a character in the classified character area using a fully connected layer that completely connects a preset feature map defined as a sum of black pixels.

Images