KR101986592B1 - Recognition method of license plate number using anchor box and cnn and apparatus using thereof - Google Patents

Abstract

A license plate number recognition apparatus, according to the present invention, comprises: an image input unit for receiving a first image; a second image generating unit for generating a second image by detecting a region of interest including characters from the first image; a feature component extracting unit for extracting feature component from the second image; an anchor box generating unit for generating a plurality of anchor boxes having the same vertical size as the second image and a horizontal size different from that of the second image, from the second image; a bounding box generating unit for comparing the feature component of the second image, corresponding to the areas of the anchor boxes, with character detection learning data to select an anchor box having the maximum matching value as a bounding box for each character area; and a character recognition unit for comparing the feature component of the area corresponding to the bounding box with the character recognition learning data to recognize the characters corresponding to the anchor box. The anchor boxes each include: an end anchor boxes (where a is a natural number) positioned to overlap each other at predetermined intervals over all areas of the second image and having the same horizontal and vertical sizes; a base anchor boxes positioned on one sides of the end anchor boxes and having a horizontal size smaller than a vertical size; and a plurality of sub-anchor boxes including the areas of the base anchor boxes and having a size that is increased than the horizontal size of each of the base anchor boxes by a predetermined size while having a horizontal size smaller than the horizontal size of the end anchor box. Thus, the license plate number can be recognized more accurately.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and method for recognizing a car number using an anchor box and a CNN feature map,

BACKGROUND OF THE INVENTION 1. Field of the Invention [0002] The present invention relates to a vehicle number recognition apparatus and a vehicle number recognition method using artificial intelligence. And more particularly, to an apparatus and method for recognizing a car number using an anchor box and a CNN feature map capable of accurately recognizing characters from a video image using an anchor box and a CNN feature map.

A vehicle license plate is a plate which is attached to the front or rear surface of a vehicle and in which standardized characters (including letters, numbers and symbols) are written in a specific pattern. The license plate recognition device automatically recognizes license plates and characters Means a device for reading. A general license plate recognition apparatus includes a step of car shooting, a license plate extraction, a preprocessing process according to shade or tilt, and recognizing the license plate according to a predetermined standard letter. Such a vehicle license plate recognizing device necessarily carries out a vehicle number recognizing device or a car number recognizing method for recognizing characters written on the license plate, so that a more accurate car number recognizing method is a very important factor in a license plate recognizing device.

Conventional license plate recognition uses a circular matching method and a syntactic method, and the circular matching method improves the geometrical matching method and refers to a method of recognizing characters by matching images inputted in a standard pattern. However, if there is a slant image or noise, the recognition rate is degraded. If the environment is changed, the standard pattern must be reconstructed. Syntactic methods use structural information such as interrelationship between character characteristics or interconnection information Character size, and slope. Especially, it is difficult to quantitatively extract structural information, and it is not easy to obtain accurate structural information between features.

Recently, much attention has been focused on the license plate recognition method using neural network. An artificial neural network (ANN) is a statistical learning algorithm inspired by the neural network of biology. An artificial neural network is a system in which artificial neurons (nodes) And refers to the overall model that has problem solving ability. Deep learning technology is a field of machine learning based on artificial neural networks in the form of a multi-layer structure. It is a set of machine learning algorithms that try to achieve a high level of abstraction through a combination of various nonlinear transformation techniques.

The method using the neural network divides the area of the input image into the input of the neural network and recognizes the character using the neural network. Therefore, the method using the neural network is very important to select the input area. Therefore, since the performance is greatly influenced by the selection of the input area, a lot of studies on the method of selecting the area occupied by the characters are required.

Korean Patent Registration No. 10-1931804 (Dec. 17, 2018)

SUMMARY OF THE INVENTION Accordingly, the present invention has been made keeping in mind the above problems occurring in the prior art, and it is an object of the present invention to provide a vehicle number recognition apparatus using an anchor box and a CNN feature map capable of accurately recognizing characters from an image using an anchor box and an artificial neural network .

Another object of the present invention is to provide a vehicle number recognition method using an anchor box and a CNN feature map that can more accurately recognize a vehicle number using an anchor box and an artificial neural network.

In order to achieve the object of the present invention, an apparatus for recognizing a car number using an anchor box and a CNN feature map includes an image input unit for receiving a first image, a second region for detecting a region of interest including a character from the first image, A feature extraction unit for extracting a feature component from the second image, a second feature extraction unit for extracting a feature component from the second image, An anchor box generating unit for generating a plurality of anchor boxes different from each other, and an anchor box having a maximum matching value by comparing feature data of the second image corresponding to the anchor boxes with character learning data A bounding box generating unit for selecting a bounding box of each character region, And a character recognizing unit for recognizing characters corresponding to the anchor box by comparing the character recognition learning data with the character recognition learning data, wherein the anchor boxes are overlapped at all of the areas of the second video image at regular intervals, (A is a natural number) having the same size, a base anchor box located at one side of the end anchor box and having a lateral size smaller than the vertical size, and an area of the base anchor box, And a plurality of sub-anchor boxes having a size smaller than a width of the end anchor box and having a size that is increased by a predetermined size in the width of the base anchor box.

In one embodiment of the present invention, the method for generating the anchor box includes generating a base anchor box located at one side of the second video image, Creating the sub-anchor box including the area of the anchor box generated immediately before and increasing the size of the anchor box to a predetermined size from the size of the previously generated anchor box, Creating an end anchor box including an area of a box, creating a new base anchor box with a horizontal position shifted by a predetermined size in a previously generated base anchor box, creating the sub anchor box, Performing the step of creating the end anchor box again, and creating the new base anchor box And repeating the step of repeating until the end anchor box located on the most opposite side of one side of the second video image is generated.

In one embodiment of the present invention, the size of the anchor box is in pixels, and the size of the base anchor box (the number of pixels) is 1 / r (number of pixels) of the size of the end anchor box And a constant size of the increasing width of the sub-anchor box is p pixels, and a constant interval of a start position between the end boxes adjacent to the end a boxes among the a end achor boxes may be q pixels (q is a natural number).

In an embodiment of the present invention, when r is 2, the total number of anchor boxes generated from one second image may be calculated by the following equation.

Here, N is the vertical size of the second video image and M is the horizontal size of the second video image.

In one embodiment of the present invention, the car number recognition apparatus further includes a vehicle detection section that detects a vehicle area from the first video image using an SDD (Single shot MultiBox Detector) algorithm, and the second image generation section The vehicle license plate area, which is the area of interest, is detected from the detected vehicle area using a SDD (Single shot MultiBox Detector) algorithm to generate a second image.

In an exemplary embodiment of the present invention, the image processing apparatus may further include a preprocessor for enhancing image quality by performing intensity balancing and morphology on the second image, wherein the morphology includes a closed morphology and an open morphology, The histogram of the image gray level is calculated and the pixel values corresponding to the lower quartile and the upper quartile are calculated from the histogram and an affine transform can be performed by the following equation.

Here, x is a pixel value at the corresponding position, max is a maximum pixel value in the corresponding region, min is a minimum pixel value in the corresponding region, Vmin is a pixel value corresponding to a lower quartile, and Vmax corresponds to a higher quartile Pixel values.

In one embodiment of the present invention, the car number recognition apparatus may further include a projection unit for calculating an upper position and a lower position of the second image by using a horizontal projection method.

In one embodiment of the present invention, the feature extraction unit extracts a convolution feature map of the second image using a Convolutional Neural Network (CNN) And the combined-product neural network may include an input layer, a convolution layer, and a pooling layer.

In an embodiment of the present invention, the character recognition unit can recognize a character corresponding to the anchor box using a fully connected network.

The method for recognizing a car number using an anchor box and a CNN feature map for realizing the object of the present invention includes the steps of receiving a first video image from a video input unit, Extracting a feature component from the second video image; extracting a feature component from the second video image by extracting a feature component from the second video image; Generating a plurality of anchor boxes having the same size and different widths from each other, and the bounding box generating unit is configured to classify the feature components of the second video image corresponding to the area of the anchor boxes into character detection learning data Selecting an anchor box having a maximum matching value as a bounding box of each character area; And recognizing the character corresponding to the anchor box by comparing the feature component of the area corresponding to the bounding box with the character recognition learning data, A number of base anchor boxes (a is a natural number) located at one side of the end anchor box and having a size smaller than the vertical size, And a plurality of sub-anchor boxes including an area of the base anchor box, the size of which is smaller than the width of the end anchor box, and the size of the sub anchor box is increased by a predetermined size in the width of the base anchor box .

In one embodiment of the present invention, the step of creating an anchor box may include the steps of creating a base anchor box located at one side of the second video image, Creating the sub-anchor box that includes the area of the anchor box generated immediately before, and the size of the sub-anchor box that is increased in size by a predetermined size from the size of the anchor box created immediately before; Creating a new base anchor box in which a horizontal position is shifted by a predetermined size in a previously generated base anchor box;

Creating the sub-anchor box and creating the end anchor box again; and creating the new base anchor box, and performing the re-performing step on the opposite side of one side of the second video image And repeating until an end anchor box positioned is created.

In one embodiment of the present invention, the size of the anchor box is in pixels, and the size of the base anchor box (the number of pixels) is 1 / r (number of pixels) of the size of the end anchor box The constant size of the increasing width of the sub-anchor box is p pixels, and a certain interval of the start position between the end-boxes closest to the a end-anchor boxes may be q pixels.

In an embodiment of the present invention, when r is 2, the total number of anchor boxes generated from one second image may be calculated by the following equation.

Here, N is the vertical size of the second video image and M is the horizontal size of the second video image.

In one embodiment of the present invention, the car number recognition method further includes the step of the vehicle detection unit detecting a vehicle area using the SDO (Single shot MultiBox Detector) algorithm from the first image, In the step of generating an image, a second image may be generated by detecting a vehicle license plate area, which is the area of interest, from the detected vehicle area using an SDD (Single shot MultiBox Detector) algorithm.

In one embodiment of the present invention, the vehicle number recognition method may further include a step of the image processing unit performing intensity balancing and morphology on the second image image to improve the image quality, Performing a closed morphology, performing an open morphology following the closed morphology, calculating a histogram for an image gray level, calculating a pixel value corresponding to a lower quartile and an upper quartile from the histogram And performing an affine transform using the following equation: < EMI ID = 1.0 >

Here, x is a pixel value at the corresponding position, max is a maximum pixel value in the corresponding region, min is a minimum pixel value in the corresponding region, Vmin is a pixel value corresponding to a lower quartile, and Vmax corresponds to a higher quartile Pixel values.

In one embodiment of the present invention, the vehicle number recognition method may further include a step of calculating an upper position and a lower position of the second image by using a horizontal projection.

In one embodiment of the present invention, in extracting the feature component, a convolution feature map of the second image is extracted using a CNN (Convolutional Neural Network) And the composite neural network may include an input layer, a convolution layer, and a pooling layer.

In one embodiment of the present invention, in the step of recognizing the character, a character corresponding to the anchor box can be recognized using a fully connected network.

According to embodiments of the present invention, a car number recognition apparatus and a car number recognition method using an anchor box and a CNN feature map may include an image input unit, a second image image generation unit, a feature component extraction unit, an anchor box generation unit, And a character recognition unit. Therefore, an anchor box is generated from the image, and a more accurate vehicle number can be recognized by comparing the feature component of the portion corresponding to the anchor box with the character recognition learning data.

In addition, since the present invention creates a bounding box that is a region where characters are located from a plurality of anchor boxes of various sizes, an area can be variably selected according to characteristics of characters, The vehicle number can be recognized.

In addition, the present invention extracts character regions using artificial neural network technology, and recognizes characters, thereby providing high scalability and accuracy of character recognition.

1 is a configuration diagram illustrating a vehicle number recognition apparatus using an anchor box and a CNN feature map according to an embodiment of the present invention.
2 is a configuration diagram illustrating a vehicle number recognition apparatus using an anchor box and a CNN feature map according to an embodiment of the present invention.
3 is a flowchart illustrating a method of recognizing a car number using an anchor box and a CNN feature map according to an embodiment of the present invention.
4 is a flowchart illustrating a method of recognizing a car number using an anchor box and a CNN feature map according to an embodiment of the present invention.
5 is a flowchart illustrating a step of improving the image quality of the car number recognition method using the anchor box and the CNN feature map according to an embodiment of the present invention.
FIG. 6 is a flowchart illustrating a step of generating an anchor box of an apparatus and method for recognizing a car number using an anchor box and a CNN feature map according to an embodiment of the present invention.
7 is a view showing an anchor box of an apparatus and method for recognizing a car number using an anchor box and a CNN feature map according to an embodiment of the present invention.
8 is a view showing an anchor box of an apparatus and method for recognizing a car number using an anchor box and a CNN feature map according to an embodiment of the present invention.
9 is a view showing an embodiment of an apparatus and method for recognizing a car number using an anchor box and a CNN feature map according to an embodiment of the present invention.
10 is a diagram showing a general SSD method.
11 is a view showing an embodiment of a second image of an apparatus and method for recognizing a car number using an anchor box and a CNN feature map according to an embodiment of the present invention.
12 is a view showing an embodiment of CNN of an apparatus and method for recognizing a car number using an anchor box and a CNN feature map according to an embodiment of the present invention.
13 is a view showing an embodiment of a fully connected network of an apparatus and method for recognizing a car number using an anchor box and a CNN feature map according to an embodiment of the present invention.

While the present invention has been described in connection with what is presently considered to be the most practical and preferred embodiment, it is to be understood that the invention is not limited to the disclosed embodiments. It is to be understood, however, that the invention is not intended to be limited to the particular forms disclosed, but on the contrary, is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. Like reference numerals are used for like elements in describing each drawing. The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms.

The terms are used only for the purpose of distinguishing one component from another. The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise.

In the present application, the term "comprises" or "comprising ", etc. is intended to specify that there is a stated feature, figure, step, operation, component, But do not preclude the presence or addition of one or more other features, integers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

In the present invention, letters include letters, numbers and symbols.

1 is a configuration diagram illustrating a vehicle number recognition apparatus using an anchor box and a CNN feature map according to an embodiment of the present invention.

1, an apparatus for recognizing a car number using an anchor box and a CNN feature map according to an embodiment of the present invention includes an image input unit 100, a second image generation unit 200, a feature extraction unit 300, An anchor box generation unit 400, a bounding box generation unit 500, and a character recognition unit 600.

The image input unit 100 may receive a first image. For example, the image input unit 100 may be a camera, a CCTV, or an image sensor. For example, the image input unit 100 may be an IP camera. For example, the image input unit 100 may be a camera installed on a road and photographing a front surface or a rear surface of the vehicle. The first video image may include characters having the same standard. The first video image may include characters arranged in a line having the same standard. For example, the first video image may include numbers, letters, or signs. For example, as shown in FIG. 11, a vehicle license plate may be included in the first image.

The second image generator 200 may generate a second image by detecting a region of interest including a character from the first image. For example, as shown in FIG. 11, the area of interest may be the area of a license plate. For detection of the region of interest, a deep learning technique may be used. For example, as shown in FIG. 9, the area of interest is an area of a vehicle license plate, and the deep learning technique is a single shot multibox detector (SSD), and the vehicle license plate can be detected using license plate detection learning data. The SSD technology used for vehicle license plate detection is a known technology as shown in FIG. 10, so a detailed description will be omitted.

The feature extraction unit 300 may extract a feature component from the second image. For example, the feature extraction unit 300 extracts a convolution feature map of the second image by using a CNN (Convolutional Neural Network), and extracts a convolution feature map of the second image, can do.

The composite neural network may include an input layer, a convolution layer, a pooling layer, and a feature map. A plurality of the composite product layer and the pooling layer may be used. The convolution layer may transform an image by applying a filter to the input data and then reflecting the activation function. The pooling layer may be a subsampling layer that reduces the dimension of the image. Thus, the pooling layer may be omitted.

For example, as shown in FIG. 12, the input image layer may normalize the second image to 112x112, use a gray level channel (@ 1 or depth 1), and in the composite product layer, The normalized second video image can be converted into a size of 56x56, and in the pooling layer, the converted second video image can be converted into a 28x28 size using a gray level 32 channel, , The second video image converted in the pooling layer can be converted into a 14x14 size using the gray level 64 channel, and in the next pooling layer, the second video image converted in the next composite product layer can be converted into the gray level 128 channel , And an image having a final converted 7x7 size and 128 depth can be converted into a 7x7 image using the above It can be used as a gong map.

The anchor box generation unit 400 may generate a plurality of anchor boxes from the second image. The plurality of anchor boxes may have the same vertical size as the second video image and have different horizontal sizes. The anchor box may be an arbitrary division area for character recognition.

The anchor boxes may include an end anchor box 403, a base anchor box 401, and a sub anchor box 402.

The end anchor box 403 may have the same width and the same length. For example, it may be an anchor box having a horizontal size and a vertical size N as shown in FIG. The end anchor box 403 may be a plurality of end anchor boxes. For example, the number of the end anchor boxes 403 may be a (a is a natural number). The end anchor box 403 may be overlaid over all areas of the second video image at regular intervals. For example, if the size of the second video image is M, the size of the vertical is N, and the end anchor boxes 403 are arranged at intervals of size 1 (r = 1) as shown in FIG. 8, The number a of the boxes 403 may be (M-N + 1).

The width of the base anchor box 401 may be smaller than the length of the base anchor box 401. For example, as shown in FIG. 7, the width of the base anchor box 401 may be 1 / r times the vertical length. For example, the width of the base anchor box 401 may be 1/2 times the length of the base anchor box 401. The base anchor box 401 may be located at one side of the end anchor box 403. The base anchor box 401 may be formed in the same number as the end anchor box 403. For example, when the size of the second video image is M, the size of the vertical is N, and the base anchor boxes 401 are arranged at intervals of size 1 (r = 1) as shown in FIG. 8, The number a of boxes 401 may be (M-N + 1).

The width of the sub-anchor box 402 may be smaller than the width of the end anchor box 403, but may be increased by a certain amount in the width of the base anchor box 401. A plurality of sub-anchor boxes 402 may be generated for one base anchor box 401. For example, as shown in FIG. 7, the width of the sub-anchor boxes 402 may be 1 / r of the vertical size plus n times of p.

The sub-anchor box 402 may include an area of the base anchor box 401. For example, as shown in FIG. 7, the sub-anchor boxes 402 may have a width larger than N / r, which is the width of the base anchor box 401, And can be disposed at a position completely including the base anchor box 401 on one side. The sub-anchor boxes 402 may be disposed at positions shifted by a distance of p from each other.

The size of the anchor box may be a pixel unit. The width (number of pixels) of the base anchor box 401 may be 1 / r times the length of the end anchor box 403 (the number of pixels). For example, the number of pixels corresponding to the vertical size may be an even number, and the r may be a divisor of the vertical size. For example, as shown in FIG. 7, the number of vertical pixels may be N and the number of horizontal pixels may be N / r in the base anchor boxes 401.

The constant size of the increasing width of the sub-anchor box 402 may be p pixels. For example, as shown in FIG. 7, the sub-anchor boxes 402 may have N number of vertical pixels and N / r number of horizontal pixels (n is a natural number).

The predetermined interval between the start positions of the end a boxes closest to the end a boxes 403 may be q pixels. A certain interval between the start positions of the base anchor boxes 401 closest to the base anchor boxes 401 may be q pixels. For example, as shown in FIG. 8B, the base anchor boxes 401 having N number of pixels in the horizontal direction and N / r number of vertical pixels can be arranged at intervals of q pixels. For example, as shown in Fig. 8C, the end anchor boxes 403 having N number of pixels in the horizontal direction and N number of pixels in the vertical direction can be arranged at intervals of q pixels.

The anchor box generating unit 400 may change r and q according to the size of the second image. For example, if the vertical size of the second image is odd, the horizontal size is an even number, and the r value is 2, the anchor box generation unit 400 may set the q value to a value including 0.5 units Can be set. The r, q, and p values may be changed by a device user or a device manager.

The method of generating the anchor box by the anchor box generating unit 400 includes the steps of creating a base anchor box (S410), creating a sub anchor box (S420), creating an end anchor box (S430), creating a new base anchor box (S440), performing again (S450), and repeating (S460) until the end anchor box located on the opposite side is created.

In step S410 of creating the base anchor box, a base anchor box 401 positioned on one side of the second video image may be created. For example, as shown in FIG. 8A, a base anchor box 401 having a length N and a width N / r may be created at one end of the second video image.

In step S420, the sub-anchor box is generated. The sub-anchor box includes an area of the anchor box generated immediately before the size of the sub-anchor box is equal to the size of the sub-anchor box. The size of the sub-anchor box 402 can be increased. For example, as shown in FIG. 8A, sub-anchor boxes 402 having a size N and a width N / r of + N * p can be generated from one end of the second video image (N is a natural number).

In step S430 of creating the end anchor box, the end anchor box 403 including an area of the anchor box previously created may be created. For example, as shown in FIG. 8A, an end anchor box 403 having a length N and a width N of length N can be generated from one end of the second video image.

In step S440 of creating the new base anchor box, a new base anchor box 401 may be created in which the previously generated base anchor box 401 has its horizontal position shifted by a predetermined size. For example, as shown in FIG. 8 (b), a base anchor box 401 having a vertical size N and a horizontal size N / r, which is shifted by q from one end of the second video image, can be generated.

In step S450, the sub-anchor box creation step S420 and the end anchor box creation step S430 may be performed again. For example, as shown in FIG. 8 (b), a sub-anchor box (N / r) + n * p having a size N and a size of horizontal (N / r) + n * p shifted by l * q from one end of the second video image 402) and end anchor boxes 403 of length N and width N (n and l are natural numbers).

In the step S460 of repeating until the end anchor box located on the most opposite side is created, the step of creating the new base anchor box S440 and the step S450 of performing the re-performing are repeatedly performed on one side Can be repeated until the end anchor box 403 located on the most opposite side of the anchor box 403 is produced. For example, as shown in FIG. 8 (c), an end anchor box 403 having a length N and a width N, which are positioned at opposite ends of one side of the second video image, may be created.

Accordingly, the number of anchor boxes generated from the second image having a vertical size of N, a horizontal size of M, and r of 2 can be calculated by the following equation (1).

Equation 1

Here, N is the vertical size of the second video image and M is the horizontal size of the second video image.

The bounding box generating unit 500 compares the feature components of the second image corresponding to the anchor boxes with the character detection learning data to generate an anchor box having a maximum matching value in a bounding box ) Can be selected.

The character recognition unit 600 can recognize the character corresponding to the anchor box by comparing the feature component of the area corresponding to the bounding box with the character recognition learning data. The character recognition unit 600 can recognize a character corresponding to the anchor box using a fully connected network. The Fully-Connected network may include an input layer, a fully-connected layer, and an output layer. For example, as shown in FIG. 13, the Fully-connected network may include an input layer, a first Fully-connected layer, and a second fully-connected layer, and the input layer may have a feature of an area corresponding to the bounding box Components can be input. Here, the Fully-connected network refers to artificial neural network technology using layers combined with all neurons of the previous layer.

 2 is a configuration diagram illustrating a vehicle number recognition apparatus using an anchor box and a CNN feature map according to an embodiment of the present invention.

The vehicle number recognizing apparatus using the anchor box and the CNN characteristic map according to the present embodiment is substantially the same as the vehicle number recognizing apparatus of Fig. 1 except for the vehicle detecting section 700, the preprocessing section 800 and the projection section 900 Do. Therefore, the same components as those of the vehicle identification apparatus of FIG. 1 are denoted by the same reference numerals, and repeated descriptions are omitted.

Referring to FIG. 2, the vehicle number recognizing apparatus using the anchor box and the CNN feature map according to an embodiment of the present invention may include a vehicle detecting unit 700, a preprocessing unit 800, and a projection unit 900.

The vehicle detecting unit 700 may detect a vehicle area using the SSD (Single Shot MultiBox Detector) algorithm from the first image. The vehicle detection unit 700 can detect the vehicle area using the SSD (Single Shot Multibox Detector) technology and vehicle detection learning data. The SSD technique used for vehicle detection is a known technology as shown in FIG. 10, and thus a detailed description thereof will be omitted. The second image generator 200 may generate a second image by detecting the vehicle license plate area, which is the ROI, from the detected vehicle area using a SSD (Single shot MultiBox Detector) algorithm.

The preprocessing unit 800 may enhance the image quality by performing intensity balancing and morphology on the second image.

The intensity balancing operation calculates a histogram of an image gray level, calculates a pixel value corresponding to a lower quartile and an upper quartile from the histogram, and performs an affine transform using Equation (2) Can be performed.

Equation 2

Here, x is a pixel value at the position, max is a maximum pixel value in the region, min is a minimum pixel value in the region, Vmin is a pixel value corresponding to a lower quartile, and Vmax is a pixel value corresponding to a higher quartile Value.

The morphology may be a method for removing noise of an image and emphasizing a character portion. The morphology may include a closed morphology and an open morphology, and the preprocessor 800 may perform the closed morphology and then the open morphology. The closed morphology may be to perform a dilation operation after performing an erosion operation. The open morphology can be an expansion operation followed by an erosion operation. The closed morphology removes noise, and the open morphology can emphasize the pixels of the character portion.

The projection unit 900 may calculate an upper position and a lower position of the second image by using a horizontal projection method. In the horizontal projection method, the second video image is binarized, and a projection is performed for each of the rows to count the number of pixels. Then, the number of pixel counts is calculated from the bottom to set the minimum count number as a lower position. Quot; portion "

The car number recognizing apparatus may further include a power supply unit for supplying power to the car number recognizing apparatus, and a storage unit for storing the car number recognizing apparatus.

The car number recognition apparatus may further include a car number output unit for outputting a car number by sequentially combining the recognized characters.

The configuration of the vehicle identification apparatus may be variously modified. The configuration of the car number recognition device may be entirely composed of one device or a separate device. For example, the image input unit 100, the second image generation unit 200, the feature extraction unit, the anchor box generation unit 400, the bounding box generation unit and the character recognition unit 600 It can be composed of one device. For example, the second image generation unit 200, the feature extraction unit, the anchor box generation unit 400, the bounding box generation unit, and the character recognition unit 600 may be constituted by one processor have. For example, the second image generation unit 200, the feature extraction unit, the anchor box generation unit 400, the bounding box generation unit, and the character recognition unit 600 may be configured as servers And a plurality of the image input units 100 may be connected to the server by wire or wirelessly. The vehicle identification apparatus may further include a communication unit for transmitting data to another component or another system.

3 is a flowchart illustrating a method of recognizing a car number using an anchor box and a CNN feature map according to an embodiment of the present invention.

The car number recognition method according to the present embodiment is performed in the car number recognition device and is substantially the same as the car number recognition device in Figs. Therefore, the same components as those of the vehicle identification apparatus of FIGS. 1 and 2 are denoted by the same reference numerals, and repeated descriptions are omitted.

Referring to FIG. 3, a car number recognition method according to an embodiment of the present invention includes receiving a first image (S100), generating a second image (S200), extracting a feature S300), extracting feature components (S400), selecting a bounding box (S500), and recognizing characters (S600).

In the step S100 of receiving the first video image, the video input unit 100 may receive the first video image. The first video image may include characters having the same standard. The first video image may include characters arranged in a line having the same standard. For example, the first video image may include numbers, letters, or signs. For example, as shown in FIG. 11, a vehicle license plate may be included in the first image.

In the step S200 of generating the second video image, the second video generator 200 may detect a region of interest including the character from the first video image to generate a second video image. For example, as shown in FIG. 11, the area of interest may be the area of a license plate. For detection of the region of interest, a deep learning technique may be used. For example, the area of interest is the area of the vehicle license plate, the deep learning technique is a single shot multibox detector (SSD), and the vehicle license plate area can be detected using license plate detection learning data. The SSD technology used for vehicle license plate detection is a known technology as shown in FIG. 10, so a detailed description will be omitted.

In step S300 of extracting the feature component, the feature component extraction unit 300 may extract the feature component from the second image. For example, the feature extraction unit 300 extracts a convolution feature map of the second image by using a CNN (Convolutional Neural Network), and extracts a convolution feature map of the second image, can do.

The composite neural network may include an input layer, a convolution layer, a pooling layer, and a feature map. A plurality of the composite product layer and the pooling layer may be used. The convolution layer may transform an image by applying a filter to the input data and then reflecting the activation function. The pooling layer may be a subsampling layer that reduces the dimension of the image. Thus, the pooling layer may be omitted.

For example, as shown in FIG. 12, the input image layer may normalize the second image to 112x112, use a gray level channel (@ 1 or depth 1), and in the composite product layer, The normalized second video image can be converted into a size of 56x56, and in the pooling layer, the converted second video image can be converted into a 28x28 size using a gray level 32 channel, , The second video image converted in the pooling layer can be converted into a 14x14 size using the gray level 64 channel, and in the next pooling layer, the second video image converted in the next composite product layer can be converted into the gray level 128 channel , And an image having a final converted 7x7 size and 128 depth can be converted into a 7x7 image using the above It can be used as a gong map.

In step S400 of extracting the feature component, the anchor box generating unit 400 may generate a plurality of anchor boxes from the second image. The plurality of anchor boxes may have the same vertical size as the second video image and have different horizontal sizes. The anchor box may be an arbitrary division area for character recognition.

The anchor boxes may include an end anchor box 403, a base anchor box 401, and a sub anchor box 402.

The end anchor box 403 may have the same width and the same length. For example, it may be an anchor box having a horizontal size and a vertical size N as shown in FIG. The end anchor box 403 may be a plurality of end anchor boxes. For example, the number of the end anchor boxes 403 may be a (a is a natural number). The end anchor box 403 may be overlaid over all areas of the second video image at regular intervals. For example, if the size of the second video image is M, the size of the vertical is N, and the end anchor boxes 403 are arranged at intervals of size 1 (r = 1) as shown in FIG. 8, The number a of the boxes 403 may be (M-N + 1).

The width of the base anchor box 401 may be smaller than the length of the base anchor box 401. For example, as shown in FIG. 7, the width of the base anchor box 401 may be 1 / r times the vertical length. For example, the width of the base anchor box 401 may be 1/2 times the length of the base anchor box 401. The base anchor box 401 may be located at one side of the end anchor box 403. The base anchor box 401 may be formed in the same number as the end anchor box 403. For example, when the size of the second video image is M, the size of the vertical is N, and the base anchor boxes 401 are arranged at intervals of size 1 (r = 1) as shown in FIG. 8, The number a of boxes 401 may be (M-N + 1).

The width of the sub-anchor box 402 may be smaller than the width of the end anchor box 403, but may be increased by a certain amount in the width of the base anchor box 401. A plurality of sub-anchor boxes 402 may be generated for one base anchor box 401. For example, as shown in FIG. 7, the width of the sub-anchor boxes 402 may be 1 / r of the vertical size plus n times of p.

The sub-anchor box 402 may include an area of the base anchor box 401. For example, as shown in FIG. 7, the sub-anchor boxes 402 may have a width larger than N / r, which is the width of the base anchor box 401, And can be disposed at a position completely including the base anchor box 401 on one side. The sub-anchor boxes 402 may be disposed at positions shifted by a distance of p from each other.

The size of the anchor box may be a pixel unit. The width (number of pixels) of the base anchor box 401 may be 1 / r times the length of the end anchor box 403 (the number of pixels). For example, the number of pixels corresponding to the vertical size may be an even number, and the r may be a divisor of the vertical size. For example, as shown in FIG. 7, the number of vertical pixels may be N and the number of horizontal pixels may be N / r in the base anchor boxes 401.

The constant size of the increasing width of the sub-anchor box 402 may be p pixels. For example, as shown in FIG. 7, the sub-anchor boxes 402 may have N number of vertical pixels and N / r number of horizontal pixels (n is a natural number).

The predetermined interval between the start positions of the end a boxes closest to the end a boxes 403 may be q pixels. A certain interval between the start positions of the base anchor boxes 401 closest to the base anchor boxes 401 may be q pixels. For example, as shown in FIG. 8B, the base anchor boxes 401 having N number of pixels in the horizontal direction and N / r number of vertical pixels can be arranged at intervals of q pixels. For example, as shown in Fig. 8C, the end anchor boxes 403 having N number of pixels in the horizontal direction and N number of pixels in the vertical direction can be arranged at intervals of q pixels.

The anchor box generating unit 400 may change r and q according to the size of the second image. For example, if the vertical size of the second image is odd, the horizontal size is an even number, and the r value is 2, the anchor box generation unit 400 may set the q value to a value including 0.5 units Can be set.

The step S400 of extracting the feature component includes a step S410 of creating a base anchor box, a step S420 of creating a sub anchor box, a step S430 of creating an end anchor box, a step of creating a new base anchor box (S440), performing again (S450), and repeating (S460) until the end anchor box located on the opposite side is generated.

In step S410 of creating the base anchor box, a base anchor box 401 positioned on one side of the second video image may be created. For example, as shown in FIG. 8A, a base anchor box 401 having a length N and a width N / r may be created at one end of the second video image.

In step S420, the sub-anchor box is generated. The sub-anchor box includes an area of the anchor box generated immediately before the size of the sub-anchor box is equal to the size of the sub-anchor box. The size of the sub-anchor box 402 can be increased. For example, as shown in FIG. 8A, sub-anchor boxes 402 having a size N and a width N / r of + N * p can be generated from one end of the second video image (N is a natural number).

In step S430 of creating the end anchor box, the end anchor box 403 including an area of the anchor box previously created may be created. For example, as shown in FIG. 8A, an end anchor box 403 having a length N and a width N of length N can be generated from one end of the second video image.

In step S440 of creating the new base anchor box, a new base anchor box 401 may be created in which the previously generated base anchor box 401 has its horizontal position shifted by a predetermined size. For example, as shown in FIG. 8 (b), a base anchor box 401 having a vertical size N and a horizontal size N / r, which is shifted by q from one end of the second video image, can be generated.

In step S450, the sub-anchor box creation step S420 and the end anchor box creation step S430 may be performed again. For example, as shown in FIG. 8 (b), a sub-anchor box (N / r) + n * p having a size N and a size of horizontal (N / r) + n * p shifted by l * q from one end of the second video image 402) and end anchor boxes 403 of length N and width N (n and l are natural numbers).

In the step S460 of repeating until the end anchor box located on the most opposite side is created, the step of creating the new base anchor box S440 and the step S450 of performing the re-performing are repeatedly performed on one side Can be repeated until the end anchor box 403 located on the most opposite side of the anchor box 403 is produced. For example, as shown in FIG. 8 (c), an end anchor box 403 having a length N and a width N, which are positioned at opposite ends of one side of the second video image, may be created.

Accordingly, the number of anchor boxes generated from the second image having a vertical size of N, a horizontal size of M, and r of 2 can be calculated by the following equation (3).

Equation 3

Here, N is the vertical size of the second video image and M is the horizontal size of the second video image.

In the step S500 of selecting a bounding box, the bounding box generating unit 500 compares the feature components of the second video image corresponding to the area of the anchor boxes with the character detection learning data, An anchor box having a value can be selected as a bounding box of each character area.

In step S600 of recognizing the character, the character recognition unit 600 can recognize the character corresponding to the anchor box by comparing the feature component of the area corresponding to the bounding box with the character recognition learning data. The character recognition unit 600 can recognize a character corresponding to the anchor box using a fully connected network. The Fully-Connected network may include an input layer, a fully-connected layer, and an output layer. For example, as shown in FIG. 13, the Fully-connected network may include an input layer, a first Fully-connected layer, and a second fully-connected layer, and the input layer may have a feature of an area corresponding to the bounding box Components can be input. Here, the Fully-connected network refers to artificial neural network technology using layers combined with all neurons of the previous layer.

4 is a flowchart illustrating a method of recognizing a car number using an anchor box and a CNN feature map according to an embodiment of the present invention. 5 is a flowchart illustrating a step of improving the image quality of the car number recognition method according to an embodiment of the present invention.

The vehicle number recognition method according to the present embodiment includes the steps of detecting a vehicle area (S110), improving an image quality (S210), and calculating a higher position and a lower position of a second image image (S220) Is substantially the same as the car number recognition method of Fig. Therefore, the same constituent elements as those of the car number recognition method of FIG. 3 are denoted by the same reference numerals, and repeated description is omitted.

4 and 5, a vehicle number recognition method according to an exemplary embodiment of the present invention includes a step S110 of detecting a vehicle area, a step S210 of improving an image quality, And calculating a position (S220).

In the step S110 of detecting the vehicle area, the vehicle detector 700 may detect the vehicle area using the SSD (Single Shot MultiBox Detector) algorithm from the first image. For example, as shown in FIG. 9, in step S110 of detecting the vehicle area, the vehicle may be detected using a single shot multibox detector (SSD) and vehicle detection learning data. The SSD technique used for the vehicle detection is a known technique as shown in FIG. 10, so a detailed description thereof will be omitted. In the step of generating the second video image, the second plate image region may be generated by detecting the plate region of the vehicle, which is the region of interest, using the SSD (Single Shot MultiBox Detector) algorithm from the detected vehicle region.

In the step of improving the image quality (S210), the preprocessing unit 800 may improve the image quality by performing intensity balancing and morphology on the second image.

The step of improving the image quality (S210) includes a step S211 of calculating a histogram, a step S212 of calculating a pixel value, an affine transform step S213, a step of performing a closed morphology (S214) and performing an open morphology (S215).

In the step of calculating the histogram (S211), a histogram of the image gray level may be calculated from the second image. In step S212, the pixel values corresponding to the lower quartile and the upper quartile can be calculated from the histogram. In step S213 of performing the affine transform, an affine transform may be performed using Equation (4).

Equation 4

Here, x is a pixel value at the position, max is a maximum pixel value in the region, min is a minimum pixel value in the region, Vmin is a pixel value corresponding to a lower quartile, and Vmax is a pixel value corresponding to a higher quartile Value.

The morphology may be a method for removing noise of an image and emphasizing a character portion. The morphology may include a closed morphology and an open morphology. The preprocessor 800 may perform the closed morphology and then the open morphology. In the step S214 of performing the closed morphology, the closed morphology may be performed to perform a dilation operation after performing an erosion operation. In the step of performing the open morphology (S215), the open morphology for performing the erosion operation after the expansion operation can be performed. The closed morphology removes noise, and the open morphology can emphasize the pixels of the character portion.

In step S220 of calculating the upper and lower positions of the second image, the projection unit 900 may calculate upper and lower positions of the second image using a horizontal projection method . In the horizontal projection method, the second video image is binarized, and a projection is performed for each of the rows to count the number of pixels. Then, the number of pixel counts is calculated from the bottom to set the minimum count number as a lower position. Quot; portion "

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that various changes and modifications may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. It will be understood.

100:
200: second image generating unit
300: Feature component extraction unit
400: anchor box creation unit
500: Bounding box generating unit
600:

Claims (12)

A video input unit receiving a first video image;
A second image generation unit for generating a second image by detecting a region of interest including a character from the first image;
A feature extraction unit for extracting a feature component from the second image;
An anchor box generating unit for generating a plurality of anchor boxes having the same size as the second video image from the second video image and having different sizes in the horizontal direction;
A bounding box generating unit for comparing the feature components of the second image corresponding to the anchor boxes with the character detection learning data and selecting an anchor box having a maximum matching value as a bounding box of each character area; And
And a character recognizing unit for recognizing the character corresponding to the anchor box by comparing the feature component of the area corresponding to the bounding box with the character recognition learning data,
The anchor boxes may include:
A a number of end anchor boxes (a is a natural number) that are overlapped with all the areas of the second video image at regular intervals and have the same size in the horizontal and vertical directions;
A base anchor box located at one side of the end anchor box and having a width smaller than a vertical size; And
An anchor box including a plurality of sub-anchor boxes including an area of the base anchor box, the size of which is smaller than the width of the end anchor box and increased by a predetermined size in the width of the base anchor box, Box and CNN feature map.
2. The method of claim 1, wherein the anchor box generating unit generates the anchor box,
Creating a base anchor box located on one side of the second video image;
The size of the sub-anchor box including the area of the anchor box generated immediately before the size of the horizontal is equal to the size of the sub-anchor box, ;
Creating an end anchor box including an area of an anchor box created immediately before;
Creating a new base anchor box in which a horizontal position is shifted by a predetermined size in a base anchor box generated immediately before;
Creating the sub-anchor box and creating the end anchor box again; And
Generating the new base anchor box and repeating the step of repeating the step until the end anchor box located on the most opposite side of the one side of the second video image is generated, Vehicle number identification device used.
The method of claim 1, wherein the size of the anchor box is in pixels,
The width of the base anchor box (the number of pixels) is 1 / r times the length of the end anchor box (the number of pixels)
The constant size of the increasing width of the sub-anchor box is p pixels,
Wherein the predetermined interval of the start positions between the end boxes adjacent to the most adjacent end boxes is a q-pixel.
The apparatus as claimed in claim 3, wherein the total number of anchor boxes generated from one second image when r is 2 is calculated by the following equation.

Here, N is the vertical size of the second video image and M is the horizontal size of the second video image.
The apparatus of claim 1, further comprising: a vehicle detection unit that detects a vehicle area using a SSD (Single Shot MultiBox Detector) algorithm from the first image,
The second image generator generates an anchor box for generating a second image by detecting the vehicle license plate area, which is the area of interest, from the detected vehicle area using an SSD (Single Shot MultiBox Detector) algorithm, Recognition device.
The method according to claim 1,
Further comprising a preprocessing unit for enhancing image quality by performing intensity balancing and morphology on the second video image,
Wherein the morphology includes a closed morphology and an open morphology,
The intensity balancing operation calculates a histogram of the image gray level, calculates a pixel value corresponding to a lower quartile and an upper quartile from the histogram, and performs an affine transform using the following equation Vehicle identification number.

Here, x is a pixel value at the corresponding position, max is a maximum pixel value in the corresponding region, min is a minimum pixel value in the corresponding region, Vmin is a pixel value corresponding to a lower quartile, and Vmax corresponds to a higher quartile Pixel values.
Receiving a first video image from a video input unit;
Generating a second image by detecting a region of interest including a character from the first image;
Extracting a feature component from the second image;
Generating an anchor box from the second video image, the anchor box generating unit having a plurality of anchor boxes having the same vertical size as the second video image and different widths from each other;
The bounding box generating unit compares the feature components of the second video image corresponding to the area of the anchor boxes with the character detection learning data and selects an anchor box having a maximum matching value as a bounding box of each character area ; And
And recognizing the character corresponding to the anchor box by comparing the feature component of the area corresponding to the bounding box with the character recognition learning data,
The anchor boxes may include:
A a number of end anchor boxes (a is a natural number) that are overlapped with all the areas of the second video image at regular intervals and have the same size in the horizontal and vertical directions;
A base anchor box located at one side of the end anchor box and having a width smaller than a vertical size; And
An anchor box including a plurality of sub-anchor boxes including an area of the base anchor box, the size of which is smaller than the width of the end anchor box and increased by a predetermined size in the width of the base anchor box, Box number and CNN feature map.
8. The method of claim 7, wherein the creating of the anchor box comprises:
Creating a base anchor box located on one side of the second video image;
The size of the sub-anchor box including the area of the anchor box generated immediately before the size of the horizontal is equal to the size of the sub-anchor box, ;
Creating an end anchor box including an area of an anchor box created immediately before;
Creating a new base anchor box in which a horizontal position is shifted by a predetermined size in a base anchor box generated immediately before;
Creating the sub-anchor box and creating the end anchor box again; And
Generating the new base anchor box and repeating the step of repeating the step until the end anchor box located on the most opposite side of the one side of the second video image is generated, A method of recognizing a used car number.
8. The method of claim 7, wherein the size of the anchor box is in pixels,
The width of the base anchor box (the number of pixels) is 1 / r times the length of the end anchor box (the number of pixels)
The constant size of the increasing width of the sub-anchor box is p pixels,
Wherein the predetermined interval of the start positions between the most adjacent end boxes among the a number of end-anchor boxes is q pixels, and the CNN feature map is used.
10. The method of claim 9, wherein the total number of anchor boxes generated from one second image when r is 2 is calculated by the following equation.

Here, N is the vertical size of the second video image and M is the horizontal size of the second video image.
8. The method of claim 7,
Further comprising the step of the vehicle detection unit detecting the vehicle area using the SSD algorithm from the first image,
In the step of generating the second video image, an anchor box and a CNN feature map for generating a second video image by detecting the vehicle license plate area, which is the area of interest, using SSD (Single shot MultiBox Detector) algorithm from the detected vehicle area A method of recognizing a car number using the.
8. The method of claim 7,
The pre-processing unit further performing an intensity balancing and a morphology on the second image to improve the image quality,
Wherein the improving the image quality comprises:
Performing a closed morphology;
Performing the closed morphology followed by an open morphology;
Calculating a histogram for the image gray level;
Calculating a pixel value corresponding to a lower quartile and an upper quartile from the histogram; And
A method for recognizing a car number using an anchor box and a CNN feature map, the method comprising: performing an affine transform using the following equation.

Here, x is a pixel value at the corresponding position, max is a maximum pixel value in the corresponding region, min is a minimum pixel value in the corresponding region, Vmin is a pixel value corresponding to a lower quartile, and Vmax corresponds to a higher quartile Pixel values.

Images