KR102181861B1 - System and method for detecting and recognizing license plate - Google Patents

Abstract

Embodiments of the present invention are an input unit that receives image information including a vehicle license plate, generates a license plate candidate region from the image information, and determines a license plate region based on the geometric characteristics of the license plate candidate region or a rectangle estimated from the license plate candidate region. It provides a vehicle license plate detection and recognition system comprising a first license plate area detection unit and a character recognition unit for recognizing characters included in the license plate based on the license plate area.
In addition, embodiments of the present invention include an input unit receiving image information including a vehicle license plate, a first character region detection unit detecting a character candidate region from the image information using a character detection model, and geometric characteristics of the character candidate region. A second character area detection unit that corrects the character candidate area by merging or deleting the character candidate area, adjusting the size of the character candidate area, or adding a new character area to the character candidate area, and a license plate based on the corrected character candidate area. It provides a vehicle license plate detection and recognition system configured including a character recognition unit for recognizing characters included in the.
Accordingly, embodiments of the present invention generate a license plate candidate region from the input image information to determine the license plate region based on the characteristics of the license plate candidate region, or the second character region detector determines the character candidate region detected by the first character region detector. By complementing it, the accuracy of license plate detection and recognition can be improved.

Description

Vehicle license plate detection and recognition system and method {SYSTEM AND METHOD FOR DETECTING AND RECOGNIZING LICENSE PLATE}

The present invention relates to a vehicle license plate detection and recognition system and method, and more particularly, a vehicle license plate detection and recognition system and method for accurately detecting a license plate region or a character region of the vehicle license plate and accurately recognizing a character included in the license plate It is about.

In the case of a vehicle driving enforcement camera, it is important to improve the accuracy of license plate recognition because administrative disposition may be imposed on the vehicle driver based on the result of recognizing the license plate included in the captured image information.

As factors that hinder the accuracy of license plate recognition, smear due to sunlight or blurring due to lighting are typical, it is necessary to overcome these factors in order to improve the accuracy of license plate recognition.

The related art is as follows.

Korean Patent Registration No. 10-1182173 relates to a vehicle license plate recognition method, comprising: receiving an image of a vehicle with a license plate attached, estimating an area of the license plate using a DoG filter, and Disclosed is a vehicle license plate recognition method comprising the steps of dividing a region to distinguish each character included in the region of the license plate, and recognizing each character using a neural network character recognizer.

However, in the prior art, since parameter values such as a threshold for binarization are not applied in various ways and characters are not recognized in various ways, high accuracy cannot be guaranteed in various situations.

In order to solve the above-described problem, embodiments of the present invention provide a vehicle license plate detection and recognition system and method for generating a license plate candidate region and determining a license plate region based on geometric characteristics of a license plate candidate region or a rectangle estimated from the license plate candidate region. Its purpose is to provide.

In addition, embodiments of the present invention provide a vehicle license plate detection and recognition system that binarizes image information based on different thresholds or adds noise of different levels to image information to binarize and estimate outlines to generate license plate candidate regions. It has its purpose to provide.

In addition, embodiments of the present invention determine the license plate area based on the number of corners, corner angles, side lengths, widths, or the ratio of the binarized pixel values of the license plate candidate area or the rectangle estimated from the license plate candidate area. Its purpose is to provide a system.

In addition, embodiments of the present invention detect a license plate region from image information, generate a license plate candidate region from image information corresponding to the detected license plate region and a peripheral region of the detected license plate region, and estimate from the license plate candidate region or the license plate candidate region. It is an object of the present invention to provide a vehicle license plate detection and recognition system that determines a license plate area based on a geometric characteristic of a square and recognizes characters included in the license plate based on the determined license plate area.

In addition, embodiments of the present invention detect a character candidate region from image information using a character detection model, merge or delete a character candidate region, adjust the size of the character candidate region, or adjust the size of the character candidate region based on the geometric characteristics of the character candidate region. An object thereof is to provide a vehicle license plate detection and recognition system and method for correcting a character candidate region by adding an area to a character candidate region and recognizing a character included in a license plate based on the corrected character candidate region.

In addition, embodiments of the present invention merge or delete character candidate regions based on the height, position, or width of the overlapped region, adjust the size of the character candidate region, or add the distance from the center position of the character candidate region. An object of the present invention is to provide a vehicle license plate detection and recognition system that calculates at least one straight line that is a minimum value, detects a new character area whose center position is close to the line, and adds it to the character candidate area.

In addition, embodiments of the present invention detect a character candidate region from image information using a character detection model, merge or delete a character candidate region, adjust the size of the character candidate region, or adjust the size of the character candidate region based on the geometric characteristics of the character candidate region. A vehicle license plate detection and recognition system and method for correcting the character candidate area by adding an area to the character candidate area, detecting the license plate area including the corrected character candidate area, and recognizing the characters included in the license plate based on the detected license plate area Its purpose is to provide.

In addition, embodiments of the present invention provide a vehicle license plate detection and recognition system and method for recognizing a character included in a license plate based on the various systems or methods described above and voting with the recognized character to infer the character recognition result. There is a purpose.

An embodiment of the present invention for achieving the above object is an input unit that receives image information including a vehicle license plate, a license plate candidate region is generated from the image information, and the geometric characteristic of a rectangle estimated from the license plate candidate region or the license plate candidate region. It provides a vehicle license plate detection and recognition system comprising a first license plate area detection unit for determining a license plate area and a character recognition unit for recognizing characters included in the license plate based on the license plate area.

In an embodiment, the first license plate region detection unit may generate the license plate candidate region by binarizing the image information based on different threshold values or binarizing the image information by adding different noises and estimating an outline. .

In an embodiment, the first license plate region detection unit may determine the license plate region based on the number of corners, corner angles, side lengths, widths, or ratios of binarized pixel values of the license plate candidate region or the rectangle estimated from the license plate candidate region. .

In one embodiment, the vehicle license plate detection and recognition system includes a license plate detection unit for detecting a license plate region from the image information, and the first license plate region detection unit corresponds to the detected license plate region and a peripheral region of the detected license plate region. A license plate candidate region is generated from the image information, and a license plate region is determined based on the geometric characteristics of the license plate candidate region or a rectangle estimated from the license plate candidate region, and the character recognition unit recognizes the characters included in the license plate based on the determined license plate region. I can recognize it.

In addition, another embodiment of the present invention for achieving the above object is an input unit for receiving image information including a vehicle license plate, a first character region detection unit for detecting a character candidate region from the image information using a character detection model, A second character area detection unit for correcting the character candidate area by merging or deleting the character candidate area, adjusting the size of the character candidate area, or adding a new character area to the character candidate area based on the geometric characteristics of the character candidate area, and the correction It provides a vehicle license plate detection and recognition system comprising a character recognition unit for recognizing characters included in the license plate based on the character candidate area.

In another embodiment, the second character region detection unit may merge or delete the character candidate region or adjust the size of the character candidate region based on the height, position, or area of the overlapping region of the character candidate region.

In another embodiment, the second character area detection unit calculates one or more straight lines in which the sum of distances from the center position of the character candidate area becomes the minimum value, and detects a new character area whose center position is close to the line Can be added to the candidate area.

In addition, another embodiment of the present invention for achieving the above object is an input unit for receiving image information including a vehicle license plate, a first character region detection unit for detecting a character candidate region from the image information using a character detection model, Compensating the character candidate area by merging or deleting the character candidate area, adjusting the size of the character candidate area, or adding a new character area to the character candidate area based on the geometric characteristics of the character candidate area, and including the corrected character candidate area. It provides a vehicle license plate detection and recognition system comprising a second license plate area detection unit for detecting a license plate area, and a character recognition unit for recognizing characters included in the license plate based on the license plate area.

In addition, another embodiment of the present invention for achieving the above object is an input unit that receives image information including a vehicle license plate, a detection unit that determines or detects at least one license plate region or a character candidate region from the image information, and the license plate. A character recognition unit for recognizing a character included in the license plate based on an area or a character candidate area, and a recognition result inferring unit for inferring a character recognition result by voting with the recognized character, and the detection unit includes a license plate candidate from the image information. A first license plate region detection unit that creates a region and determines a license plate region based on the geometrical features of the license plate candidate region or a rectangle estimated from the license plate candidate region, and a first detects a letter candidate region from the image information using a character detection model. A second character area for correcting the character candidate area by merging or deleting the character candidate area, adjusting the size of the character candidate area, or adding a new character area to the character candidate area based on the character area detection unit and the geometric characteristics of the character candidate area A first character region detection unit that detects a character candidate region from the image information using a detection unit and a character detection model, and merges or deletes the character candidate region or adjusts the size of the character candidate region based on the geometric characteristics of the character candidate region. A vehicle license plate detection and recognition system comprising at least one of the second license plate area detection units for correcting the character candidate area by adding a new character area to the character candidate area and detecting a license plate area including the corrected character candidate area. to provide.

In addition, another embodiment of the present invention for achieving the above object is an input step of receiving image information including a vehicle license plate, a license plate candidate region is generated from the image information, and a license plate candidate region or a rectangle estimated from the license plate candidate region. It provides a vehicle license plate detection and recognition method comprising a first license plate region detection step of determining a license plate region based on the geometric characteristics of, and a character recognition step of recognizing a character included in the license plate based on the license plate region. .

In addition, another embodiment of the present invention for achieving the above object is an input step of receiving image information including a vehicle license plate, and a first character region detection for detecting a character candidate region from the image information using a character detection model. Step, A second character area detection step of correcting the character candidate area by merging or deleting the character candidate area, adjusting the size of the character candidate area, or adding a new character area to the character candidate area based on the geometric characteristics of the character candidate area. And a character recognition step of recognizing a character included in the license plate based on the corrected character candidate area, and provides a vehicle license plate detection and recognition method.

In addition, another embodiment of the present invention for achieving the above object is an input step of receiving image information including a vehicle license plate, and a first character region detection for detecting a character candidate region from the image information using a character detection model. Step, Based on the geometric characteristics of the character candidate region, the character candidate region is corrected by merging or deleting the character candidate region, adjusting the size of the character candidate region, or adding a new character region to the character candidate region, and correcting the corrected character candidate region. It provides a vehicle license plate detection and recognition method comprising a second license plate region detection step of detecting the included license plate region and a character recognition step of recognizing characters included in the license plate based on the license plate region.

In addition, another embodiment of the present invention for achieving the above object is an input step of receiving image information including a vehicle license plate, a detection step of determining or detecting at least one license plate region or character region from the image information, the A character recognition step of recognizing a character included in the license plate based on the license plate area or the character area, and a recognition result inferring step of inferring the character recognition result by voting with the recognized character, wherein the detection step includes: The first license plate region detection step of creating a license plate candidate region and determining the license plate region based on the geometrical features of the license plate candidate region or the rectangle estimated from the license plate candidate region, and detecting the letter candidate region from the image information using a character detection model. The first character area detection step and correcting the character candidate area by merging or deleting the character candidate area, adjusting the size of the character candidate area, or adding a new character area to the character candidate area based on the geometric characteristics of the character candidate area. A second character area detection step and a first character area detection step of detecting a character candidate area from the image information using a character detection model, and a character candidate area are merged or deleted based on the geometric characteristics of the character candidate area, or a character candidate Comprising at least one or more of a second license plate area detection step of correcting the character candidate area by adjusting the size of the area or adding a new character area to the character candidate area and detecting a license plate area including the corrected character candidate area. It provides a vehicle license plate detection and recognition method.

As described above, the embodiments of the present invention create a license plate candidate region and determine the license plate region based on the geometric characteristics of the license plate candidate region or a rectangle estimated from the license plate candidate region, thereby improving the accuracy of license plate region detection and license plate recognition. I can.

In addition, embodiments of the present invention are binary image information based on different threshold values or by adding noise of different levels to the image information to binarize and estimating the outline to generate a license plate candidate region, thereby detecting the license plate region and recognizing the license plate. Accuracy can be improved.

In addition, embodiments of the present invention determine the license plate area based on the number of corners, corner angles, side lengths, widths, or the ratio of the binarized pixel values of the number of squares estimated from the license plate candidate area or the license plate candidate area. The accuracy of license plate recognition can be improved.

In addition, embodiments of the present invention detect a license plate region from image information, generate a license plate candidate region from image information corresponding to the detected license plate region and a peripheral region of the detected license plate region, and estimate from the license plate candidate region or the license plate candidate region. By determining the license plate area based on the geometrical characteristic of the rectangle and recognizing the characters included in the license plate based on the determined license plate area, it is possible to improve the speed and accuracy of license plate area detection and license plate recognition.

In addition, embodiments of the present invention detect a character candidate region from image information using a character detection model, merge or delete a character candidate region, adjust the size of the character candidate region, or adjust the size of the character candidate region based on the geometric characteristics of the character candidate region. By adding an area to the character candidate area, correcting the character candidate area, and recognizing the characters included in the license plate based on the corrected character candidate area, it is possible to improve the speed and accuracy of character area detection and license plate recognition.

In addition, embodiments of the present invention merge or delete character candidate regions based on the height, position, or width of the overlapped region, adjust the size of the character candidate region, or add the distance from the center position of the character candidate region. By calculating one or more straight lines that are the minimum values, detecting a new character area whose center position is close to the line, and adding it to the character candidate area, it is possible to improve accuracy of character area detection and license plate recognition.

In addition, embodiments of the present invention detect a character candidate region from image information using a character detection model, merge or delete a character candidate region, adjust the size of the character candidate region, or adjust the size of the character candidate region based on the geometric characteristics of the character candidate region. By adding the area to the character candidate area, correcting the character candidate area, detecting the license plate area including the corrected character candidate area, and recognizing the characters contained in the license plate based on the detected license plate area. Accuracy can be improved.

Further, embodiments of the present invention can improve the accuracy of license plate recognition by recognizing a character included in a license plate based on the above-described various systems or methods and voting with the recognized character to infer the character recognition result.

It is not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those of ordinary skill in the art from the following description.

1 is a diagram illustrating an example of a vehicle license plate detection and recognition system according to the present invention.
2 is a diagram illustrating an example of a vehicle license plate detection and recognition apparatus according to the present invention.
3 and 4 are diagrams illustrating an example of an algorithm performed by a license plate detection unit.
5 is a diagram illustrating an example of contour estimation according to the present invention.
6 is a diagram illustrating an example of square estimation.
7 is a diagram illustrating an example of a corner angle and a side length of a square according to the present invention.
8 is a diagram illustrating an example of a rectangle estimated from a binarized image.
9 and 10 are diagrams illustrating an example of an algorithm performed by the first license plate area detection unit.
11 and 12 are diagrams illustrating an example of an algorithm for generating a license plate candidate region performed by a first license plate region detection unit.
13 and 14 are diagrams illustrating an example of an algorithm for determining a license plate candidate area performed by a first license plate area detection unit.
15 to 17 are diagrams exemplarily showing an example of an algorithm performed by a first character region detection unit.
18 and 19 are diagrams illustrating an example of an algorithm performed by a second character region detection unit.
20 and 21 are views exemplarily showing an example of a first character candidate area adjustment algorithm performed by a second character area detection unit.
22 and 23 are views exemplarily showing an example of a straight line detection algorithm performed by a second character region detection unit.
24 and 25 are diagrams illustrating an example of an algorithm for adding a character candidate area performed by a second character area detection unit.
26 and 27 are views exemplarily showing an example of a second character candidate area adjustment algorithm performed by a second character area detection unit.
28 is a diagram illustrating an example in which a second license plate area detection unit is performed.
29 and 30 are diagrams illustrating an example of an algorithm performed by a character recognition unit.
31 and 32 are diagrams illustrating an example of an algorithm performed by the recognition result inference unit.
33 is a flowchart illustrating an example of a vehicle license plate detection and recognition method 500 according to the present invention.
34 and 35 are flow charts illustrating an example of voting with characters recognized based on the license plate region determined in the first license plate region detection step.
36 is a flowchart illustrating an example of voting with a character recognized based on the character candidate area detected in the first character area detection step and corrected in the second character area detection step.
37 and 38 are flowcharts exemplarily showing an example of voting a character candidate region detected in a first character region detection step with a recognized character based on the detected license plate region by correcting in a second license plate region detection step.
39 is a flowchart illustrating an example of voting using all of the methods of FIGS. 34 to 38.

Since the description of the present invention is merely examples for structural or functional description, the scope of the present invention should not be construed as being limited by the embodiments described in the text. That is, since the present embodiments can be variously changed and have various forms, the scope of the present invention should be understood to include equivalents capable of realizing the technical idea.

In addition, since the object or effect presented in the present invention does not mean that a specific embodiment should include all of them or only those effects, the scope of the present invention should not be understood as being limited thereto.

In addition, the accompanying drawings are provided to aid in the understanding of the present invention, and provide embodiments together with a detailed description. However, the technical features of the present invention are not limited to a specific drawing, and features disclosed in each drawing may be combined with each other to constitute a new embodiment.

A vehicle license plate detection and recognition system disclosed in the following embodiments will be described in more detail with reference to each drawing.

1 is a diagram illustrating an example of a vehicle license plate detection and recognition system according to the present invention.

Referring to FIG. 1, a vehicle license plate detection and recognition system 100 according to an embodiment includes a vehicle license plate detection and recognition apparatus 200 and a camera 300. The vehicle license plate detection and recognition device 200 and the camera 300 may be connected through a communication network.

Here, the communication network may mean a network of a wide concept including a wired or wireless communication network.

The vehicle license plate detection and recognition device 200 may be connected to the camera 300 and may include an input unit 210 to a control unit 294. Each configuration of the vehicle license plate detection and recognition device 200 will be described in FIG. 2.

The camera 300 may be connected to the vehicle license plate detection and recognition device 200, and transmit image information captured by photographing the license plates 400a and 400b of the vehicle to the vehicle license plate detection and recognition device 200 through a communication network. have.

2 is a diagram illustrating an example of a vehicle license plate detection and recognition apparatus according to the present invention.

2, the vehicle license plate detection and recognition apparatus 200 according to an embodiment includes an input unit 210, a license plate detection unit 220, a first license plate region detection unit 230, a second license plate region detection unit 240, Includes a first character area detection unit 250, a second character area detection unit 260, a character recognition unit 270, a recognition result inference unit 280, a transmission/reception unit 290, a database 292 and a control unit 294 It is composed by

The input unit 210 may receive image information including a vehicle license plate from the camera 300 through the transmission/reception unit 290 or may receive image information from the database 292.

Before examining the license plate detection unit 220, a brief comparison of the license plate detection unit 220, the first license plate region detection unit 230, and the second license plate region detection unit 240 is as follows.

Since the license plate detection unit 220, the first license plate area detection unit 230, and the second license plate area detection unit 240 all detect the license plate area, they perform similar functions.

However, the license plate detection unit 220 is different from the first license plate region detection unit 230 and the second license plate region detection unit 240 in which the detection method is specified in that it can detect the license plate region in various ways. Here, a method used in the first license plate region detection unit 230 and the second license plate region detection unit 240 is excluded from the various methods.

The first license plate region detection unit 230 and the second license plate region detection unit 240 are different in that they detect the license plate region using different specific methods.

Hereinafter, the license plate detection unit 220, the first license plate region detection unit 230, and the second license plate region detection unit 240 will be described.

The license plate detection unit 220 can detect the license plate area from the image information transmitted from the input unit 210 and transfer the detected license plate area to the first license plate area detection unit 230 or to the first character area detection unit 250. have. At this time, the license plate detection unit 220 may detect the license plate region in various ways, for example, using an artificial neural network such as deep learning.

When transferring the detected license plate area to the first license plate area detecting unit 230, the license plate detection unit 220 may also transfer image information corresponding to the peripheral area of the detected license plate area.

3 and 4 are diagrams illustrating an example of an algorithm performed by a license plate detection unit according to the present invention.

3 and 4, the license plate detection unit 220 according to an embodiment receives image information and coordinate information of a motorcycle vehicle (eg, outline information of a motorcycle) present in the image information, and By setting the value to 0, the license plate region may be detected using the license plate detection model, and the detected license plate region may be transmitted to the first license plate region detection unit 230 or the first character region detection unit 250.

At this time, unlike FIG. 3, the license plate detection unit 220 may directly calculate coordinate information of a motorcycle vehicle.

In addition, as described above, the license plate detection unit 220 may use, for example, a deep learning model as a license plate detection model. However, the license plate detection model is not limited to the deep learning model.

In addition, when transferring the detected license plate area to the first license plate area detecting unit 230, the license plate detection unit 220 transmits the image information corresponding to the surrounding area of the detected license plate area. It is possible to generate the detected license plate area including. Here, the peripheral area may mean an area extended by 20 pixels each up, down, left, and right to the outside of the rectangle from four sides of the rectangle including the license plate area.

In this way, the license plate detection unit 220 detects the license plate area before the first license plate area detection unit 230 or the first character area detection unit 250 performs detection, and the first license plate area detection unit 230 or the first Since the character area detection unit 250 narrows the range of the image in which the license plate area or the character area should be detected, the first license plate area detection unit 230 or the first character area detection unit 250 quickly applies various methods to detect and recognize the license plate. Speed and accuracy can be improved.

In addition, the license plate detection unit 220 transmits the detected license plate area and the surrounding area to the first license plate area detection unit 230 together, so that even when the license plate detection unit 220 erroneously detects the license plate area, the first license plate area detection unit 230 ), the accuracy of license plate detection and recognition can be improved.

The first license plate region detection unit 230 receives the image information corresponding to the license plate region and its surrounding region detected by the license plate detection unit 220, generates a license plate candidate region from the received image information, and generates a license plate candidate region from the license plate candidate region or the license plate candidate region. The license plate candidate area (the first license plate area) may be determined based on the estimated geometrical characteristics of the square.

Meanwhile, the first license plate area detection unit 230 may not receive image information from the license plate detection unit 220 but may directly receive the image information from the input unit 210.

That is, the first license plate region detection unit 230 receives image information from the input unit 210 and generates a license plate candidate region from the received image information, and based on the geometric characteristics of the license plate candidate region or the rectangle estimated from the license plate candidate region. A candidate area (a first license plate area) can be determined.

A method of generating a license plate candidate region in the first license plate region detection unit 230 and determining a license plate candidate region (first license plate region) will be described.

Regarding the generation of the license plate candidate region, the first license plate region detection unit 230 binarizes the received image information based on different threshold values or binarizes the image information by adding different noises, and estimates the outline to determine the license plate candidate region. Can be generated.

Here, binarization may mean changing a value of each pixel of the image information to be black (0) or white (255) based on a threshold (hereinafter, binarization threshold). When the binarization threshold is set to a value (a value between A and B) between the value of the pixel at the license plate boundary (A) and the value of the pixel surrounding the license plate boundary (B), the license plate area can be accurately detected through binarization. have.

However, in order to accurately calculate the binarization threshold as a value between A and B, it is necessary to distinguish between the pixels at the license plate boundary and the pixels surrounding the license plate boundary. This is difficult. In particular, when the values of A and B are similar, the license plate area is It can be more difficult to calculate the binarization threshold between A and B as it is difficult to distinguish.

Therefore, the first license plate region detection unit 230 does not calculate the binarization threshold to be a value between A and B, but rather sets several binarization thresholds in advance and binarizes them to a predetermined threshold to generate several license plate candidate regions. After that, by evaluating the license plate candidate area and determining the license plate area, the accuracy of license plate detection can be improved even if the values of A and B are similar. A method of determining the license plate candidate area (first license plate area) will be described later.

In addition, adding noise here is to change pixel values of image information so that successively arranged pixels do not have a specific pattern. For example, when pixels that are consecutively arranged in a row have the same value and thus have a linear pattern of a specific length, adding noise to the pixels may cause the linear pattern of a specific length to disappear.

If different amounts of noise are added around the license plate boundary and the license plate boundary, it may be easier to distinguish the license plate boundary from the surrounding license plate boundary. For example, a method of adding Gaussian noise may correspond to this.

If Gaussian noise is added, the brighter (the larger pixel value) or the darker (the smaller pixel value) part of the image information, the less noise is added, and the medium brightness (the medium pixel value part) adds more noise. I can.

When Gaussian noise is added, a lot of noise is added to the rest of the image information except for the bright or dark parts, so the pattern disappears, but the bright or dark parts have less noise and the pattern is maintained, so the pattern becomes clearer than the rest.

However, the border of the license plate corresponds to the edge of the object, and the edge of the object may be brighter or darker than other parts. Therefore, when Gaussian noise is added to the image information, the pattern constituting the border of the license plate (for example, a rectangle) becomes clear and the license plate area The accuracy of detection can be improved.

In particular, even if the value of the pixel (C) at the border of the license plate and the value of the pixel (D) surrounding the border of the license plate are similar, C corresponding to the corner may be brighter or darker than D. C can be clearer due to the addition of a lot of noise, and if the image to which Gaussian noise is added is binarized, C and D can be more clearly distinguished due to the difference in the amount of added noise, thus improving the accuracy of license plate area detection I can.

In addition, here, as a method of estimating the contour, canny edge detection may be used.

5 is a diagram illustrating an example of contour estimation according to the present invention.

Referring to FIG. 5, when the first license plate region detection unit 230 according to an embodiment estimates an outline from a binary image, a polygon having four or more vertices (points, corners) constituting the outline is formed due to noise. It can be confirmed.

Regarding the determination of the license plate candidate region, the first license plate region detection unit 230 determines the license plate based on the number of corners, corner angles, side lengths, widths, or the ratio of the binarized pixel values estimated from the license plate candidate region or the license plate candidate region. Area can be determined.

Here, an example of determining the license plate area based on the number of corners of the license plate candidate area is as follows.

Unlike FIG. 5, if the number of vertices (points, corners) constituting the outline is four, the first license plate region detection unit 230 can know that the license plate candidate region is normally generated in a square shape. That is, the first license plate region detection unit 230 may easily determine the license plate region based on the number of edges of the license plate candidate region.

In addition, here, referring to FIG. 6 with respect to a method of estimating a rectangle from the license plate candidate region, and referring to FIG. 7 with respect to the corner angle and side length of the rectangle, and FIG. Take a look.

6 is a diagram illustrating an example of square estimation.

Referring to FIG. 6, a square according to an embodiment can be estimated using, for example, a Ramer-Douglas-Peucker algorithm. Specifically, a parameter (Ratio) related to polygon estimation accuracy is increased until a square is estimated. The rectangle can be estimated by removing some of the points (vertexes, corners) constituting the outline (polygon) estimated in 5.

In FIG. 6, as the Ratio value increases from 1% to 10%, the number of points (corners) constituting the outline gradually decreases, and when the Ratio value is 10%, the number of points (corners) becomes 4, indicating that a square is estimated. I can.

In this way, the first license plate region detection unit 230 may easily determine the license plate region based on whether a square is estimated from the outline of the license plate candidate region.

7 is a diagram illustrating an example of a corner angle and a side length of a square according to the present invention.

Referring to FIG. 7, the length of a side of a rectangle according to an embodiment may correspond to the length of an actual side of the rectangle (eg, w1 or w2) estimated in FIG. 6, and for convenience of calculation, based on the coordinate axis of image information. The vertical height (h1, h2) or the horizontal width of the square can be substituted. w1 and w2 represent two opposite sides of a rectangle, and an angle between two adjacent sides corresponds to an angle. The length of the side is equal to the distance between the two vertices at both ends of the side, and the corner angle can be obtained by using the inner product of the two vectors. h1 and h2 correspond to the difference between the Y-axis values of the two vertices of the vertical side of the rectangle.

8 is a diagram illustrating an example of a rectangle estimated from a binarized image.

Referring to FIG. 8, when the first license plate region detection unit 230 binarizes the input image (left) and estimates (right) the square, the license plate region is not clearly visible inside the estimated square r (black pixels, e. ) May be included.To prevent such a square r from being determined as the license plate area, the black and white ratio (binarization ratio) of the pixel values inside the square is calculated. Can be excluded.

Specifically, in FIG. 8, the ratio of the black pixel value inside r may be larger than the ratio of the black pixel value inside the binarized license plate area normally detected because of the e portion. Since the ratio of values can be easily calculated in advance, it can be determined whether to determine r as the license plate area based on the pre-calculated value.

In this way, the first license plate region detection unit 230 can easily determine the license plate region based on the binarization ratio.

9 and 10 are diagrams illustrating an example of an algorithm performed by the first license plate area detection unit.

9 and 10, the first license plate area detection unit 230 according to an embodiment performs pre-processing (pre-processing) such as bilateral filtering on the input image when it is performed N times and each time (I-th) is performed. After creating and determining the license plate candidate area, normalize the license plate area (image warping) and convert it into a rectangle of the same size to achieve the same effect as seeing an inclined license plate from the front, and to obtain the same effect as seeing the inclined license plate from the front, and to obtain the black and white of the pixel values of the binary license plate area. If the ratio (binarization ratio) is less than a specific value (40), the normalized license plate candidate region is determined as the first license plate region, and can be included as one of the N first license plate region candidates. It can be configured to have to re-detect the license plate area.

In FIG. 9, since the first license plate region is determined by determining the binarization ratio after determining the license plate candidate region, the license plate candidate region determination and the first license plate region determination are distinguished, but unlike FIG. 9, the binarization ratio for the determined license plate candidate region The determination may be regarded as being included in the method of determining the license plate candidate region, and the determined license plate candidate region and the first license plate region may be viewed identically.

The generation of the license plate candidate region and determination of the license plate candidate region will be described in FIGS. 11 to 14.

11 and 12 are diagrams illustrating an example of an algorithm for generating a license plate candidate region performed by a first license plate region detection unit.

Referring to FIG. 11, the first license plate area detection unit 230 according to an embodiment modifies (pre-processes) an input image, and a noise level (noise intensity) of 10 is random noise (e.g., Gaussian random noise). ) Is added.

In addition, the first license plate region detection unit 230 may binarize the image in this state or add random noise again before performing binarization. When performing binarization or adding noise, the first license plate region detection unit 230 may use different binarization threshold values Thr or different noise levels NL as described above.

At this time, the first license plate region detection unit 230 uses various binarization thresholds Thr when performing binarization, and each binarization threshold Thr is set to have a fine difference (+3) from 20 to 255. As described above, even if the value of the pixel at the license plate boundary (A) and the value of the pixel surrounding the license plate boundary (B) are similar, it is possible to distinguish between A and B into black and white through binarization, thereby increasing the accuracy of license plate region detection. Can be improved.

In addition, at this time, the first license plate area detection unit 230 uses various noise levels (NL, noise intensity) when adding noise, and each noise level NL has a minute difference (+3) from 0 to 30. Since it is set, even if the value of the pixel (C) at the license plate boundary and the pixel (D) surrounding the license plate boundary are similar, the amount of noise added to C and D varies, so that C and D can be more clearly distinguished. . Therefore, it is possible to improve the accuracy of detecting the license plate area.

In addition, the first license plate area detection unit 230 adds random noise and estimates the outline from the binarized image, so that the ratio of the height (h) and width (w) of the rectangle including the outline is less than 2 and is composed of the outline. The license plate candidate area can be created by including the estimated contour in the license plate candidate area only when the area of the polygon is 9000 or less or the number of points (corners) of the polygon consisting of the outline is 4 and the area is 2400 or more.

Here, the height and width of the rectangle including the outline are the minimum and maximum values for each of the X and Y axes among coordinates of the vertices constituting the estimated outline as shown in FIG. It may correspond to the height and width of a rectangle with a diagonal line segment connecting coordinates consisting of. The number of points (corners) is as described in FIGS. 5 and 6.

In addition, here, the height of the rectangle including the outline, the ratio of the width, or the number of points (corners), and the width may vary according to the geometric characteristics of the license plate to be detected. FIG. 11 is only an embodiment, so the height and width of the rectangle The value of the ratio, the number of points (edges), and the area of is not limited to the values of FIG. 11.

In addition, since it is determined whether or not the estimated contour is included in the license plate candidate area according to the ratio of the height and width of the rectangle including the contour line or the number of points (corners), such a determination corresponds to the method of determining the license plate candidate area.

In this way, the first license plate region detection unit 230 not only changes the binarization threshold, but also adds random noise (eg, Gaussian random noise) and adds noise having changed the noise level (noise intensity) to the same input image. For each, various contours may be estimated to generate various license plate candidate areas.

Referring to FIG. 12, the first license plate area detection unit 230 may generate various binary images while changing noise and binarization threshold values added from the input image.

13 and 14 are diagrams illustrating an example of an algorithm for determining a license plate candidate area performed by a first license plate area detection unit.

13, the first license plate area detection unit 230 according to an embodiment estimates a polygon using the Ramer-Douglas-Peucker algorithm from the outline of the license plate candidate area for license plate candidate areas, and the edge of the polygon is 4 If it is a dog, that is, if a square is estimated, it is included in the license plate candidate area, and if a square is not estimated, it is excluded from the license plate candidate area.

At this time, the first license plate area detection unit 230 uses various values as parameters related to the polygon estimation accuracy when estimating the polygon, and each Ratio value has a minute difference (+0.1) from 10 to 15. Since it is set so that various polygons can be estimated, the accuracy of detecting the license plate area can be improved.

Here, the Ramer-Douglas-Peucker algorithm estimates a polygon while removing some of the points (vertexes, corners) constituting the outline.At this time, as the ratio increases, the polygon with fewer vertices is estimated. The probability can be higher.

In addition, the first license plate area detection unit 230 requires that the estimated corner angle value of the square is between 60 degrees and 120 degrees and the length difference values (w1-w2 and h1-h2) of opposite sides are less than 5, If not, it may be excluded from the license plate candidate area.

Here, the corner angles, w1, w2, h1 and h2 are as described in connection with FIG. 7 and the binarization ratio is as described in FIG. 8.

In addition, as shown in FIG. 9, if the binarization ratio determination is included in the method of determining the license plate candidate region, the binarization ratio must be less than 40 to be included in the license plate candidate region, otherwise it can be excluded from the license plate candidate region.

In addition, the first license plate region detection unit 230 may determine a license plate candidate region in which the sum of length difference values (sum of w1-w2 and h1-h2) of opposite sides of the square is the minimum as the final license plate candidate region.

Meanwhile, since specific methods and values for determining the license plate candidate area in FIG. 13 may be changed according to geometric characteristics of the license plate, the present invention is not limited to the method shown in FIG. 13.

Referring to FIG. 14, the first license plate area detection unit 230 estimates a rectangle according to the algorithm of FIG. 13, selects only rectangles close to the shape of a parallelogram or rectangle among the estimated rectangles as a license plate candidate area. Among them, the final license plate candidate area (first license plate area) is determined.

In this way, the first license plate region detection unit 230 generates a license plate candidate region in various ways and determines the license plate candidate region (first license plate region) based on the geometric characteristics of the license plate candidate region or the rectangle estimated from the license plate candidate region. The accuracy of license plate detection and recognition can be improved.

In addition, when the first license plate area detection unit 230 receives the license plate area and its surrounding area from the license plate detection unit 220, the number plate candidate is obtained from the license plate area detected by the license plate detection unit 220 and image information corresponding to the surrounding area. It is possible to correct the license plate region detected by the license plate detection unit 220 by creating an area and determining the license plate candidate region (first license plate region) based on the geometric characteristics of the license plate candidate region or the rectangle estimated from the license plate candidate region. Since the license plate candidate region is re-detected only for the license plate region and the surrounding region detected at (220), the speed and accuracy of license plate detection and recognition can be improved.

In addition, since the first license plate region detection unit 230 detects the license plate region again with respect to the license plate region detected by the license plate detection unit 220, the license plate detection model used by the license plate detection unit 220 may be supplemented. For example, the first license plate area detection unit 230 may supplement the deep learning model used by the license plate detection unit 220.

Since the second license plate area detection unit 240 is related to character area detection, the first character area detection unit 250 and the second character area detection unit 260 will be described.

A brief comparison of the first character area detection unit 250 and the second character area detection unit 260 before examining the first character area detection unit 250 is as follows.

Since both the first character area detection unit 250 and the second character area detection unit 260 detect a character area, they perform similar functions.

However, the first character area detection unit 250 is different from the second character area detection unit 260 in which the detection method is specified in that it can detect the license plate area in various ways. Here, a method used in the second character region detection unit 260 is excluded from various methods.

Hereinafter, the first character area detection unit 250 and the second character area detection unit 260 will be described.

The first character region detection unit 250 receives image information from the input unit 210 or from the license plate region detection unit 220, the first license plate region detection unit 230, or the second license plate region detection unit 240. 1 The image information corresponding to the license plate area (the first license plate area candidate) or the second license plate area (hereinafter, the license plate area) is transmitted, and the character candidate area can be detected using a character detection model from the received image information. The first character candidate area may be transferred to the second character area detection unit 260, the second license plate area detection unit 240, or the character recognition unit 270. In this case, the first character region detection unit 250 may detect the first character candidate region in various ways, and for example, an artificial neural network such as deep learning may be used.

15 to 17 are diagrams exemplarily showing an example of an algorithm performed by a first character region detection unit.

15 to 17, the first character area detection unit 250 according to an embodiment detects a first character candidate area from image information corresponding to an input image or license plate area using a character detection model, You can generate coordinate information.

Here, one input image of FIG. 16 can be transmitted from the input unit 210 or the second license plate area detection unit 240 and corresponds to the N first license plate areas (first license plate area candidates) of FIG. May be transmitted from the first license plate area detection unit 230. There is a difference between the input image of FIG. 16 and the first license plate region candidate of FIG. 17 in whether the license plate region is warped.

At this time, as described above, a deep learning model may be used as a character detection model. However, the character detection model is not limited to the deep learning model.

As described above, the first character area detection unit 250 detects the character area first before the second character area detection unit 260 or the second license plate area detection unit 240 performs detection, so that the second character area detection unit 260 It is possible to improve the accuracy of detecting the character region in the second license plate region detection unit 240 and narrow the range of the image in which the license plate region is to be detected, thereby improving the speed and accuracy of the license plate region detection.

The second character region detection unit 260 receives the character candidate region from the first character region detection unit 250 and merges or deletes the character candidate region, adjusts the size of the character candidate region, or adjusts the size of the character candidate region based on the geometric characteristics of the character candidate region. The character candidate area can be corrected by adding the character area to the character candidate area.

In this case, the second character region detection unit 260 may merge or delete the character candidate region or adjust the size of the character candidate region based on the height, position, or area of the overlapping region.

In addition, at this time, the second character region detection unit 260 calculates one or more straight lines in which the sum of the distances from the center position of the character candidate region becomes the minimum value, and detects a new character region whose center position is close to the straight line, so that the character candidate region is Can be added.

18 and 19 are diagrams illustrating an example of an algorithm performed by a second character region detection unit.

Referring to FIG. 18, the second character area detection unit 260 according to an embodiment adjusts (corrects) the character candidate area by receiving the character candidate area, detects a straight line, preprocesses and binarizes the image, and then determines the character candidate area. The primary character candidate group can be detected by adding (correcting) and adjusting (correcting) the character candidate area again. Here, the primary character candidate group may correspond to a character region that is detected relatively independently of the character arrangement format of the license plate.

By the way, the second character area detection unit 260 further corrects the character area according to the character arrangement format of a specific license plate (for example, a license plate of a specific country) based on the detected character area of the primary character candidate group. Can improve.

For example, in FIG. 18, the second character area detection unit 260 does not detect a part of the character area of the upper line or the lower line in a license plate in which 4 characters are arranged in the upper line and 5 characters are arranged in the lower line. If the average distance between neighboring character areas is calculated and the distance between adjacent character areas R1 and R2 is greater than the average distance multiplied by 1.3, the character area is detected again for the area between R1 and R2 Can be added to the area.

Referring to FIG. 19, the second character region detection unit 260 according to an embodiment merges or merges the character candidate regions even when a duplicate detection, erroneous detection, or undetected character candidate region is input from the first character region detection unit 250. The character candidate area can be corrected by deleting, adjusting the size of the character candidate area, or adding a new character area to the character candidate area.

The first character candidate region adjustment, straight line detection, character candidate region addition, and second character candidate region adjustment of FIG. 18 will be described in FIGS. 20, 22, 24, and 26, respectively.

20 and 21 are views exemplarily showing an example of a first character candidate area adjustment algorithm performed by a second character area detection unit.

20 and 21, the second character region detection unit 260 according to an embodiment receives M character candidate regions and compares two different character candidate regions, and the overlapping region between the two regions is 80% or more. If so, you can merge the two areas.

In addition, the second character region detection unit 260 calculates the average height of the character candidate region, and if the height of each candidate region is less than 60% of the average, it is regarded as an erroneous detected character candidate region and excluded (delete) from the candidate region. I can.

Also, the second character region detector 260 may adjust the height of the candidate region to the average height if the height of each candidate region is 60% or more of the average and less than 80%.

22 and 23 are views exemplarily showing an example of a straight line detection algorithm performed by a second character region detection unit.

22 and 23, the second character region detection unit 260 according to an embodiment receives M character candidate regions, obtains a straight line L passing through the center positions of two different character candidate regions, and M If the distance is less than D, the distance value is accumulated, and if it is greater than D, the distance value is not accumulated. The second character region detection unit 260 may set a straight line L having a minimum distance value of the M*(M-1) straight lines L obtained by this method as straight line 1.

Here, the center position of the character candidate area can be calculated from the coordinate values of the four vertices constituting the character candidate area. For example, the maximum and minimum values for each of the X and Y axes are obtained from the coordinate values of the four vertices. The value and the minimum value are summed and the summed value divided by 2 can be set as the center position of the character candidate area for the X and Y axes.

In addition, here, if the distance between the straight line L and the center position of the letter candidate area is greater than D, the reason for not accumulating the distance value is that when obtaining the straight line L passing through the upper line of the license plate in Fig. 23, the distance from the straight line L is greater than D. This is to exclude the character candidate area located on the line. Therefore, it is desirable to set the value of D to a sufficiently large value to exclude the character candidate area located in the lower line.

In addition, here, the reason why the straight line L, which has the minimum distance value for the straight line L, is set to straight line 1 is to obtain a straight line that best penetrates the center position of the letter candidate regions located in the upper line in FIG. 23, for example.

In addition, after obtaining the straight line 1, the second character region detecting unit 260 excludes a candidate region having a distance of D or less from the straight line 1 from the candidate region. For example, in FIG. 23, when straight line 1 is a straight line that best penetrates the center of the character candidate area located on the upper line, this is to obtain the straight line 2 for the character candidate area located on the lower line, excluding the character candidate area located on the upper line.

In addition, the second character region detection unit 260 may obtain the straight line 2 in the same manner as the method of obtaining the straight line 1 for the remaining P candidate regions excluding the candidate regions having a distance of D or less from the straight line 1 from the candidate region.

Here, finding straight line 1 and straight line 2 may only be useful if the license plate consists of two lines. That is, since detecting two straight lines in FIGS. 22 and 23 is only an example, the number of straight lines calculated by the second character region detection unit 260 is not limited to two.

24 and 25 are diagrams illustrating an example of an algorithm for adding a character candidate area performed by a second character area detection unit.

Referring to FIG. 24, the second character area detection unit 260 according to an embodiment calculates a straight line 1 and a straight line 2, deletes all existing character areas, and creates a character based on a center line (straight line 1 and straight line 2). Area can be extracted. Specifically, the second character region detection unit 260 extracts the outline of the binary image and performs labeling, and if the number of labeled regions is M, it satisfies a certain criterion by examining the size ratio of the labeled regions to the M labeled regions. The labeled area can be added to the character candidate area only if the center of the labeled area exists near the straight line 1 or the straight line 2, and the binarization ratio of the labeled area has been checked and passed.

Here, as an example of a criterion for checking the size ratio of the labeled area, the second character area detection unit 260 is used when the width of the rectangle surrounding the labeled area is less than twice the height or the area of the rectangle is within a certain range. Only if the criteria are satisfied can be determined.

Also here, the presence of the center of the labeled area (center position) near straight line 1 or straight line 2 means that the distance between the center of the labeled area (center position) and straight line 1 or straight line 2 is less than a certain value. can do. The center (center position) of the labeled area can be obtained by the same method as the method of obtaining the center position of the character candidate area described above.

In addition, here, as an example of checking the binarization ratio, the second character region detection unit 260 may determine to pass the check only when the binarization ratio is less than 40.

Referring to FIG. 25, even if there is an undetected character in the character candidate area received by the second character area detection unit 260, the second character area detection unit 260 detects binarization and outlines, and labels the detected outlines. Since is newly detected, when the center position of the character area is close to the calculated straight line 1 or the calculated straight line 2, the undetected character area may be added to the character candidate area.

Meanwhile, the second character region detection unit 260 may exclude the character candidate region if the distance between the center position of the character candidate region and the straight line 1 or the straight line 2 among the character candidate regions is a predetermined value or more. In this way, the second character area detection unit 260 may exclude the erroneously detected character area shown in FIG. 23 from the character candidate area.

26 and 27 are views exemplarily showing an example of a second character candidate area adjustment algorithm performed by a second character area detection unit.

Referring to FIGS. 26 and 27, the second character region detection unit 260 according to an embodiment receives M character candidate regions and compares two different character candidate regions, and the overlapping region between the two regions is 80% or more. In this case, the two regions are merged, and if the overlapping region between the two regions is less than 80% and more than 30%, the smaller of the two regions may be deleted from the candidate region.

In addition, the second character region detection unit 260 calculates the average height of the character candidate region, and if the height of each candidate region is less than or equal to 60% of the average, it is regarded as a falsely detected character candidate region and excluded from the candidate region (deleted). )can do.

Also, the second character region detection unit 260 may adjust the height of the candidate region to the average height if the height of each candidate region is greater than 60% of the average and less than or equal to 80%.

In this way, the second character region detection unit 260 may improve the accuracy of character region detection by supplementing the character candidate region detected by the first character region detection unit 250.

In addition, the second character region detection unit 260 may supplement a character detection model used by the first character region detection unit 250. For example, the second character area detection unit 260 may supplement a deep learning model used by the first character area detection unit 250.

As described above, since the first character region detection unit 250 and the second character region detection unit 260 have been described, the second license plate region detection unit 240 will be described.

First, briefly, the second license plate area detection unit 240 is very similar to the second character area detection unit 260, but the biggest difference is that it detects the license plate area including the corrected character candidate area.

The second license plate area detection unit 240 receives the first character candidate area from the first character area detection unit 250 and merges or deletes the first character candidate area based on the geometrical characteristics of the first character candidate area. The first character candidate area may be corrected by adjusting the size of the candidate area or adding a new character area to the first character candidate area, and a second license plate area including the corrected first character candidate area may be detected.

At this time, the second license plate area detection unit 240 may merge or delete the first character candidate area or adjust the size of the first character candidate area based on the height, location, or area of the overlapping area. have.

In addition, at this time, the second license plate area detection unit 240 calculates one or more straight lines in which the sum of the distances from the center position of the first character candidate area becomes the minimum value, and detects a new character area whose center position is close to the straight line. Can be added to the character candidate area.

In addition, the second license plate region detection unit 240 may warping the license plate region when detecting the second license plate region including the corrected first letter candidate region.

28 is a diagram illustrating an example in which a second license plate area detection unit is performed.

Referring to FIG. 28, the second license plate area detection unit 240 according to an embodiment corrects the character candidate area by receiving the character candidate area, detects a second license plate area including the corrected character candidate area, and detects the detected second license plate area. 2 The license plate area can be normalized and printed. Here, the correction of the character candidate region may be performed through at least one of merging, deleting, resizing, or adding the character candidate region.

In this case, the addition of the undetected character area may be performed by detecting a straight line passing through the center of the character area and newly detecting a character area centered on the detected line, similarly to the second character area detection unit 260.

In this way, the second license plate area detection unit 240 corrects the character candidate area detected by the first character area detection unit 250 and detects the second license plate area including the corrected character candidate area. Accuracy can be improved.

The character recognition unit 270 may recognize characters included in the license plate based on the license plate area or the character candidate area, and may use a deep learning model for character recognition.

29 and 30 are diagrams illustrating an example of an algorithm performed by a character recognition unit.

29 and 30, the character recognition unit 270 according to an embodiment may receive a license plate image and a character candidate region and recognize a character corresponding to the character candidate region. At this time, when a specific rule is determined for the character arrangement of the license plate, the speed and accuracy of character recognition can be improved by applying the rule to recognize the character.

29 shows an algorithm for recognizing a license plate in which only numbers are arranged in the lower row, only the third position is arranged in the upper row, and numbers are arranged in the remaining positions. Accordingly, it is possible to prevent the character candidate region from being erroneously recognized as a character other than a number by forcing the character candidate region of the lower line to be recognized only as numbers.

The recognition result inference unit 280 may infer a character recognition result by voting with the character recognized by the character recognition unit 270.

Specifically, the recognition result inference unit 280 detects the same image information by (i) the first license plate area determined by the first license plate area detection unit 230, and (ii) the first character area detection unit 250 and the second The second character candidate region corrected by the character region detection unit 260 and (iii) the second license plate region detected by correcting the character candidate region detected by the first character region detection unit 250 by the second license plate region detection unit 240 The character recognition result may be inferred by voting with the character recognized based on at least one of them.

For example, the recognition result inference unit 280 votes as a character string recognized based on the N number of first license plate areas determined by the first license plate area detection unit 230, and determines the most present character string among the N character strings as a final recognition result. Can be calculated as

In relation to (i), (ii), and (iii), it will be described in FIGS. 34 to 39.

31 and 32 are diagrams illustrating an example of an algorithm performed by the recognition result inference unit.

Referring to FIGS. 31 and 32, the recognition result inference unit 280 according to an embodiment receives character string candidates, and only when 4 characters exist for a character string above the candidate character string and the character string candidate is 2 or more, If the maximum number of voting times and the second maximum number of voting are different from each other, the character with the maximum number of voting is inferred from the recognition result. Characters can be inferred from recognition results.

In addition, the recognition result inference unit 280 differs only in that the candidate character string is a voting target only when five characters exist for the lower line character string of the candidate character string, and the recognition result is determined in the same manner as the algorithm for the upper line character string. I can infer.

31 shows an algorithm for voting a character string for a license plate in which 4 characters are arranged in the upper row and 5 characters are arranged in the lower row, and thus is only an example of the inference of the license plate recognition result.

As described above, since the recognition result inferring unit 280 infers the recognition result by voting the characters recognized in various ways, the accuracy of license plate recognition may be improved.

The transmission/reception unit 290 may enable the input unit 210 to receive image information from the camera 300.

The database 292 may store image information and the like.

This database 292 is a concept including a computer-readable recording medium, and refers to not only a database of consultation, but also a database in a broad sense including data recording based on a file system, and searches for a simple set of logs. Thus, if data can be extracted, it is included within the scope of the database referred to in the present invention.

The controller 294 controls the overall operation of the vehicle license plate detection and recognition system 200, and the input unit 210, the license plate detection unit 220, the first license plate region detection unit 230, the second license plate region detection unit 240, Control flow or data flow between the first character area detection unit 250, the second character area detection unit 260, the character recognition unit 270, the recognition result inference unit 280, the transmission/reception unit 290, and the database 292 Can be controlled.

33 is a flowchart illustrating an example of a vehicle license plate detection and recognition method 500 according to the present invention.

Referring to FIG. 33, a vehicle license plate detection and recognition method 500 according to an embodiment includes an input step (S510), a license plate detection step (S520), a first license plate region detection step (S530), and a second license plate region detection step. (S540), a first character region detection step (S550), a second character region detection step (S560), a character recognition step (S570), and a recognition result inference step (S580).

In the input step (S510), the vehicle license plate detection and recognition system 200 may receive image information including the vehicle license plate from the camera 300 through the transceiver 290 or receive image information from the database 292. have.

In the license plate detection step S520, the vehicle license plate detection and recognition system 200 may detect the license plate region from the image information transmitted in the input step S510. At this time, the vehicle license plate detection and recognition system 200 may detect the license plate area in various ways, and for example, deep learning may be used.

In the first license plate region detection step (S530), the vehicle license plate detection and recognition system 200 generates a license plate candidate region from the image information input in the input step (S510), and A license plate candidate area (first license plate area) can be determined based on geometric features.

In addition, the vehicle license plate detection and recognition system 200 generates a license plate candidate region from the image information corresponding to the license plate region and the surrounding region detected in the license plate detection step (S520), The license plate candidate area (the first license plate area) can be determined based on the geometric characteristics of.

At this time, the vehicle license plate detection and recognition system 200 may binarize the image information based on different threshold values, or add different noises to the image information to binarize the image information, and estimate the outline to generate a license plate candidate region.

In addition, at this time, the vehicle license plate detection and recognition system 200 is based on the number of corners, corner angles, side lengths, widths, or the ratio of the binarized pixel values, estimated from the license plate candidate region or the license plate candidate region. The first license plate area) can be determined.

In the second license plate area detection step (S540), the vehicle license plate detection and recognition system 200 determines the first character candidate area based on the geometrical characteristics of the first character candidate area detected in the first character area detection step (S550). Merge or delete, adjust the size of the first character candidate area, or add a new character area to the first character candidate area to correct the first character candidate area, and detect the second license plate area including the corrected first character candidate area. can do.

At this time, the vehicle license plate detection and recognition system 200 merges or deletes the first character candidate area or adjusts the size of the first character candidate area based on the height, location, or area of the overlapping area. I can.

In addition, at this time, the vehicle license plate detection and recognition system 200 calculates one or more straight lines in which the sum of the distances from the center position of the first character candidate area becomes the minimum value, and detects a new character area whose center position is close to the straight line. 1 Can be added to the character candidate area.

Further, the vehicle license plate detection and recognition system 200 may warping the license plate region when detecting the second license plate region including the corrected first letter candidate region.

In the first character region detection step (S550), the vehicle license plate detection and recognition system 200 includes the image information or license plate region detection step (S520) received in the input step (S510), the first license plate region detection step (S530), or The character detection model is used from image information corresponding to the license plate area, N number of first license plate areas (first license plate area candidates), or the second license plate area (hereinafter, license plate area) received in the second license plate area detection step (S540). Thus, the first character candidate area can be detected. At this time, the vehicle license plate detection and recognition system 200 may detect the first character candidate region in various ways, and for example, deep learning may be used.

In the second character area detection step (S560), the vehicle license plate detection and recognition system 200 determines the first character candidate area based on the geometric characteristics of the first character candidate area detected in the first character area detection step (S550). The first character candidate area may be corrected by merging or deleting, adjusting the size of the first character candidate area, or adding a new character area to the first character candidate area.

At this time, the vehicle license plate detection and recognition system 200 merges or deletes the first character candidate area or adjusts the size of the first character candidate area based on the height, location, or area of the overlapping area. I can.

In addition, at this time, the vehicle license plate detection and recognition system 200 calculates one or more straight lines in which the sum of the distances from the center position of the first character candidate area becomes the minimum value, and detects a new character area whose center position is close to the straight line. 1 Can be added to the character candidate area.

In the character recognition step (S570), the vehicle license plate detection and recognition system 200 may recognize the characters included in the license plate based on the license plate area or the first character candidate area, and use a deep learning model for character recognition. .

In the recognition result inference step (S580), the vehicle license plate detection and recognition system 200 may infer the character recognition result by voting with the text recognized in the character recognition step (S570).

Specifically, the vehicle license plate detection and recognition system 200 uses (i) the first license plate region determined in the first license plate region detection step (S530), and (ii) the first text region detection step (S550) for the same image information. The second character candidate region detected and corrected in the second character region detection step (S560) and (iii) the character candidate region detected in the first character region detection step (S550) are corrected in the second license plate region detection step (S540). Thus, a character recognition result may be inferred by voting with a character recognized based on at least one or more of the detected second license plate areas.

Here, in relation to (i), FIGS. 34 and 35 are examined, in relation to (ii), FIG. 36 is examined, and in relation to (iii), FIGS. 37 and 38 are described.

34 and 35 are flowcharts illustrating an example of voting with characters recognized based on the license plate region determined in the first license plate region detection step.

Referring to FIG. 34, a vehicle license plate detection and recognition method 500 according to an embodiment includes an input step (S510), a license plate detection step (S520), a first license plate region detection step (S530), and a first character region detection step. (S550), the second character region detection step (S560), the character recognition step (S570), and the recognition result inferring step (S580) to the recognized character based on the license plate region determined in the first license plate region detection step (S530). You can infer the character recognition result by voting.

In FIG. 35, only the second character region detection step S560 in FIG. 34 is not performed.

The vehicle license plate detection and recognition method 500 in the first license plate region detection step (S530) of FIGS. 34 and 35 may determine N different first license plate region candidates, as shown in FIG. 8. For the region candidates, N character strings may be recognized through character region detection steps (S550 and S560) and character recognition step (S570), respectively, and the character recognition result may be inferred by voting with N character strings.

36 is a flowchart illustrating an example of voting with a character recognized based on a second character candidate area detected in a first character area detection step and corrected in a second character area detection step.

Referring to FIG. 36, a vehicle license plate detection and recognition method 500 according to an embodiment includes an input step (S510), a license plate detection step (S520), a first character region detection step (S550), and a second character region detection step. (S560), through the character recognition step (S570) and the recognition result inference step (S580), the second character candidate region detected in the first character region detection step (S550) and corrected in the second character region detection step (S560) The character recognition result can be inferred by voting with the character recognized as the basis.

When each step of FIG. 36 is performed as it is, the vehicle license plate detection and recognition method 500 can recognize one character string for the same image information input in the input step (S510), and recognize the character by voting with one character string. The results can be inferred.

However, when several different image information is generated by adding random noise to the image information input in the input step S510 immediately after the input step S510 of FIG. 36, several character strings can be recognized. Therefore, it is possible to infer the character recognition result by voting with several recognized strings.

37 and 38 are flow charts exemplarily showing an example in which a character candidate region detected in a first character region detection step is corrected in a second license plate region detection step and voting with a recognized character based on the detected second license plate region to be.

Referring to FIG. 37, the vehicle license plate detection and recognition method 500 according to an embodiment includes an input step (S510), a license plate detection step (S520), a first character region detection step (S550), and a second license plate region detection step. (S540), a first character area detection step (S550), a second character area detection step (S560), a character recognition step (S570), and a recognition result inference step (S580), and then the first character area detection step (S550). The character candidate region detected in is corrected in the second license plate region detection step (S540) and voting with the recognized text based on the detected second license plate region may infer the character recognition result.

38 does not go through only the second character region detection step S560 in FIG. 37.

When each step of FIGS. 37 and 38 is performed as it is, the vehicle license plate detection and recognition method 500 can recognize one character string for the same image information input in the input step (S510) and voting with one character string. Thus, the character recognition result can be inferred.

However, when several different image information is generated by adding random noise to the image information input in the input step S510 immediately after the input step S510 of FIGS. 37 and 38, several character strings are used. Since it can be recognized, it is also possible to infer the character recognition result by voting with several recognized character strings.

39 is a flowchart illustrating an example of voting using all of the methods of FIGS. 34 to 38.

Referring to FIG. 39, a vehicle license plate detection and recognition method 500 according to an embodiment recognizes N character strings, respectively, using the methods of FIGS. 34 and 35, and uses the methods of FIGS. 36, 37, and 38. Therefore, since each character string is recognized, the recognition result can be inferred by voting a total of 2*N+3 strings.

As described above, although it has been described with reference to the preferred embodiments of the present application, those skilled in the art will variously modify the present application within the scope not departing from the spirit and scope of the present invention described in the following claims. And it will be appreciated that it can be changed.

100: vehicle license plate detection and recognition system
200: vehicle license plate detection and recognition device
210: input unit 220: license plate detection unit
230: first license plate area detection unit 240: second license plate area detection unit
250: first character region detection unit 260: second character region detection unit
270: character recognition unit 280: recognition result inference unit
290: transceiver unit 292: database
294: control unit
300: camera 400: license plate
500: vehicle license plate detection and recognition method

Claims (16)

An input unit receiving image information including a vehicle license plate;
A license plate detection unit for detecting a license plate area including a license plate through an artificial neural network based on the image information by the input unit;
A first license plate region detector configured to filter the license plate region to generate a license plate candidate region, and determine registration or deletion of the generated license plate candidate region by comparing a plurality of filtering results having different filtering criteria;
A first character region detection unit for detecting a character candidate region from the license plate candidate region through an artificial neural network;
A second character area detection unit performing correction for merging or deleting character candidate areas based on the height or area of the character candidate area; And
Vehicle license plate detection and recognition system comprising a character recognition unit for recognizing characters included in the license plate based on the corrected character candidate area.
The method of claim 1,
The filtering criterion is at least one of a binarization threshold value and at least two noises, and the first license plate region detection unit generates the license plate candidate region by estimating an outline as a result of the filtering.
The method of claim 2,
The first license plate area detector estimates a rectangle based on the outline, and at least one of the estimated height, width ratio, area, number of corners, corner angle, side length, and binarized pixel value ratio A vehicle license plate detection and recognition system, characterized in that it is determined whether a predetermined criterion is satisfied and the license plate candidate region is included in the character candidate region.
delete An input unit receiving image information including a vehicle license plate;
A first character region detection unit for detecting a character candidate region from the image information using a character detection model;
A second character region detection unit correcting the character candidate region based on geometric characteristics of the character candidate region; And
And a character recognition unit for recognizing a character included in the license plate based on the corrected character candidate area,
The second character region detection unit merges or deletes the character candidate region or adjusts the size of the character candidate region based on the height, position, or area of the overlapping region,
A vehicle license plate detection and recognition system for calculating one or more straight lines in which the sum of distances is a minimum value from the center position of the character candidate area, detecting a new character area whose center position is close to the straight line, and adding it to the character candidate area.
The method of claim 5,
Vehicle license plate detection and recognition system further comprising a second license plate area detection unit for detecting a license plate area including the corrected character candidate area.
delete delete delete Receiving image information including a vehicle license plate from an input unit;
Detecting a license plate region including a license plate through an artificial neural network, in a license plate detection unit based on the image information from the input unit;
Generating a license plate candidate region by filtering the license plate region in a first license plate region detection unit, and comparing a plurality of filtering results with different filtering criteria to determine registration or deletion of the generated license plate region;
Detecting a character candidate region from the license plate candidate region through an artificial neural network by a first character region detection unit;
Performing correction of merging or deleting character candidate regions based on the height or area of the character candidate region in a second character region detection unit; And
A method for detecting and recognizing a vehicle license plate, comprising the step of recognizing a character included in the license plate based on the corrected character candidate region by a character recognition unit.
The method of claim 10,
The filtering criterion is at least one of a binarization threshold value and two or more noises, and the first license plate region detection unit generates the license plate candidate region by estimating an outline as a result of the filtering.
The method of claim 11,
The first license plate area detector estimates a rectangle based on the outline, and at least one of the estimated height, width ratio, area, number of corners, corner angle, side length, and binarized pixel value ratio A method for detecting and recognizing a vehicle license plate, comprising determining whether a predetermined criterion is satisfied, and including the license plate candidate region in a character candidate region.
delete Receiving image information including a vehicle license plate by an input unit;
Detecting a character candidate area from the image information using a character detection model by a first character area detection unit;
Correcting the character candidate region based on geometric characteristics of the character candidate region by a second character region detection unit; And
And recognizing a character included in the license plate based on the corrected character candidate area by a character recognition unit,
The second character region detection unit merges or deletes the character candidate region or adjusts the size of the character candidate region based on the height, position, or area of the overlapping region,
A vehicle license plate detection and recognition method in which one or more straight lines in which the sum of distances is the minimum value from the center position of the character candidate area are calculated, and a new character area whose center position is close to the straight line is detected and added to the character candidate area. delete delete

Images