KR102001002B1 - Method and system for recognzing license plate based on deep learning - Google Patents

Abstract

The present invention relates to a number recognition system based on deep learning. According to one embodiment of the present invention, the number recognition system based on deep learning comprises: a communication unit receiving an image photographed by a camera; and a detection unit recognizing one or more objects matching a pre-stored criterion based on the image received from the communication unit.

Description

[0001] METHOD AND SYSTEM FOR RECOGNIZING LICENSE PLATE BASED ON DEEP LEARNING [0002]

The present invention relates to a deep learning based number recognition system.

In recent years, smart traffic has been used in urban planning for smart city by analyzing traffic volume in real time based on image recognition technology, and researches on violation of traffic regulation violation and security have been actively carried out.

In the case of the existing intermittent system, the loop detector is used for detecting the violation, but it is difficult to apply it because the vehicle and the motorcycle are mixed in the sensor of the traffic environment in Southeast Asia.

Further, in the case of detecting the speed of the vehicle using the laser and the radar sensor, the motorcycle having a small size is mixed with the automobile, so that there is a problem that it is not recognized properly.

Furthermore, in emerging countries where traffic conditions are poor, motorcycles and automobiles are intertwined, causing many confusion. At the intersection, a large number of motorcycles and vehicles are simultaneously experiencing traffic violations such as signal violations. Therefore, the incidence of traffic accidents is high, and 75% of these accidents are caused by motorcycles.

Especially, it is difficult to control the traffic police due to the large number of motorcycles during the commute time, and a solution is required to effectively crack down on them.

The present invention provides a deep learning based number recognition system using only a CCTV camera without an LPR (License Plate Recognition) camera or a loop detector.

Further, the present invention is for effectively recognizing a license plate of a motorcycle in which a license plate is disposed on the rear surface.

The present invention is also intended to improve the accuracy of recognition of license plates using regression analysis.

According to one aspect of the present invention, a deep learning based number recognition system is a number recognition system for photographing an object passing through two predetermined points, comprising: an object passing through a predetermined first point and a predetermined second point from a camera A detection unit which receives the first frame and the second frame including the received frame and detects at least one of the object whose matching rate with the previously stored object is equal to or greater than a predetermined numerical value based on the received frame; And when the object detected by the detection unit corresponds to a predetermined reference, a frame photographed at a point where the distance from the point at which the object was photographed and the camera at the received frame is shortened, And a recognizing unit for recognizing the user.

A detection unit that detects at least one object stored in the n-th frame based on the image received from the camera and an object whose matching rate is equal to or greater than a predetermined numerical value; And a recognition unit for recognizing the license plate in the object detected by the detection unit and recognizing the number of the object in the license plate of the recognized object (n is an integer of 1 or more).

And a processing unit for determining whether the object detected in the n-th frame corresponds to a preset reference when at least one object detected in the n-th frame is present, It is possible to collect a frame including an object corresponding to a predetermined criterion.

And a processing unit for calculating the velocity of the object detected in the n-th frame when there is at least one or more objects detected in the n-th frame, wherein the processing unit determines that the velocity of the calculated object exceeds the predetermined velocity It is possible to collect a frame including an object exceeding a preset speed.

The recognition unit recognizes license plate information including the number of recognized license plates, position information of license plates, reliability of license plates, and type of license plates, compares the recognized characters with pre-stored characters in the license plate of the object, It is possible to select a license plate having a higher reliability and a license plate having the highest reliability.

The recognition unit may also determine one of the recognized characters from the at least two character regions overlapping when at least two of the recognized character regions in the selected license plate are partially overlapped have.

In addition, the determined character may be a character corresponding to the largest value among the determined character reliability values.

Further, the processing unit may determine a center point of character regions including the determined character, and determine a straight line as a center line with a minimum sum of distances from a center point of character regions including the determined character.

In addition, the processing unit may determine a character recognized at the upper end with respect to the center line as a character of one line, and a character recognized at the lower end with respect to the center line as a character of two lines.

In addition, the processing unit determines that the characters are arranged in two rows when the characters are recognized at the upper and lower ends with respect to the center line, and when the center line passes through the recognized character area The second type of plate can be determined by arranging characters in one line.

In addition, the processing unit can match characters in order from the left to the right in the order of smaller x-coordinate of the center point on the basis of the x-coordinate of the center point of the character area including the character determined based on the determined number plate type have.

The distance from the center of the camera to the ground is H, the actual distance from the center of the camera to the bottom of the image screen taken by the camera at the intersection of the ground and the camera is L ₁ , The actual distance to the upper portion on the screen of the image taken by the camera is L ₂ , the distance between the point displayed on the image screen photographed by the camera and the center of the screen when the object is located at the first point is P, The distance between the point displayed on the image screen photographed by the camera and the center of the screen when the object is located at the second point is Q and the distance between the upper direction on the image screen photographed by the camera at the photographing center of the camera, The angle formed in the direction from the photographing center to the ground is the first angle alpha, The angle formed in the upper direction on the image plane captured by the camera at the center and in the lower direction on the image screen taken by the camera at the photographing center of the camera is the second angle? The angle formed by the direction toward the point A displayed on the image screen and the direction from the photographing center of the camera to the point orthogonal to the ground is the third angle alpha _P , And the angle formed by the direction toward the point B shown in Fig. 4A and the direction toward the point orthogonal to the ground at the photographing center of the camera is the fourth angle? _Q , and the processing unit calculates the angle? The actual distance between two points can be obtained.

Further, the processing unit may be configured to calculate the time at which the object detected at the nth frame passes the first point, the time at which the object detected at the nth frame passes through the second point, and the distance between the first point and the second point It is possible to calculate the velocity between the first point and the second point of the object as a basis.

According to another aspect of the present invention, a deep learning based number recognition method includes: Detecting at least one object having a matching rate with the object stored in the frame equal to or greater than a predetermined numerical value; Recognizing the license plate in the detected object and recognizing the number of the object in the license plate of the recognized object; Calculating the velocity of the object detected in the nth frame when there is at least one object detected in the Nth frame; Collecting a number plate of an object exceeding a predetermined speed when the speed of the calculated object exceeds a predetermined speed; And comparing the recognized characters with the recognized characters in the license plate of the collected object to select the license plate having the highest reliability and the highest reliability as the license plate having a higher sum of matching rates of the characters .

In addition, after the step of selecting the license plate having the highest reliability, the character recognized from the at least two character areas overlapping when at least two character areas of the recognized character areas in the selected license plate are partially overlapped, Determining one of the characters; Determining a center point of the determined character areas in the license plate and determining a center line at which the sum of the distances from the center points of the determined character areas is the minimum; Matching the characters according to the determined license plate type using the determined center line; And outputting the result.

According to an embodiment of the present invention, it is possible to provide a deep learning based number recognition system using only a CCTV camera without an LPR (License Plate Recognition) camera or a loop detector.

Further, according to the embodiment of the present invention, the license plate of the motorcycle having the license plate on the rear surface can be recognized effectively.

Also, according to an embodiment of the present invention, the accuracy of recognition of a license plate can be improved by using regression analysis.

1 is an overall block diagram of a deep learning based multi-object detection system in accordance with an embodiment of the present invention.
2 is a detailed block diagram of a deep learning based multi-object detection system according to an embodiment of the present invention.
3 is a flowchart of a deep learning based multi-object detection method according to an embodiment of the present invention.
4 is an exemplary diagram for explaining the deep learning based multi-object detection of FIG.
5 is a flowchart for explaining a method of calculating the velocity of a recognized object.
Fig. 6 is a view for explaining S20 in Fig. 5. Fig.
FIG. 7 is a flowchart specifically illustrating S20 in FIG.
8 is a flow chart showing a license plate area detection algorithm.
9 is a flow chart showing the license plate recognition algorithm.
FIG. 10 is a diagram for explaining the center line algorithm of S44 in FIG. 9; FIG.

Other advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims.

Unless defined otherwise, all terms (including technical or scientific terms) used herein have the same meaning as commonly accepted by the generic art in the prior art to which this invention belongs. Terms defined by generic dictionaries may be interpreted to have the same meaning as in the related art and / or in the text of this application, and may be conceptualized or overly formalized, even if not expressly defined herein I will not.

The terminology used herein is for the purpose of illustrating embodiments and is not intended to be limiting of the present invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. As used herein, the terms' comprise 'and / or various forms of use of the verb include, for example,' including, '' including, '' including, '' including, Steps, operations, and / or elements do not preclude the presence or addition of one or more other compositions, components, components, steps, operations, and / or components. The term 'and / or' as used herein refers to each of the listed configurations or various combinations thereof.

It should be noted that the terms such as '~', '~ period', '~ block', 'module', etc. used in the entire specification may mean a unit for processing at least one function or operation. For example, a hardware component, such as a software, FPGA, or ASIC. However, '~ part', '~ period', '~ block', '~ module' are not meant to be limited to software or hardware. Modules may be configured to be addressable storage media and may be configured to play one or more processors. ≪ RTI ID = 0.0 >

Thus, by way of example, the terms 'to', 'to', 'to block', 'to module' refer to components such as software components, object oriented software components, class components and task components Microcode, circuitry, data, databases, data structures, tables, arrays, and the like, as well as components, Variables. The functions provided in the components and in the sections ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' , '~', '~', '~', '~', And '~' modules with additional components.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings attached hereto.

1 is an overall block diagram of a Deep Learning based multi-object detection system 100 in accordance with an embodiment of the present invention.

Referring to FIG. 1, a multi-object detection system 100 receives an image of an object 20 taken from an image collection device 30. The multi-object detection system 10 is connected to the image collection device 30 via a network.

The deep learning based multi-object detection system 100 according to the present invention can improve the accuracy of performance for object detection while reducing the amount of computation and memory required in the deep-learning-based multi-class object detection engine.

In addition, when the region of the estimated objects is created, the multi-object is detected by removing the low-reliability bounding box among the bounding boxes overlapping each other using NMS (non-maximum suppression) technique.

2 is a detailed block diagram of a deep learning based multi-object detection system according to an embodiment of the present invention.

Referring to FIG. 2, the multi-object detection system 100 includes a communication module (not shown), a detection module 110, a tracking module 120, a classification module 130, and an analysis module 140.

First, the communication module receives the photographed image from the camera and detects one or more regions of interest (ROI) including the object. The communication module extracts a region of interest using a stixel technique. Here, the stick cell technique refers to a method of representing an image in the same group as a stick of a certain width instead of a pixel on the assumption that obstacle objects standing up on the road have the same parallax information, Thereby extracting a region of interest.

The detection module 110 recognizes one or more objects matching the previously stored reference based on the image received from the communication module. For example, the detection module 110 recognizes each object existing in the ROI by using the deep learning-based multi-object classification technique.

Here, the detection module 110 recognizes the classes of multiple objects using a deep learning technique, but can use a technique such as a random forest. Here, the random forest is an ensemble method in which a plurality of decision trees are randomly learned in machine learning. The random forest method consists largely of a learning step for constructing a plurality of decision trees and a test step for classifying or predicting input vectors. Random forests are used in various applications such as detection, classification, and regression.

Deep learning is also a technique used to cluster or classify objects or data. For example, computers can not distinguish dogs and cats from photographs, but people can easily distinguish between them. To do this, machine learning technology is used to enter data into a computer and classify similar things. In other words, Deep Learning is a sort of a photo similar to a stored dog photo, and the computer classifies it as an open photo.

In addition, the detection module 110 generates labeling data for learning using a labeling authoring tool. In the present invention, the labeling authoring tool is used to generate data regarding the object's type value and the object's position coordinates. The type value of an object can be divided into a car, a van, a bus, a truck, a front surface of a motorcycle, a rear surface of a motorcycle, and a side surface of a motorcycle. Here, the front, back, and sides of the motorcycle are distinguished by the reason that cars have license plates on both the front and rear sides, but generally motorcycles have license plates on the back.

The tracking module 120 recognizes at least one object similar to at least one object recognized by the detection module 110. [ Here, the tracking module 120 tracks the object based on the format value of the detected object. The tracking module 120 estimates the detected object using a correlation-filter based tracking technique. This tracking technique shows robust tracking performance even for objects whose shape changes with time like a human being.

In object tracking and data association, it is possible to minimize the computation time required for extracting feature vectors from each module by reusing multi-channel feature vectors.

Details of the tracking module 120 will be described later.

The classification module 130 determines at least one object recognized by the detection module 110 as a detection object and determines an object similar to at least one object recognized by the tracking module 120 as a tracking object.

FIG. 3 is a flow chart of a deep learning based multi-object detection method according to an exemplary embodiment of the present invention, and FIG. 4 is an exemplary diagram illustrating deep learning based multi-object detection of FIG.

Referring first to FIG. 3, the deep learning based multi-object detection system 10 of the present invention comprises the following steps.

First, an image is received from the image collection device 30, and the image is recognized in the system (S100). Thereafter, the detection module 110 recognizes at least one object having a pre-stored object format value and a matching rate equal to or higher than a specific numerical value in n (n is an integer of 2 or more) frame (S200).

Also, the tracking module 120 recognizes at least one object recognized by the detection module 110 in the n-th frame and at least one object whose similarity rate is not less than a specific numerical value (S300).

For example, the detected object may be classified into an image of a bounding box and an image of a previously detected image. If the image of the bounding box is matched with the image of the detected image, It can be judged. In the present embodiment, whether or not the object information is matched using the object information related to the image is confirmed, but the object information may use various information in the image.

Thereafter, the analysis module 140 compares the position coordinates of at least one object recognized by the detection module 110 with the coordinates of at least one object recognized by the tracking module 120 in the n-th frame, The coordinates of at least one object are determined (S400).

For example, the position coordinates of the object recognized in the n-th frame are the coordinates of the upper left, upper right, lower left, and lower right of the bounding box of the object.

The present invention can obtain the positional coordinates of the corresponding object recognized in the second frame by the average value of the position coordinates of the object recognized in the first frame and the position coordinates of the object recognized in the third frame. Using this method, data that can be missed in object detection can be supplemented with object tracking.

According to an embodiment of the present invention, object information in the present invention includes information on at least one of a format value of an object and a position coordinate of an object. The format value of the object includes information on at least one of the front face of the automobile, the motorcycle, and the rear face of the motorcycle. Here, in the case of motorcycles, since the license plate is only on the back side, it is necessary to distinguish the front side of the motorcycle and the rear side of the motorcycle in order to apply the technology to the traffic analysis and control system.

Further, the type value of the object can further distinguish the side of a motorcycle, a motorcycle driver wearing a helmet, a motorcycle driver not wearing a helmet, and a pedestrian. Depending on local traffic conditions, the object's type value can be added to anything through machine learning.

The deep learning based multi-object recognition system 100 of the present invention recognizes at least one object including the format value of the same object as the format value of the object detected in the n-th frame by the tracking module 120. [ Then, an object including the format value of the same object as the format value of the object detected in the (n + k) -th frame is determined as a tracking object, and the tracking object determined in the subsequent frame is traced (k is an integer of 1 or more).

For example, if the object's format value in the first frame is the back side of the motorcycle, and if the object's format value in the third frame is the back side of the motorcycle, the object is set as the tracking object, Generates data that tracks the objects behind the motorcycle in succession.

However, there are many vehicles and motorcycles on the road. Therefore, there is a need for a method to distinguish and track multiple objects.

Further, the tracking module 120 may determine whether the center point of the bounding box of the object including the format value of the object identical to the format value of the object detected in the nth frame and the object value of the object To determine the centerline by connecting the center points of the bounding boxes of the objects containing the type values of < RTI ID = 0.0 > Thereafter, when the object having the same format value as the detected object in the m-th frame is detected, the object closest to the vertical distance from the center point of the bounding box of the object detected in the m-th frame is traced And determines it as an object.

Furthermore, the tracking module 120 may be configured to detect the area of the bounding box of the object including the format value of the object identical to the format value of the object detected in the nth frame, the format value of the object detected in the (n + Determines the area of the bounding box of the object that contains the format value of the same object. If the area of the bounding box of the object determined in the n-th frame and the area of the bounding box of the object determined in the n + k-th frame are overlapped by a predetermined ratio or more, The object is determined as a trace object.

In this method, the position and motion information of the object can be estimated by referring to only the input image so far, so that it can be utilized in the field of image analysis or autonomous travel requiring real-time property.

5 is a flowchart for explaining a method of calculating the velocity of a recognized object.

The license plate recognition system 200 of the present invention stores an image of an object passing through a first boundary of two preset boundaries using an image collecting device 30 only without a loop detector and stores an image of an object passing through a second boundary. Then, the speed of the object is calculated by comparing the image of the object passing through the first boundary line with the image of the object passing through the second boundary line.

If the velocity of the calculated object is judged to be a speed violation, the license plate is recognized based on the image of the object passing through the first stored boundary line. Here, the image of the underlying object for recognizing the license plate can be determined as an image of the object passing through the first boundary and the image of the object passing through the second boundary, which is relatively easy to process the image.

Referring again to FIG. 2, the deep learning based number recognition system 200 includes a communication unit (not shown), a detection unit 210, a processing unit 230 and a recognition unit 220. A communication unit (not shown) receives the photographed image from the camera, and the detection unit 210 detects one or more objects that match the previously stored reference based on the image received in the communication unit (not shown). The processing unit 230 calculates the speed of the recognized object. In addition, the recognition unit 220 recognizes an object exceeding a predetermined speed in the frame collected by the processing unit 230, recognizes the license plate of the object in the recognized object, .

According to an embodiment of the present invention, the detection unit 210 detects at least one object having a match rate with the pre-stored object in the n-th frame or higher than a predetermined numerical value. Thereafter, when the processing unit 230 has at least one object recognized in the n-th frame, the speed of the object recognized in the n-th frame is calculated.

Referring to FIG. 5, a license plate of at least one object recognized in the n-th frame is recognized (S10), at least one object is recognized in the n-th frame, and when there is an object recognized in S10, Speed is calculated (S20). A detailed description of S20 will be given later with reference to Fig. 6 and Fig.

FIG. 6 is a view for explaining S20 in FIG. 5, and FIG. 7 is a flowchart specifically illustrating S20 in FIG.

When the actual distance between the pixel A and the pixel B is obtained on the camera screen (Screen) and the photographing time of the frame including the pixel A and the pixel B is compared, the speed of the object can be obtained using only the camera.

6, the actual distance from the center of the camera to the ground (the height at which the camera is installed) is H, the actual distance from the camera center to the bottom of the image plane photographed by the camera is L ₁ , The actual distance from the camera's center of gravity to the top of the image taken by the camera at the intersection of the ground and the camera is L ₂ .

When the distance between the point displayed on the image screen photographed by the camera and the center of the screen is P and the object is located at the second point when the object is located at the first point on the image screen photographed by the camera The distance between the displayed point and the center of the screen is Q.

The angle formed by the upper direction on the image screen taken by the camera at the photographing center of the camera and the direction from the photographing center of the camera toward the ground is the first angle alpha, The angle formed in the lower direction on the image screen taken by the camera at the upper direction and the photographing center of the camera is the second angle (?), The direction toward the point A displayed on the image screen photographed by the camera at the photographing center of the camera And an angle formed in a direction orthogonal to the ground plane from the photographing center of the camera is a third angle alpha _P and a direction from the camera center to a point B displayed on the image plane photographed by the camera, In the direction from the photographing center to a point orthogonal to the ground Degrees is the fourth angle (α _Q).

The processing unit 230 obtains the actual distance between the first point and the second point. The concrete procedure is as follows.

First, the distance D from the camera to the lower portion of the screen can be obtained by summing up the Pythagorean based on the length of H and L ₁ , by obtaining the angle α as arctan (L ₂ / H) (S 21, (S22, equation (2)). Angle θ is can be determined by the product of D and cos (θ / 2) obtained by arctan (L ₁ / H) at an angle α (S23, equation (3)), the distance C from the camera to the screen (S24, equation 4).

The screen can be obtained by multiplying sin (? / 2) by D and doubling it (S25, Equation 5). Next, θ _P is obtained by subtracting θ / 2 from angle α and adding arctan (P / C), θ _Q is obtained by subtracting θ / 2 from angle α and subtracting arctan (Q / C) 6).

Finally, the actual distance of the pixel A can be obtained by subtracting the product of tan (α _P ) and H at L ₂ , and the actual distance of the pixel B can be obtained by subtracting the product of tan (α _Q ) and H at L ₂ Equation 7). The speed of the object can be obtained by comparing the actual distance between the pixel A and the pixel B with the input time of the image.

[Equation 1]

&Quot; (2) "

&Quot; (3) "

&Quot; (4) "

&Quot; (5) "

&Quot; (6) "

&Quot; (7) "

The processing unit 230 determines the time at which the object detected in the nth frame passes through the first point, the time at which the object detected in the nth frame passes through the second point, and the distance between the first point and the second point To calculate the velocity between the first point and the second point of the object.

In other words, it detects the pixel B in the frame of the first image, and detects the pixel A in the frame after a certain time. And detects a speed of the object. If the detected speed corresponds to the speed of violation, a frame capable of easily recognizing the license plate of a corresponding object among the frames from the frame of the first image to a frame after a predetermined time is selected to perform license plate recognition .

For example, in the case of a motorcycle, a license plate is arranged on the rear side, so images are first taken in order to detect the speed, and if a speed violation is detected afterwards, the license plate must be recognized through regression analysis and the violation controlled.

8 is a flow chart showing a license plate area detection algorithm.

In the present invention, the processing unit 230 collects a frame including an object exceeding a predetermined speed when the speed of the calculated object exceeds a predetermined speed.

Referring to FIG. 8, data is received from the image collection device 30 (S30), and at least one license plate is recognized in the area of the object using the Yolo3 based localization network model (S31, S32). If there is not at least one recognized license plate, at least one license plate is recognized again in the area of the object.

When recognizing at least one license plate, the number of recognized license plates, position information of the license plate, reliability of the license plate, and type of license plate are recognized (S34). The license plate having the highest reliability of the license plate determined in step S34 is selected (S35). Here, the position of the license plate refers to the position of at least one of the left upper end, the lower left end, the right upper end, and the lower right end of the bounding box including the license plate, and if necessary, the center of the bounding box may be the position of the license plate. The reliability of the license plate is given according to the recognition rate of the stored character compared with the pre-stored character in the area of the object. In addition, the type of license plate varies depending on the type of vehicle and the time of release of the vehicle for each country. For example, there may be a license plate placed in one row or in two rows. In order to increase the recognition rate of the license plate, the license plate type is determined in the step of detecting the license plate area.

According to an embodiment of the present invention, the recognition unit 220 compares characters recognized in the license plate of the object with pre-stored characters, assigns a higher reliability to a license plate having a higher average value of the matching rates of the respective characters, Select the highest plate number.

FIG. 9 is a flowchart showing a license plate recognition algorithm, and FIG. 10 is a diagram for explaining a center line algorithm in S44 of FIG.

The license plate recognition algorithm determines one of the recognized characters from the at least two character regions overlapping when at least two of the recognized character regions in the selected license plate are partially overlapped.

The determined character is a character corresponding to the largest value among the determined character reliability values.

Referring to FIG. 9, the character of the license plate selected in step S35 is recognized (S410). The overlap character is removed based on the recognized character (S43). The reference of the overlap character is that the intersection area is 70% or more. Select the character with the higher score among the overlapped characters, and remove the characters excluding the selected character. Here, the score is given higher as the character recognition rate is higher.

Thereafter, characters are arranged using the type of license plate and the center line algorithm (S44), the recognized letters and license plates are matched (S45), and the result is outputted (S46).

Referring to FIG. 10, the centerline algorithm is performed as follows.

First, the center point of the recognized character bounding box is determined in the license plate area, and the line with the minimum distance from the center point of each character bounding box is determined as the center line. Since the line classification is not performed when the license plate is recognized, the line classification can be accurately performed on the basis of the determined center line, and the tilting of the license plate in the license plate area can be enhanced by correcting the tilted license plate.

According to the present invention, the processing unit 230 determines a center point of character areas including the determined character, and determines a straight line as a center line that minimizes a sum of distances from a center point of character areas including the determined character. A character recognized at the upper end is determined as a character of one line on the basis of the center line, and a character recognized at the lower end of the center line is determined as a character of two lines.

It is also possible to determine a license plate of a first type in which characters are arranged in two rows when characters are recognized at the upper and lower sides with respect to the center line and when the center line passes through the recognized character area, Lt; RTI ID = 0.0 > 2 < / RTI > The characters are sequentially matched from left to right in each line in order of decreasing x-coordinate of the center point based on the x-coordinate of the center point of the character area including the character determined based on the determined type of license plate.

For example, if the license plate type is determined by a plate divided into two rows, the upper bounding box based on the determined center line is a character arranged in one row, the lower bounding box based on the determined center line is arranged in two rows Can be done with the characters. Thereafter, characters can be arranged in order from the smallest size based on the x-coordinate value of the bounding box of each character, and characters can be matched according to the type of plate.

For another example, a license plate in which the license plate type is divided into one row will have its center line passing through the center point of each bounding box determined and based on the determined center line based on the x coordinate value of the bounding box of each character Characters can be arranged in order, in ascending order of size.

The present invention also provides a number recognition system for photographing an object passing through two predetermined points, comprising: a first frame including an object passing through a predetermined first point and a predetermined second point from a camera; A detection unit for detecting at least one object having a matching rate higher than a preset numerical value and a previously stored object based on the received frame, And a recognition unit for calling a frame photographed at a point where the distance between the photographed point and the camera is short and recognizing the license plate of the object based on the loaded frame.

And a processing unit for determining whether the object detected in the n-th frame corresponds to a predetermined criterion when at least one object detected in the n-th frame is present, And may collect frames that include objects corresponding to predetermined criteria if so.

The deep learning based number recognition system 200 described above takes a picture of the back side object of a motorcycle passing through the first boundary line of the two boundaries and, based on the back side object picture of the motorcycle passing through the second boundary line, It will recognize the license plate with a clearer image by calling up the image of the identified rear motorcycle object. Here, the reason for recognizing the license plate by calling the previously identified motorcycle rear object photograph is because the image obtained from the image with a relatively short distance is clear and the image processing is easy to perform.

According to an embodiment of the present invention, when the speed of an object calculated based on an image received from a camera exceeds a preset speed, an object exceeding a predetermined speed is identified, and license plates and numbers of the object are recognized .

In other words, the present invention recognizes the license plate number and the number of the identified object when the object identified based on the image received from the camera meets predetermined conditions. For example, in the event of a violation of traffic regulations, including a violation of a lane, a violation of a signal, a violation of a motorcycle helmet, a violation of a motorcycle rider, a violation of a lane-breaking, a violation of an intersection tail, You can recognize the license plate number and number of objects that violate the law.

A method of discriminating a violation of a traffic regulation is a method of recognizing an object and discriminating a traffic regulation violation by installing a hardware sensor installed separately in the lane. However, according to the present invention, it is possible to discriminate a traffic regulation violation with only a video image without a hardware sensor installed separately in a lane, so that a more effective intelligent traffic system can be constructed.

The foregoing detailed description is illustrative of the present invention. In addition, the foregoing is intended to illustrate and explain the preferred embodiments of the present invention, and the present invention may be used in various other combinations, modifications, and environments. That is, it is possible to make changes or modifications within the scope of the concept of the invention disclosed in this specification, within the scope of the disclosure, and / or within the skill and knowledge of the art. The embodiments described herein are intended to illustrate the best mode for implementing the technical idea of the present invention and various modifications required for specific applications and uses of the present invention are also possible. Accordingly, the detailed description of the invention is not intended to limit the invention to the disclosed embodiments. It is also to be understood that the appended claims are intended to cover such other embodiments.

10: Multiple object detection and license plate recognition system
20: Object
30: Image collecting device
100: Multiple object detection system
110: Detection module
120: Trace module
130: Classification module
140: Analysis module
200: Number recognition system
210:
220: recognition unit
230: Processing unit

Claims (15)

A number recognition system for photographing an object passing through two predetermined points,
Receiving a first frame and a second frame including an object passing through a predetermined first point and a predetermined second point from a camera, and receiving an object having a matching rate with the pre-stored object based on the received frame, A detection unit for detecting at least one of the detection signals;
If the object detected by the detection unit corresponds to a predetermined reference, a frame photographed at a point where the distance from the point at which the object was photographed and the camera at the received frame is retrieved, and the license plate of the object is recognized based on the loaded frame ; And
And a processing unit for distinguishing a line of characters from the recognized license plate and determining the type of the license plate,
Wherein the recognition unit comprises:
Recognizes the number plate on the object detected by the detection unit, recognizes the number of the object on the number plate of the recognized object, recognizes the number plate information including the number of the recognized number plate, the position information of the plate, the reliability of the plate, And comparing the recognized characters with the stored characters in the license plate of the object to give a higher reliability to the license plate having a higher average value of the matching rates of the respective characters and selecting the license plate having the highest reliability, Determining one character out of the recognized characters from the at least two character regions overlapping when at least two of the character regions overlap partially,
The processing unit includes:
Determining a center line of the character areas including the determined character and determining a straight line having a minimum sum of distances from the center points of the character areas including the determined character as a center line and dividing a line of characters based on the center line ,
A character recognized at the upper end with respect to the center line is determined as a character of two lines based on the center line,
A first type of number plate in which characters are arranged in two rows when characters are recognized at the upper and lower sides with respect to the center line, and when the center line passes through the recognized character area, A deep running based number recognition system for determining a numbered plate of a second type arranged.
A detection unit for detecting at least one object stored in the n-th frame based on the image received from the camera and an object whose matching rate is not less than a predetermined numerical value;
A recognizing unit for recognizing the number plate on the object detected by the detecting unit and recognizing the number of the object on the license plate of the recognized object; And
And a processing unit for distinguishing a line of characters from the recognized license plate and determining the type of the license plate,
Wherein the recognition unit comprises:
Recognizes license plate information including the number of recognized license plates, position information of the license plate, the reliability of the license plate, and the type of the license plate, compares the recognized characters with the stored characters in the license plate of the object, Selecting a number plate having the highest reliability and assigning the highest degree of reliability to the higher numbered plate and selecting the numbered characters of the at least two characters overlapping when at least two of the recognized character areas in the selected numbered plate overlap partially, Determining one of the recognized characters from the region,
The processing unit includes:
Determining a center line of the character areas including the determined character and determining a straight line having a minimum sum of distances from the center points of the character areas including the determined character as a center line and dividing a line of characters based on the center line ,
A character recognized at the upper end with respect to the center line is determined as a character of two lines based on the center line,
A first type of number plate in which characters are arranged in two rows when characters are recognized at the upper and lower sides with respect to the center line, and when the center line passes through the recognized character area, (N is an integer greater than or equal to 1).
3. The method of claim 2,
and a processing unit for determining whether the object detected in the n-th frame corresponds to a predetermined criterion when at least one object detected in the n-th frame is present,
The processing unit includes:
And acquires a frame including an object corresponding to a predetermined criterion when the calculated object corresponds to a predetermined criterion. 3. The method of claim 2,
and a processing unit for calculating the velocity of the object detected in the n-th frame when there is at least one or more objects detected in the n-th frame,
The processing unit includes:
And collects a frame including an object exceeding a preset speed when the speed of the calculated object exceeds a predetermined speed. delete delete delete delete delete delete The method according to claim 1,
The processing unit includes:
Based on the x-coordinate of the center point of the character area including the character determined on the basis of the determined number plate type, in order from the left side in each row in the order of smaller x-coordinate of the center point. 5. The method of claim 4,
The distance from the camera's center of gravity to the ground is H,
The actual distance from the photographing center of the camera to the lower end portion of the image screen captured by the camera at the intersection of the ground and the ground is L ₁ ,
The actual distance from the camera's center of gravity to the top of the image plane captured by the camera at the intersection of the ground and the camera is L ₂ ,
The distance between the point displayed on the image screen photographed by the camera and the center of the screen when the object is located at the first point is P,
The distance between the point displayed on the image screen photographed by the camera and the center of the screen when the object is located at the second point is Q,
The angle formed by the upper direction on the image screen taken by the camera at the photographing center of the camera and the direction from the photographing center of the camera toward the ground is the first angle [alpha]
The angle formed in the upper direction on the image screen taken by the camera at the photographing center of the camera and the lower direction on the image screen taken by the camera at the photographing center of the camera is the second angle [theta]
The angle formed by the direction from the photographing center of the camera toward the point A displayed on the image plane photographed by the camera and the direction from the photographing center of the camera to the point orthogonal to the ground is the third angle alpha _P ,
And an angle made in a direction toward the point where the ground and the perpendicular from the point B toward the image displayed on the screen captured by the camera in the shooting direction of the camera center and the center of the camera shooting a fourth angle (α _Q),
[Equation 1]

&Quot; (2) "

&Quot; (3) "

&Quot; (4) "

&Quot; (5) "

&Quot; (6) "

&Quot; (7) "

The processing unit includes:
Wherein the actual distance between the first point and the second point is obtained by the equations (1) to (7).
(Where A _Real is the actual distance from the intersection of the top of the image shot by the camera and the ground to the point A, and B _Real is the distance from the intersection of the top of the image shot by the camera to the ground, Actual distance)
13. The method of claim 12,
The processing unit includes:
th frame, a time at which the object detected at the nth frame has passed through the first point, a time at which the object detected at the nth frame has passed through the second point, and a distance between the first point and the second point, And the second point is calculated based on the difference between the first point and the second point. Detecting at least one object stored in the n-th frame based on the image received from the camera, the object having a matching rate equal to or higher than a predetermined numerical value;
Recognizing the license plate in the detected object and recognizing the number of the object in the license plate of the recognized object;
Calculating the velocity of the object detected in the nth frame when there is at least one object detected in the Nth frame;
Collecting a number plate of an object exceeding a predetermined speed when the speed of the calculated object exceeds a predetermined speed;
Comparing the characters recognized in the license plate of the collected object with pre-stored characters, selecting a license plate having the highest reliability and having the highest reliability as the license plate having a higher sum of matching rates of the respective characters is selected;
Determining one of the recognized characters from the at least two character regions overlapping when at least two of the recognized character regions in the selected license plate are partially overlapped;
Determining a center point of the determined character areas in the license plate and determining a center line at which the sum of the distances from the center points of the determined character areas is the minimum; And
And matching the characters according to the license plate type determined using the determined center line,
Wherein the step of matching characters according to the license plate type determined using the determined center line comprises:
Dividing a line of characters based on the centerline;
Determining a character recognized at the upper end with respect to the center line as a character of one line and a character recognized at the lower end with respect to the center line as a character of two lines; And
A first type of number plate in which characters are arranged in two rows when characters are recognized at the upper and lower sides with respect to the center line, and when the center line passes through the recognized character area, Determining a second type of number plate arranged in the second direction. delete

Images