KR102095685B1 - vehicle detection method and device - Google Patents

Abstract

The present invention relates to a vehicle detection method based on a Fast R-convolution neural network (CNN) and an abnormal parking and stopping vehicle identification system using the same. According to an embodiment of the present invention, the improved vehicle identification method using a top-down method comprises: a first step of acquiring an image through a camera; a second step of dividing an entire region of the image into a short-distance region and a long-distance region; a third step of dividing the image into a plurality of preset block units; a fourth step of deriving a feature map to which the plurality of block units are applied; a fifth step of determining whether an object included in the image is at least one vehicle; a sixth step of determining whether a license plate of the at least one vehicle and a vehicle number in the license plate can be identified when the object is the at least one vehicle; a seventh step of monitoring the first vehicle when the first vehicle, in which at least one of identification of the license plate and identification of the vehicle number is impossible, exists; and an eighth step of retrying the identification of the license plate of the first vehicle and the identification of the vehicle number in the license plate when movements of the first vehicle occur during monitoring operation.

Description

Vehicle identification method and device {vehicle detection method and device}

The present invention relates to a vehicle identification method and apparatus.

While the demand for vehicles is rapidly increasing due to economic growth and income increase, the manpower managing road conditions and traffic conditions is not reaching this. Therefore, many efforts have been made to overcome the current poor traffic management system with limited personnel.

As part of such efforts, development of an automatic recognition (including number recognition) system of a vehicle is being actively conducted. Through vehicle recognition or vehicle number (character) recognition, it can be effectively used in various fields such as traffic control, traffic investigation, stolen vehicle detection, entrance vehicle control, and parking facility management, and many studies have been conducted to date.

In particular, in the process of recognizing the characters on the license plate of the vehicle, robust processing methods should be considered because, unlike other character recognition, a distorted phenomenon occurs due to camera noise, lighting changes, weather, etc. due to environmental influence. However, in the case of the license plate area of the vehicle, the contents are limited due to its unique characteristics, and the structure is simple compared to the general character recognition (pattern recognition). For this reason, License Plate Recognition (LPR) can be said to be the most common system for efficiently operating environmental characteristics, increasing demand for vehicles, manpower supply, and resource management of parking spaces.

The LPR system or number recognition technology was first developed in 1976 in the UK. Over the next decades, as technology evolved and market demand increased, the LPR system grew steadily and gradually expanded in Southeast Asia and other European countries. As a result, the market for LPR systems in North America is also growing significantly. This led to a strong motivation for effective crime suppression and prevention technologies, which prompted the market to become more active.

Past vehicle number recognition (LPR) or Automated LPR (ALPR) systems use the Optical Character Recognition (OCR) on images (images) acquired from the camera to read the vehicle license plate (Surveillance Method) Was applied to. Recently, the parking management system is under the name of efficient parking space. Currently, the LPR system solves the burden of manpower supply and labor costs and the leakage of fees by setting the parking fee by the manager in relation to the parking environment, and the demand for the LPR system is constantly increasing and the technology is continuously changing and developing.

In addition, for reasons of public safety and personal security, CCTV (closed circuit television) is installed everywhere in the present, and it is still in progress, and the data recorded through CCTV is used as evidence for crime prevention and crime occurrence. have.

CCTVs (closed circuit televisions) are installed in various places in modern times for threats from public safety, facilities, and personal safety, or for the convenience of living.

These backgrounds play a role in monitoring, disaster, and disaster prevention through object detection and behavioral analysis of next-generation unmanned traffic control equipment and CCTV intelligent video solutions that can flexibly accommodate functions according to the continuous development of IT technology and respond to future environmental changes. To develop.

In particular, systems and solutions in the form of inducing compliance with traffic regulations such as unmanned traffic control equipment are being installed and operated as part of reducing traffic accidents and ultimately aiming for safety of road traffic.

In fact, if the effect of crackdown was highly dependent on the traffic engineering aspect in the past (2012 ~ 2013), nowadays, it requires a complex system with safety, welfare and convenience as well as traffic.

In addition, by improving the driver's driving mode through crackdown and securing traffic communication and traffic safety, it is also used to adjust the traffic accident according to changes in road and traffic conditions and to adjust the urban environment considering the serious distribution of traffic accidents.

Among the backgrounds that could be reflected in the characteristics of the industry, it can be easily examined in the field of intelligent transportation system (ITS).

It maximizes the efficiency of transportation by organically grafting it with information and communication technology, sensor and control technology, image analysis and processing technology across all areas of transportation, such as providing road and traffic information, operation of public transportation and freight vehicles. Not only that, it was also an opportunity to improve cost savings.

In addition, this is also explained in ITS Basic Plan 21 in Korea and has a basic plan to complete the state-of-the-art infrastructure in three stages.

In recent years, it is difficult to secure profitability for related companies in various industries to participate in the market, so it is pursuing new services through the development of business models and continuous technology development with the participation of private companies and the center of the marketable sector. It seeks to improve through a wide range of system designs and solutions such as business.

In this industry, the higher the reliability of the traffic video surveillance function, the lower the probability of an error in the system, the better it is possible to ensure the superiority of performance compared to the existing CCTV products and solutions to secure competitiveness and gain market advantage.

However, in order to manage and supervise the traffic flow in real time, the systems related to transportation must be expanded and the performance of reliable technology must be maintained.

However, in the current situation, it is difficult to obtain effective results in situations where video surveillance is performed using a camcorder, information is collected from a vehicle user driving in a road environment, or video collection cameras with unmanned traffic monitoring systems and methods are used. You have to rely on a passive system rather than an automatic one.

Therefore, a solution that can solve the above problems is required.

Republic of Korea Patent Office Registration No. 10-1565978

The present invention seeks to provide a user with a vehicle identification method and apparatus.

The present invention relates to an effective method for detecting a target vehicle in an intermittent system, and to solve the problem of calculating the bounding box in Max-Pooling according to the size, position, and distance difference for object detection in an existing convolutional neural network (CNN) environment. do.

Specifically, in the case of most conventional technologies, in the case of an illegally parked target vehicle, the target object is detected in a patterned form in order to analyze the characteristic components, whereas in the present invention, the same object changes according to the feature vector and position according to the size. And by setting the feature vector distribution to be analyzed differently according to the distance, it can be effectively processed compared to the existing method.

On the other hand, the present invention is to provide an improved vehicle detection method using a receptive filed top-down method and an abnormal parking vehicle identification system using the same.

Specifically, the present invention utilizes a DSSD method that effectively solves the problem of the SSD applied to the vehicle detection and classification method to solve the above-described problems, and a layer layer to improve accuracy of classification rather than processing speed. It is intended to provide a method and system that is applied so that only the specific layer structure can be used in the existing network structure.

In addition, the present invention is a method capable of controlling a vehicle through a re-identification and license plate recognition of a vehicle to be avoided when the object of the object to be controlled has a movement or when an object or a mask used in a screening phenomenon is removed. And by providing a system, it is possible to classify vehicles to be covered and avoided, and to improve the recognition rate for detection by analyzing moving components.

On the other hand, the technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems that are not mentioned are clearly apparent to those skilled in the art from the following description. Will be understandable.

A vehicle identification method, which is an aspect of the present invention for achieving the above technical problem, comprises: a first step of acquiring an image through a camera; A second step of dividing the entire region of the image into a short-distance region and a long-distance region based on a predetermined reference line in the image; A third step of dividing the image into a plurality of preset block units; A fourth step of deriving a feature map using the plurality of block units using at least one of a plurality of convolutional layers of a convolutional neural network (CNN); A fifth step of determining whether an object included in the image is at least one vehicle using the feature map; A sixth step of determining whether a license plate identification of the at least one vehicle and identification of a vehicle number in the license plate are possible when the object is at least one vehicle; A seventh step of monitoring the first vehicle when there is a first vehicle in which at least one of identification of the license plate and identification of the vehicle number is unavailable among the at least one vehicle; And an eighth step of retrying identification of the license plate of the first vehicle and identification of the vehicle number in the license plate when the movement of the first vehicle occurs during the monitoring operation, wherein the object is included in the remote area If it is, the feature map for the region including the object is derived using the plurality of convolution layers, and if the object is included in the short-range region, only a predetermined number of layers among the plurality of convolution layers By using it, a feature map for an area including the object can be derived.

In addition, when the object is included in the remote area in the second step, between the second step and the third step, a 2-1 step of converting the scale of the remote area may be further included. .

In addition, the fifth step includes: a 5-1 step of comparing the feature map associated with the object for each of the plurality of block units; A 5-2 step of determining a max pooing area in which the object exists most frequently among the plurality of block units; And a 5-3 step of determining whether the object is a vehicle based on the feature value of the vehicle based on the max polling area.

In addition, the 5-3 steps include steps 5-4 of setting a region of interest (ROI) based on the max polling area; Step 5-5 of setting a bounding box based on the set ROI; And comparing the feature value of the object in the bounding box with the feature value of the vehicle to determine whether the object is the vehicle.

In addition, the fifth step is performed by a deconvolutional single shot detector (DSSD) method, and the DSSD method is a single shot detector (SSD) method that additionally uses a deconvolution layer to improve identification speed. For this, a feed forward connection can be used.

In addition, the DSSD method can perform only 3x3 convolution in the process of obtaining the coordinates and class score of the object region detected through the SSD method.

In addition, the fifth step may be performed using coordinates derived according to an aspect ratio size in a region of the at least one vehicle detected through the DSSD method.

In addition, the coordinates may be derived based on the arrangement information of the camera and the position information of the license plate of the at least one vehicle obtained in advance.

Further, trajectory information is generated using the derived coordinates, and in the sixth step, it is determined whether identification of the license plate of the at least one vehicle and identification of a vehicle number in the license plate are possible using the generated trajectory information. You can.

Further, in the seventh step, the first vehicle may be a vehicle in which at least one of identification of the license plate and identification of the vehicle number is impossible by at least one of a person and an object other than the at least one vehicle.

On the other hand, another aspect of the present invention for achieving the above technical problem is a vehicle identification system, a camera for acquiring an image; Includes a control unit for dividing the entire region of the image into a short-distance region and a long-distance region based on a predetermined reference line in the image, and dividing the image into a plurality of preset block units, wherein the control unit includes a convolutional neural network A feature map to which the plurality of block units is applied is derived using at least one of a plurality of convolutional layers of a Neural Network (CNN), and the feature map includes the feature map. It is determined whether the object is a vehicle, and if the object is a vehicle, the vehicle number is recognized from the vehicle number area of the vehicle, and when the object is included in the remote area, the plurality of convolutional layers are used to A feature map for an area including an object is derived, and at least one object included in the image is obtained using the feature map. It is determined whether it is a vehicle, and if the object is at least one vehicle, it is determined whether identification of the license plate of the at least one vehicle and identification of a vehicle number in the license plate are possible, and identification of the license plate among the at least one vehicle and If there is a first vehicle in which at least one of the identification of the vehicle number is impossible, monitoring of the first vehicle is performed, and if movement of the first vehicle occurs during the monitoring operation, the license plate of the first vehicle The identification and identification of the vehicle number in the license plate can be retried.

The present invention can provide a vehicle identification method and apparatus to a user.

The present invention relates to an effective method for detecting a target vehicle in an intermittent system, and can solve the problem of calculating the bounding box in Max-Pooling according to the size, position, and distance difference for object detection in an existing Convolution Neural Network (CNN) environment. have.

Specifically, in the case of most conventional technologies, in the case of the main vehicle target vehicle, the target object is detected in a patterned form in order to analyze the characteristic components, whereas in the present invention, the same object changes according to the feature vector and position according to the size and By setting the feature vector distribution to be analyzed differently according to the distance, it can be effectively processed compared to the existing method.

In addition, the present invention can provide an improved vehicle detection method using a top-down method of receptive filed and an abnormal parking vehicle identification system using the same.

Specifically, the present invention utilizes a DSSD method that effectively solves the problems of SSDs applied to vehicle detection and classification methods, and specifies a layer layer configuration in an existing network structure to improve classification accuracy rather than processing speed. Only layer structures can be used.

In addition, the present invention is a method capable of controlling a vehicle through a re-identification and license plate recognition of a vehicle to be avoided when the object of the object to be controlled has a movement or when an object or a mask used in a screening phenomenon is removed. And by providing a system, it is possible to classify the vehicle to be covered and avoided, and by analyzing the moving component, the recognition rate for detection can be improved.

On the other hand, the effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the following description. Will be able to.

Figure 1 shows an example of a block diagram of a parking control system that can be applied to the present invention.
2 shows an example of a block diagram of a control unit applicable to the present invention.
3A and 3B are plan and side views of a structure in which a parking control system related to an example of the present invention is installed.
4 shows an example of a neural network structure related to the present invention.
5 shows an example of a convolutional neural network related to the present invention.
6 shows an improved F-CNNs based system flow diagram proposed by the present invention.
7 shows an example of Fast R-CNN based object detection in relation to the present invention.
8 illustrates an example of setting a block unit in an image in connection with the present invention.
9 shows an example of an AlexNet structure applied to the present invention.
10 shows an example of a structure for fixing the size of a region in the Faster R-CNN structure in connection with the present invention.
11 shows an example of a fixed area output value in relation to the present invention.
12 illustrates an example of a convolutional layer structure for detecting a spatial location area in an image in connection with the present invention.
13 shows an example of a feature vector component in a space for detecting an object region in connection with the present invention.
14 is a flowchart illustrating an object detection step applied to the present invention.
FIG. 15 shows experimental results, FIG. 16 shows experimental results according to object loss, and FIG. 17 shows experimental results showing improved processing results.
18 shows an example of an overall system configuration in connection with the present invention.
19 shows an example of a network structure of a single shot detector (SSD) in connection with the present invention.
20 shows an example of a network structure of a DSSD (Deconvolution Single Shot Detector) in connection with the present invention.
21 shows an example of an improved structure in the SSD and DSSD structures in the overall system in relation to the present invention.
22 shows an example of extracting an object trajectory from a detection result in relation to the present invention.
23 shows an example of judging the license plate covering from different object results in relation to the present invention.
24 is a diagram illustrating an example of judging a vehicle license plate when a data value of a trajectory extraction result is small in relation to the present invention.
25 is a flowchart illustrating a proposed abnormal vehicle detection and classification system module in connection with the present invention.
26 is a view illustrating the characteristics of experimental images and images in relation to the present invention.
FIG. 27 is a view showing the experimental results (processing speed) and image characteristics of the input image in FIG. 26.

Parking control system

Hereinafter, a preferred embodiment of the present invention will be described with reference to the drawings. In addition, one embodiment described below does not unduly limit the content of the present invention as set forth in the claims, and the entire configuration described in this embodiment cannot be said to be essential as a solution to the present invention.

Conventionally, in order to prevent traffic jams and congestion caused by illegal parking vehicles, surveillance cameras have been installed in prohibited parking areas to control illegal parking vehicles.

However, there is a problem in that a problem of false detection in which an object other than a vehicle is detected is frequently included in an image photographed by the detection camera because an object other than the vehicle existing around the forbidden parking area is included. In addition, there was also a problem that it is difficult to control the vehicle parked far away from the detection camera.

Hereinafter, the illegal parking control system proposed by the present invention will be described in detail.

1 shows an example of a block diagram of an illegal parking control system that can be applied to the present invention.

The illegal parking control system 10 includes a camera module 100, an output unit 200, a communication unit 300, a memory 400, an interface unit 500, a power supply unit 600, a control unit 700, etc. can do.

However, since the components shown in FIG. 1 are not essential, an illegal parking control system having more or fewer components may be implemented.

Hereinafter, the components will be described in turn.

The camera module 100 processes an image frame such as a still image or a video image obtained by photographing a prohibited parking area. The image frames processed by the camera module 100 may be stored in the memory 400 or transmitted to the outside through the communication unit 300. The processed image frame may be displayed on the display unit 210.

At this time, two or more cameras used in the camera module 100 may be provided according to the use environment.

In one example, the camera module 100 may include a detection camera 110 and an intermittent camera 120. However, the present invention is not limited to such a configuration, and a configuration including other imaging means may also be included.

The detection camera 110 photographs a forbidden parking area, and can detect whether a vehicle enters the forbidden parking area using the captured detection image.

The detection camera 110 may be implemented using a camera equipped with a fisheye lens. When a fish-eye lens having a wide angle of view is used, an image in an omnidirectional (360 °) region centered on the detection camera 110 may be captured and used as a detection image.

However, even if the detection camera 110 is installed, it is not possible to enlarge and view a specific point in the forbidden parking area, and thus, an intermittent camera 120 capable of performing pan, tilt, and zoom operations is required.

The intermittent camera 120 may perform at least one of a Pan, Tilt, and Zoom operation to photograph a license plate of a vehicle that is illegally parked in a parked prohibited area. The intermittent camera 120 may preferably use a LPR (License Plate Recognition) camera.

The LPR camera is equipped with a tilting device capable of tilting in the x-axis, y-axis, and z-axis directions, and a detailed description is omitted since this configuration is easy for a person having ordinary knowledge in the technical field to which this invention belongs. I will do it. The LPR camera has a configuration capable of photographing a zoom-in image or a zoom-out image with respect to a vehicle by rotating the camera at designated x and y coordinates.

Meanwhile, the output unit 200 is for generating output related to vision or hearing, and may include a display unit 210, a warning unit 220, and the like.

The display 210 may display (output) information processed by the illegal parking control system 10. For example, a user interface (UI) or a graphical user interface (GUI) associated with the illegal parking control system 10 is displayed. A still image or a video captured by the camera module 100, an image transmitted / received through the communication unit 300, a UI, a GUI, etc. may be displayed.

In addition, the display unit 210 may be used as a lighting for controlling the crime prevention area in the event of illegally stopping a vehicle parked at night or in a dark space or when the illegal parking control system 10 is used for crime prevention purposes.

The display 210 includes a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display ( Flexible Display), 3D display (3D Display) may include at least one.

The warning unit 220 outputs a signal for notifying the occurrence of an event in the illegal parking control system 10. An example of an event occurring in the illegal parking control system 10 is illegal parking of a vehicle in a prohibited parking area.

The warning unit 220 may output other forms in addition to an audio signal or a video signal, for example, a warning guide document, and in this case, a printer for outputting a warning guide document may be connected.

Since the video signal may also be output through the display unit 210, in this case, the display unit 210 may be classified as a kind of the warning unit 220.

On the other hand, the communication unit 300 is one or more modules that enable communication between the illegal parking control system 10 and the wired / wireless communication system or between the illegal parking control system 10 and the network where the illegal parking control system 10 is located. It may include. For example, the communication unit 300 may include a wired / wireless Internet module 310, a short-range communication module 320, and a location information module 330.

The wired / wireless internet module 310 refers to a module for wired / wireless internet access, and may be built in or external to an illegal parking system.

The wireless Internet technology includes a wireless LAN (WLAN) (Wi-Fi), a wireless broadband (Wibro), a world interoperability for microwave access (Wimax), a wideband code division multiple access (WCDMA), and a high speed downlink packet access (HSDPA). , Long Term Evolution (LTE) may be used.

The short-range communication module 320 refers to a module for short-range communication. The short-range communication technology may be Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, or the like.

The location information module 330 is a module for obtaining the location of the illegal parking system 10, and a representative example thereof is a Global Position System (GPS) module. According to the current technology, the GPS module calculates distance information and accurate time information from three or more satellites, and then applies trigonometry to the calculated information to obtain current location information in three dimensions according to latitude, longitude, and altitude. It can be calculated accurately. Currently, a method of calculating position and time information using three satellites and correcting an error of the calculated position and time information using another one satellite is widely used. In addition, the GPS module can calculate speed information by continuously calculating the current position in real time.

Meanwhile, the memory 400 may store a program for processing and control of the control unit 700, and temporarily store data input / output (eg, feature vector information such as still images and videos). You can also perform functions. The frequency of use for each of the data (for example, the frequency of use for each feature vector information) may also be stored in the memory 400.

The memory 400 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory, etc.). ), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), Magnetic A storage medium of at least one type of memory, magnetic disk, and optical disk may be included. The illegal parking control system 10 may operate in connection with a web storage that performs a storage function of the memory 400 on the Internet.

Meanwhile, the interface unit 500 serves as a passage to all external devices connected to the illegal parking control system 10. The interface unit 500 receives data from an external device, receives power, and transmits data to each component inside the illegal parking control system 10, or transmits data inside the illegal parking control system 10 to external devices. do.

For example, a port for externally connecting an image or video captured by the camera module 100, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module , An audio input / output (I / O) port, a video input / output (I / O) port, and the like, may be included in the interface unit 500.

Meanwhile, the power supply unit 600 receives external power and internal power under the control of the controller 700 and supplies power required for the operation of each component.

Meanwhile, the controller 700 controls the overall operation of the illegal parking control system 10. For example, it performs related control and processing for video shooting, illegal parking vehicle detection, and control data generation.

For example, the control unit 700 may receive an image photographed by the detection camera 110 and determine whether to control the vehicle when the vehicle stops in the forbidden parking area. In addition, when the illuminance of the parking stop prohibition zone is non-uniform, the control unit 700 detects the illuminance and adjusts the light intensity of the display unit 210 toward the bright area weakly, and the light intensity of the display unit 210 toward the dark area. Can be strongly adjusted.

The various embodiments described herein can be implemented in a computer- or similar device-readable recording medium using, for example, software, hardware, or a combination thereof.

According to a hardware implementation, the embodiments described herein include Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), It may be implemented using at least one of processors, controllers, micro-controllers, microprocessors, and electrical units for performing other functions. The described embodiments may be implemented by the control unit 700 itself.

According to the software implementation, embodiments such as procedures and functions described herein may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein. Software code can be implemented in a software application written in an appropriate programming language. The software code may be stored in the memory 400 and executed by the control unit 700.

On the other hand, Figure 2 shows an example of a block diagram of a control unit that can be applied to the present invention.

As shown in Figure 2, the control unit 700 that can be applied to an example of the invention is a vehicle recognition means 710, illegal parking stop determination means 720, license plate recognition means 730, filtering means 740, camera control Means 750, a memory control means 760 may be included. However, the configuration of the control unit 700 is not limited to this, and it may be implemented with more or less components.

First, the vehicle recognition means 710 receives a detection image of a parking stop prohibited area photographed by the detection camera 110 and may analyze feature vector information from the detection image using a feature vector.

The feature vector is a method of extracting and analyzing a feature vector for recognizing a vehicle that is an object of enforcement of the illegal parking control system 10, such as Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), Adaboost, BMA ( Block Matching Algorithm (HOG), Histogram of Oriented Gradients (HOG), Haar-like features, and Gabor Wavelet (filter) can be used.

Here, the SIFT algorithm is a representative algorithm for extracting a feature vector of an image, and has invariant characteristics in image rotation, scaling, movement, partial illumination change, and projective transform. The SIFT algorithm can extract properties such as location, scale, and direction of features in consideration of characteristics of a local image.

That is, in the first step, a sample point (or candidate pixel) is selected by searching for a maximum and minimum (extrema) in a scale space generated through a difference-of-gaussian (DOG) function (Scale-space extrema detection).

In the second step, keypoints are selected based on stability values.

In the third step, one or more directions are assigned to each key point (Orientation assignment).

As a final step, a keypoint descriptor is generated using local image gradients.

The detailed description of the SIFT algorithm is given in detail in the paper "Distinctive image features from scale-invariant keypoints".

In addition, the SURF algorithm is an algorithm that finds characteristics that are invariant to environmental changes by considering environmental changes such as size, lighting, and viewpoint from multiple images, while simultaneously showing performance comparable to the SIFT algorithm, which is known to have excellent performance. It is a greatly improved algorithm.

The detailed description of the SURF algorithm is described in detail in a paper entitled "Robust Interest Point Detection and Descriptor".

In addition, the HOG algorithm is based on the fact that the appearance and shape of a specific object in an image can be determined by the size of the gradient or the distribution in the corner direction.

The HOG algorithm can recognize a specific object by dividing the image into a number of regions called a cell, and creating a histogram of the inclination or corner direction for each cell.

A detailed description of the HOG algorithm is given in detail in the paper "Histograms of oriented gradients for human detection".

In addition, the Haar-like features algorithm can recognize a specific object by using a prototype that represents feature information of a line and a prototype that represents feature information of a corner. This is described in detail in the paper "An Extended Set of Haar-like Features for Rapid Object Detection".

In addition, a detailed description of the Gabor Wavelet (filter) algorithm is described in detail in "Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters."

Also, detailed description of the BMA algorithm is described in detail in "Combined shape and feature-based video analysis and its application to non-regid object tracking".

The feature vector information extracted by the feature vector is compared with learning data stored in the memory 400.

Since the learning data includes information of vector components of a feature vector for an image of a general vehicle, the vehicle recognition means 710 can determine whether there is a vehicle among the objects captured in the detection image and can recognize the vehicle. have.

In addition, in the process of recognizing the vehicle from the detection image, a linear nearest-neighbor search may be used, which is a method of determining according to the number of feature vectors having high similarity among the compared feature vectors. However, since the linear nearest point search needs to be compared with all the learning data stored in the memory 400, the calculation amount is too large and the calculation speed may be slow. Therefore, a method of speeding up search by creating an index related to the feature vector information in the learning data may be used.

In addition, the vehicle recognition means 710 may determine whether the vehicle enters the no-parking zone and the position where the vehicle has stopped in the no-parking zone.

To this end, a motion detection algorithm that detects the motion of a vehicle or an image detection algorithm that contrasts the background image of a no-parking zone with an image when parking a vehicle can be used. Preferably, both algorithms can be used for more accurate vehicle detection. have.

Meanwhile, the illegal parking stop determining means 720 may determine whether the vehicle has parked on the parking stop prohibited area.

That is, when the vehicle recognition means 710 recognizes the vehicle in the forbidden parking area by receiving the detection image captured by the detection camera 110, the illegal parking stop determination means 720 determines whether the vehicle has been illegally parked. You can decide whether or not to perform the crackdown process.

It cannot be concluded that the vehicle was illegally parked due to the fact that the vehicle was recognized in the detection image, and the vehicle was said to have been illegally parked only when the vehicle was stopped in a no-parking zone for more than a predetermined time.

Therefore, the illegal parking stop determination unit 720 may determine whether the vehicle is illegally stopped using the detection image photographed at a predetermined time interval.

That is, the illegal parking stop determination unit 720 controls the detection camera 110 to photograph the parking stop prohibited area again after a predetermined time has elapsed when initially detecting the presence of the vehicle in the parking stop prohibited area. To this end, the camera control means 750 may control the operation of the detection camera 110 together.

The vehicle recognition unit 710 receives the detected image captured after the predetermined time elapses, and analyzes the feature vector again to extract feature vector information from the detected image.

The extracted feature vector is compared with the learning data stored in the memory 400, and the vehicle recognition means 710 can determine whether there is a vehicle among the photographed objects in the detection image captured after a preset time elapse. Can be aware of.

When the illegal parking stop determining means 720 determines that the vehicle recognized in the detection image after a predetermined time has passed is the same as the vehicle recognized in the initial detection image, the vehicle is considered to have illegally stopped.

Conversely, when it is determined that the vehicle recognized in the detection image after the preset time elapses and the vehicle recognized in the initial detection image are not the same, it is recognized that the vehicle is not illegally parked, and the detection camera 110 is used again. The parked area is photographed and the vehicle is searched for entry.

On the other hand, the license plate recognition means 730 may receive the image captured by the intermittent camera 120 to recognize the characters of the license plate of the illegal parking vehicle.

For character recognition, one of three detection methods of the vehicle license plate position can be executed.

The first is to detect the feature region of the license plate using vertical and horizontal edge information from the captured image.

The second is to detect the position of the license plate by analyzing the scan data.

The third is to detect the correct license plate by directly searching for numbers and letters.

When the location of the vehicle license plate is detected, the character recognition (template + number) by template matching, character recognition by structure features (Korean vowel), and internal recognition algorithms reduce numbers and characters (consonants, vowels) to reduce misrecognition. ) By classifying features in detail, such as re-checking the recognized characters, minimizing errors in character decoding.

On the other hand, the filtering means 740 may determine whether to exclude the object from the crackdown by considering the size of the vehicle captured on the detection image.

In the case of using the detection camera 110 to photograph the detection image for the parking stop prohibited area, the size of the vehicle captured in the captured detection image varies depending on the distance from the detection camera 110 to the actual vehicle location.

That is, since the lower part of the detection image is a space close to the prop where the illegal parking control system 10 is installed, if the actual vehicle is located close to the detection camera 110, the vehicle captured on the detection image is located at the lower portion of the detection image The size of the vehicle captured on the detection image will be large.

Conversely, if the actual vehicle is located far from the detection camera 110, the vehicle captured in the detection image will be located in the upper portion of the detection image, and the size of the vehicle captured in the detection image will appear small.

In this way, the filtering means 740 may calculate the distance from the detection camera 110 to the actual vehicle position by using the position in the detection image of the vehicle recognized by the vehicle recognition means 710.

In addition, the filtering means 740 may calculate the predicted size of the vehicle image in the detection image using the calculated distance, and the size of the vehicle image on the detection image is significantly different from the expected size of the vehicle image. You can prevent false positives by excluding them from the target.

That is, it is calculated that the vehicle recognized by the vehicle recognition means 710 is close to the detection camera 110 because it is stamped on the lower portion of the detection image. If the size of the vehicle image on the detection image is small, the recognition is performed. An old vehicle is not actually a vehicle, but it is likely to be a false detection, so it can be excluded from enforcement.

Conversely, it was calculated that the vehicle recognized by the vehicle recognition means 710 is located on the upper part of the detection image and is far from the detection camera 110. If the size of the vehicle image on the detection image is large, the recognition is performed. The old vehicle is not actually a vehicle, but is likely to be a false detection, so this can also be excluded from enforcement.

Meanwhile, the camera control means 750 may control the overall operation of the detection camera 110 and the intermittent camera 120 belonging to the camera module 100.

The camera control means 750 may control the illegal parking control system 10 to take a detection image for the parking stop prohibited region using the detection camera 110 to control the vehicle stopping at the parking stop prohibited region.

In addition, the camera control means 750 may be controlled to take a detection image for the forbidden parking area again after a preset time to determine whether a vehicle stopped in the forbidden parking area is parked on the forbidden parking area. have. This may be controlled together with the illegal parking stop determining means 720.

In addition, the camera control means 750 receives the location information of the stopped vehicle determined by the vehicle recognition means 710, and by using this, the intermittent camera 120 can photograph the license plate of the vehicle stopped in the no-parking zone. In order to be able to control to perform at least one of the pan, tilt and zoom operation of the intermittent camera 120.

Meanwhile, the memory control means 760 may control the overall operation of various data input / output to the memory 400.

The vehicle recognition means 710 of the control unit 700 extracts a feature vector from the detection image, and compares the extracted feature vector with feature vector information for a general vehicle image in the memory 400.

To this end, the memory control means 760 may control the memory 400 to output learning data including feature vector information for a general vehicle image to the vehicle recognition means 710.

In addition, when feature vector information for a vehicle recognized by the vehicle recognition means 710 and / or the filtering means 750 does not exist in the training data of the memory 400, it can be used for comparison with the next training data. So, the feature vector information can be stored in the memory 400.

To this end, the memory control means 760 may control the feature vector information for the vehicle recognized by the vehicle recognition means 710 and / or the filtering means 750 to be input to the memory 400.

Hereinafter, the installation structure of the illegal parking control system having the above-described configuration will be described with reference to the drawings.

3A and 3B are plan and side views of a structure in which an illegal parking control system related to an example of the present invention is installed.

As can be seen in Figures 3a and 3b, the illegal parking control system 10 structure of the present invention may include a prop (12), a crossbar (14), a detection camera (110), an enforcement camera (120), and the like.

However, since the components shown in FIGS. 3A and 3B are not essential, an illegal parking control system having more or fewer components may be configured. Hereinafter, the installation structure of FIGS. 3A and 3B will be described.

The support 12 is installed to be fixed to the ground in or near the parking stop zone, and is preferably fixed to the ground by a bolt or inserted into the ground. The prop 12 may be installed at a sufficient height so that the intermittent camera 120 can photograph a large area.

The holding 12 may be formed inside the hollow so that the wires connected from the underground can be penetrated to supply power to the detection camera 110, the intermittent camera 120, the output unit 200, the communication unit 300, and the like. It is preferable that the material is formed of a metal material that does not corrode in rainwater or the like.

At least one crossbar 14 may be connected to the upper part of the post 12, and like the post 12, the inside thereof may be hollow, and the material is preferably formed of a metal material. .

The crossbar 14 may be positioned horizontally to the ground, and preferably may be disposed over a no-parking zone.

The detection camera 110 and the intermittent camera 120 may process images such as still images or videos obtained by photographing a prohibited parking area. The detection camera 110 and the intermittent camera 120 may be installed at the ends of the crossbar 14, and preferably, may be disposed over a no-parking zone to be intermittent.

As shown in FIG. 3A, the illegally stopped parking control system 10 is installed in the forbidden parking area, as shown in FIG. 3A, the vehicle 21 parked near the detection camera 110 and the vehicle 22 parked away from the detection camera 110. This can be.

As described above, when the detection image for the forbidden parking area is photographed using the detection camera 110, the vehicle 21 parked near the detection camera 110 is positioned at the bottom of the detection image.

Therefore, the filtering means 740 may calculate the distance from the detection camera 110 to the vehicle 21 using the location of the vehicle captured on the detection image.

In addition, the vehicle 22 parked far away from the detection camera 110 is located at the top of the detection image, from which the distance from the detection camera 110 to the vehicle 22 can be calculated.

Illegal parking control system using Fast R-CNN based vehicle detection

CCTVs (closed circuit televisions) are installed in various places in modern times for threats from public safety, facilities, and personal safety, or for the convenience of living.

Against this background, CCTV intelligent video security plays a role of video surveillance and disaster prevention through disaster detection and behavior analysis.

Existing object detection methods and methods use background modeling to detect due to the difference between the background and foreground, extract features using HoG, LBP, and color histogram, and detect and use them through machine learning such as Adaboost, SVM, and ASEF Filter. Doing.

Later, object tracking uses several machine learning techniques such as Basian tracking, particle filtering, Mean-shift, and Cam-Shift.

4 shows an example of a network structure of a neural network.

As shown in FIG. 4, the recent deep learning image recognition technology has been continuously developed based on CNN, and has been developed through RCNN, Fast R-CNN, and Faster R-CNN to a fast and high detection rate such as YOLO. .

These object detection and tracking (analysis) algorithms have been steadily developed and developed to a level that can be replaced by human cognitive abilities, and are being used and applied in all industries, including driverless vehicles.

It is being developed and applied to the recognition rate and the completeness of image security by applying the deep learning algorithm based on the recent CNN beyond the existing method from the CCTV image.

With the development of these technologies, it is a trend that is being introduced into application services for vehicles, and it is used to recognize the contents of video, natural language, voice, control, and communication in various systems ranging from mobile apps to autonomous vehicles. It is used to analyze complex data and various data types.

In particular, the trend is to increase the accuracy of vehicle detection in various industries such as illegal parking control, traffic control of buses, parking control, and autonomous driving, and utilize and introduce them.

The detector corresponding to the vehicle detection unit in the existing illegal parking control system selects a method of extracting feature information and detecting a vehicle according to a threshold value set based on this, or uses a difference between a background component and a component containing vehicle information Therefore, the vehicle is classified and used.

Such methods can be false detections that are recognized in the form of objects (persons, objects with similar characteristics, etc.) other than the vehicle depending on whether or not the feature information is lost. It is difficult to detect vehicle information of a desired target because it is vulnerable to changes in the environment by weather and time.

Therefore, unlike the existing methods for vehicle detection, this development uses a method using machine learning learning based on deep learning to apply the detection method to detect the characteristics of the vehicle and applies various learning patterns to increase accuracy. Detect complex and various types of vehicles in the road environment.

The deep learning method defined in most general terms is applied differently according to the applied technology method in the basic structure, and the accuracy of detection differs depending on the size of the learning data structure (layout), whereas it is used for vehicle detection of processing speed and distance. There are pros and cons that can appear effectively. In addition, even though the contents of the technology according to the application method are different, it is commonly referred to as deep learning due to the configuration in the same system structure.

First of all, if you introduce the meaning of deep learning, deep learning is a kind of machine learning, and it is used in various fields such as image recognition and speech recognition as well as natural language processing, and has good performance. Unlike the existing method of extracting vehicle feature information, in deep learning, it is a “feature quantity extraction”. In the existing method, a person had to specify each feature in the image, but in the case of deep learning, the machine automatically extracts features from the learning data. Learning about is a big difference.

This feature can lead to learning about feature extraction because it has the meaning of the neural network (NN: Neural network) network structure in the past, and there are numerous neurons (neurons) in the human brain and one neuron is different. It is connected to the flow of signals that receive or transmit signals to neurons, thereby making various information possible.

The part implemented in association with this structure is a neural network (NN: Neural network).

The structure shown in FIG. 4 is called a network (network) because several neurons of a neural network are connected, and when data to be learned enters the input layer, multiple inputs are received from the structure of the middle layer. Depending on the signal is sent to one other node (unit, connection strength) and output from the output layer (Output layer).

When three or more of these neural networks overlap, it is called a deep neural network (DNN) and machine learning using this is called deep learning.

By adjusting the weight of neurons in the structure of learning by inputting a large amount of data, it can be designed as a method to maximize the efficiency and effectiveness of learning because it is related to the signal strength of the output according to calculations such as the sum of weights or products of the connected strengths. .

Such a structure requires modification to process vast amounts of data, and can be a more complex structure when the amount of training data is large. Convolutional Neural Network (CNN), one of the features of this structure, is an omni-directional neural network that configures the connection pattern of neurons and is designed to perform processing between layers and consists of several convolutional layers. do.

The structure of CNN is suitable for 2D input data, and can be used and applied to video fields such as CCTV cameras.

5 shows an example of a convolutional neural network applied to the present invention.

5 is a CNN structure, similar to the existing neural network structure, but it extracts meaningful features through convolution and several layers are stacked, so feature extraction, dimensionality reduction through subsampling, and top layer And plays a role of classifying features transmitted from lower layers.

In this complex structure, each layer has a learning process, and has a process of obtaining an answer to a new input based on the learned content, and provides a variety of learning models by setting learning parameters (weights), etc. Can be applied.

For example, there are models such as a federated multi-layer perceptron (MLP) and a convolutional deep belief network (CDBN).

These models are used as a classification method rather than detecting an object in an image (image), and analyze and classify by calculating the similarity between the learned data and the input data.

In this development, it is necessary to use a learning model that targets detection or detection methods rather than classification to extract vehicle information. To proceed.

At this time, in the case of R-CNN, an area is designated centering on a target to be searched in advance, and an object is detected based on this.

However, if the number of target objects is specified as multiple, it is difficult to guarantee real-time performance due to the high cost of processing time.

In addition, a plurality of target vehicles exist in an image (image) for extracting vehicle information, and in some cases, objects (packaging boxes, two-wheeled vehicles, bicycles, wagons, etc.) having properties similar to vehicles exist. Although accuracy in the area is important, there are also systems that need to reflect processing speed such as Fast RCNN (Fast RCNN) to secure real-time.

In general, in the R-CNN and Fast RCNN approach, high accuracy and processing speed can be improved to detect (detect) objects in an image (image), while streaming processing between multi-channels in a real environment Since it does not solve the problem of accuracy and processing speed, in particular, there are many changes in the accuracy performance depending on the distance between objects at the camera position, and the processing speed according to the size of the image (image) is also measured. need.

Therefore, in this development, it is necessary to secure the system configuration on the basis of a multi-channel camera according to real-time processing to require high accuracy along with the processing speed, so it shows the structural improvement in the basic structure and the performance according to the optimization of the system. Was made. In particular, for the vehicle detection, the vehicle was recognized through selective determination of specific layer layers, accuracy and identification in the location section according to distance, and number recognition.

6 shows an improved F-CNNs based system flow diagram proposed by the present invention.

Referring to FIG. 6, in general, since convolutional-based features in CNN can obtain more complex, sophisticated, and abstract information than Haar-like-based features that interpret only the difference between horizontal and vertical blocks, It is more useful at processing speed than Haar-like features obtained with a small number of search kernels based on one location.

However, the obtained features have only simple information.

On the other hand, in CNN, various search sizes (kernels) can be used and features can be obtained, and processing speed is slower than Haar-like features, but it has features as complex and sophisticated information.

This feature has high accuracy for Haar-like features by using multiple layers of convolutional layers.

However, since CNN has a problem in processing speed to find features, it is necessary to modify the convolutional layer layer, and the ratio (size) or distance of the target object (vehicle) in the image (between the target objects from the camera) Distance) Depending on the location, processing speed and accuracy are affected, so instead of using the entire layer layer, select only the layer layer in a range that can be distinguished by the size of the vehicle and develop it as a processing speed and effective detection method.

Each step of FIG. 6 will be described below in detail.

First, steps S10 and S11, input and image calibration, are described.

The input of the image is digitized data obtained from the camera. The video is a target, and each frame (image) is composed of two-dimensional data.

In addition, the image correction designates a detection area between input coordinates to detect an object (vehicle) of interest in advance.

At this time, the section in the area is redefined to be divided into two areas, and an object (vehicle) detection area is performed for each area.

The part set with two areas is to distinguish between long distance and short distance.

This is because the object (vehicle) has different sizes even when the same object (vehicle) exists in the image, so the texture (texture) of the object (vehicle) is different in feature extraction. Information is displayed so that the input data of the processing result is reliable.

In addition, in the case of the first area among the two areas, that is, a short-distance object (vehicle) is detected, and the size of the object (vehicle) is close to the distance from the camera, thereby having a large object area.

In addition, in the case of the second area among the two areas, that is, a remote object (vehicle) is detected, and since the object (vehicle) has a small object area because the size of the object (vehicle) is far from the camera, there is an object of interest (vehicle). Since the size difference (distribution) between the unspecified objects is similar, it is set to an area by a scale conversion or a zoom in / out of the camera in the camera setting (function) to easily identify feature information in the area.

Next, step S12 will be described.

S12 ESKRP is an optional (Selected) layers within F-CNNs step, and applies deep learning technology to detect a desired target object (vehicle) as a step after setting input data and image correction.

In most cases, the range using deep learning sets a category within the range of detecting and recognizing objects, and defining it to calculate the values and distribution for the conditions.

In this system, a detector including a target object (vehicle) and a non-target object (person) is processed based on CNN.

At this time, the learning data for the objects is used for learning as VOC Pascal 2007, VOC Pascal 2012, MS COCO, Caltech pedestrian dataset, ImageNet 2012, 2014, MIOTCD dataset.

The general case of Fast-CNN for improving the processing speed on the basis of CNN applied to this system is shown in FIG. 7 below.

7 shows an example of Fast R-CNN based object detection in relation to the present invention.

Referring to FIG. 7, in order to detect an object (vehicle), a result is output in the form of a region from each layer based on CNN, and the values are reconstructed into a fully-connected network, and then each again. In the area result for the layer layer, it is set to reset (reposition) based on the size and position of the target object (vehicle).

In addition, the configuration of the Fully-Connected layer is composed of a convolution layer and a pooling layer, and has a structure of connecting or weighting each other according to parameters, extracting features from the convolution and pooling layers, and from the fully-connected layer. It refers to the process of determining which class (category) belongs to.

At this time, the image of the input data is divided into small block units, and feature information is extracted for blocks through each network in the CNN structure.

8 illustrates an example of setting a block unit in an image in connection with the present invention.

As shown in FIG. 8, the reason for processing in the form of a block unit is to process the number of outputs in a fixed form due to structural characteristics, and output the corresponding position and size information in the result of detecting an object (feature information, object vehicle). This is because the positioning is easy.

In addition, in FIG. 7, a selective search method is used as a method for region extraction from a feature map, and AlexNet is used for feature extraction.

At this time, it is used as a feature vector for n outputs in the fully connected layer, and classification is learned by using a support vector machine (SVM).

On the other hand, Figure 9 shows an example of an AlexNet structure applied to the present invention.

Referring to FIG. 9, the AlexNet structure is composed of a total of 8 layers, the first to fifth are convolutional layers, and the next three layers are fully connected layers.

For each of the parallel layers, each independently extracted feature for the image is extracted and trained.

In addition, as a result, in order to extract the region, the size of the feature map is used as input data according to the order of the AlexNet feature vector and the classifier, and at this time, the size must be converted to a fixed size.

As a result, there is distortion due to size conversion in the form of block units, but there is also a loss in the characteristic area due to the size and processing speed of the input.

In addition, if the size of the input data is large or small, the location and size of the result area due to information lost in the feature extraction process for each layer in the feature map is affected, and the learning patterns ( Feature information) is important, so it is important to select size information.

In this step, the output location and size information may be determined differently according to the location type of the short-distance object information or the far-distance object information.

In the present invention, in the process of processing in the object feature map and the respective layers, the input data is divided into two areas and is set and processed as input.

In addition, the reason why the input data is different is that the object characteristics that exist at short distances and distances are set to be processed by strictly classifying the accuracy and processing speed, and the loss of the connection structure between the layers in the AlexNet structure is lost. Should be minimal.

In addition, in order to extract the location and size information of the detection area in order to set the output position according to the fixed number of outputs in object feature extraction, the type of the detected area and classifier for each layer layer is changed to a single structure, and at this time, the maximum output of the object position The number is defined and displayed as the object detection area for the final output area.

10 shows an example of a structure for fixing the size of a region in the Faster R-CNN structure in connection with the present invention.

10, when there are different size area values of different positions in the object detection result, each size position is distributed differently and is calculated as a single output value, and is selected as an improvement method for accuracy.

The distribution of the output values for each region and the output values in which the regions are expressed in the region in the object information are as shown in FIG. 11 below.

11 shows an example of a fixed area output value in relation to the present invention.

In Figures 10 and 11, in order to set the maximum number of output objects that can be expressed at the corresponding position in n area values, it is reconstructed to have the setting value for the central position in each area, and the output signal according to the size is the primary array space. Select the position of the final object area by setting the position of.

For a result as shown in FIG. 11, when an image is input as a pixel in a convolution layer, it is generally composed of a matrix NxNx3, horizontal and vertical, and a channel (dimensional), and each channel is composed of R, G, and B.

In addition, when calculating the feature pattern for each channel in the convolution layer, it is calculated as the expression of the determinant and is independent of the size.

12 is reconstructed regions for region extraction in a convolution layer, and has an object position and size including all object feature information.

At this time, the shape of the feature map for detection of the feature region is shown in FIG. 13.

The detailed steps of object (vehicle feature) detection using the method of improving the reconstruction structure of the position output according to the fixed size between the convolutional layers in the CNN and the calculation of the feature vector are as shown in FIG. 14.

14 is a flowchart illustrating an object detection step applied to the present invention.

Referring to FIG. 14, an input image is acquired (S21), block units are generated (S22), Faster T-CNN is applied (S23), ROI characteristics are calculated (S24), and a bounding box is set ( S25), object recognition is performed (S26).

Meanwhile, in relation to object detection in illegal parking control, the calculation of the pooling layer in the Fast R-CNN structure in the object detection step is determined according to Equations 1 and 2 below.

Equation 1

Equation 2

로 정의한다.Equation 1 is the area of the object (object vehicle) for polling calculation.

Is defined as

In Equation 1, kth is a block unit region in an image and is used for comparison between learning data.

At this time, the polling result of the object (object vehicle) area is output to the polling layer of jth.

는 필터계수이고 이미지내의 블록단위에서 필터계수와의 컨볼루션을 의미하며 영역내의 후보영역을 찾고, 해당 높은 값을 계산하여 Max-pooling 영역을 계산한다.

Is the filter coefficient and means the convolution with the filter coefficient in the block unit in the image, finds the candidate area in the area, and calculates the high value to calculate the max-pooling area.

At this time, it is determined that an object of interest is included in at least the area as a problem, but it may be selected as an inaccurate area of the feature vector in the case of areas with background component rather than the object area and when the size of the object of interest is small.

In order to correct such distorted (lost) information, it is calculated by Equation 3 below.

Equation 3

에 대해서 손실정보를

로 나타내며, 수학식 1에 의해

는 max-pooling 인텍스를 나타낸다. In Equation 3, the input variable

About loss information

Denoted by Equation 1

Denotes a max-pooling index.

This represents the location information found in each area after calculating the convolution.

는 미분에 의해서 계산되고 손실값을 나타낸다.

Is calculated by derivative and represents the loss value.

에 누적된다.At this time, the area extraction results for all ROI

Accumulate on.

Through this iterative process, the Region of Interest (RoI) pooling is performed for the Region of Interest (RoI) generated by the Region proposal network (RPN), and all feature maps for each region are created in the same fixed size. . Through this, objects (objects) in each Region of Interest (RoI) are classified. Object classification is based on the support vector machine (SVM), whether it is an object or not, and if the object is correct, only the object, and in this system, only the vehicle object is determined.

Finally, for the final object (object), the bounding box is calculated for the final output, and at this time, linear regression is simply used.

The calculation of the bounding box is as shown in Equation 4 and Equation 5 below.

In Equation 4, BBO represents the most ideal bounding box.

Equation 4

Equation 5

는 Max pooling에서 인덱스된 영역을 말한다. In Equation 4, G is the ground truth box,

Is the area indexed in Max pooling.

Equation (5) calculates the range by overlapping between the corresponding areas to generate the most ideal bounding box.

The system divides this process into two areas at the input stage and improves the problem of polling, thereby improving the accuracy of long-distance and short-range objects.

Returning to FIG. 6 again, after step S12, a step (S13) of extracting the license plate area proceeds, and if successful, the number is recognized (S14), and if it fails, the detected frame is stored (S15).

Hereinafter, experimental results and results of applying the present invention to illegal parking will be described.

FIG. 15 shows experimental results, FIG. 16 shows experimental results according to object loss, and FIG. 17 shows experimental results showing improved processing results.

As a vehicle detection method in illegal parking control, after detecting the feature vector of the convolutional layer layers on the basis of Fast R-CNN, the object detection due to the bounding box setting in the polling step shows differently distributed results according to the object size, and this is applied to the result of object detection. It occurred as a factor that could degrade the performance of the system by influencing it.

In order to improve this problem, it is possible to improve the performance of object detection within the original and near areas by setting the area in the input step.

Figure 15 shows the result of using the proposed method (based on Fast R-CNN) to detect a vehicle subject to illegal parking control in a real environment.

At this time, the input data was used without dividing into two areas in the image, but a problem was not found in detecting the target vehicle in the short distance, but in the case of a long distance, the characteristics and ROI due to the ambiguous boundary in the interval of the vehicle or a single vehicle type As the result of Max Pooling is different from the pooling result, it was analyzed as a factor that has inaccurate data.

In the case of FIG. 16, as in the results of FIG. 15, the feature distribution for some regions is similar to the surrounding region or the feature vector components according to the size are differently distributed, resulting in inaccurate results.

In addition, in the case of FIG. 17, in order to improve the results of FIGS. 15 and 16, the two areas of the input data were divided and performed by the same improved method, and the difference between the object size and POI polling was used to differentiate the desired target vehicle. It is the result of improving the accuracy in detection.

As described above, the present invention can provide the user with an illegal parking control system using Fast R-CNN based vehicle detection.

In addition, the present invention relates to an effective method for detecting a target vehicle in an illegal parking control system, and calculation of a bounding box in Max-Pooling according to a size, position, and distance difference for object detection in an existing convolutional neural network (CNN) environment You can solve the problem.

Specifically, in the case of most conventional technologies, in the case of an illegally parked target vehicle, the target object is detected in a patterned form in order to analyze the characteristic components, whereas in the present invention, the same object changes according to the feature vector and position according to the size. And by setting the feature vector distribution to be analyzed differently according to the distance, it can be effectively processed compared to the existing method.

Improved vehicle detection method using receptive filed top-down method and abnormal parking vehicle identification system using the same

In order to manage and supervise the traffic flow in real time, the systems related to traffic must be expanded and the performance of reliable technology must be maintained.

However, in the current situation, it is difficult to obtain effective results in situations where video surveillance is performed using a camcorder, information is collected from a vehicle user driving in a road environment, or video collection cameras with unmanned traffic monitoring systems and methods are used. You have to rely on a passive system rather than an automatic one.

Therefore, this development report utilizes the video analysis and analysis technology in unmanned traffic monitoring related to enforcement among systems that intensively manage and supervise images through CCTV cameras, while securing the technology of the product against the performance of the existing system and unmanned with high reliability. It is the development of traffic surveillance enforcement (illegal parking control) system.

In order to fundamentally eradicate illegal parking and stopping vehicles that interfere with the smooth flow of traffic, cameras can be installed in areas where illegal parking and parking are routinely carried out to control and guide them at all times.

The characteristics of such a system refers to a system that installs a camera in a road environment to detect moving objects (generally people, cars), and supervises and manages traffic accidents and flows through the information, and requires prompt / accurate processing and support. .

A system having this purpose is not only for storing the image of the road environment, but is used as a level of simple monitoring and controls, manages, and supervises the flow unless the function through detection / detection and recognition of objects is supported. It is difficult to obtain a practical effect.

Recently, with the development of machine learning, especially deep learning technology, incorporating artificial intelligence and image recognition technology, it has been showing the most remarkable achievements in various industries, and medical imaging technology, autonomous vehicle, smart home, security and imaging. In various fields such as the surveillance industry, visual intelligence of industrial robots, service robots and intelligent robots, smart factories, defense and aerospace, healthcare, consumer electronics, smart shopping, and entertainment, it has become the basis for realizing new values for industry and society.

As the era has changed from PC to mobile, the next paradigm is artificial intelligence (AI), and while focusing on the greater development of artificial intelligence both domestically and internationally, AI and image recognition technology beyond our related technologies such as natural language processing and voice recognition are ours. It has become a part of industry and society.

In the existing image detection technology, the object detection method is detected by the background modeling method due to the difference between the background and the foreground, and features are extracted using the histogram of gradient (HOG), local binary pattern (LBP), and color histogram and adaboost. , SVM (Support Vector machine), ASEF (average of Synthetic Exact) Filter, etc. are used for detection through machine learning.

Thereafter, in continuous monitoring of object detection results or object tracking, Bayesian tracking, Particle filtering, Mean-shift, and Cam-Shift are used.

Recently, by applying artificial intelligence and deep learning technology, the accuracy of detection has been improved and applied to the system.

This is the basic element technology in the field of image analysis and computer vision for object classification and detection.

In most cases, the improvement of the algorithm for detection technology related to the event in the image has been intensively improved, but an environmental factor to emphasize the importance of data (all image data including collection, classification, analysis, and results) to improve the accuracy within object classification. Because (event detection within the category to be solved) is complex and diverse, the results of subjective comparative analysis have appeared, and the boundary is the object detection and classification extraction method in a structure such as CNN (Convolutional Neural Network), which is the core of AI-based vision technology. The concept of (grid), YOLO (You only Look Once), and SSD (Single Shot Detector) were applied.

For these reasons, the range and category of data are subdivided or system structure is improved to solve causes in complex and diverse environments, thereby increasing diversity and accuracy for more accurate object classification or detection.

This assumes that when a person sees an image, he can easily comprehend the details of the objects in the image and perform complex actions.

However, in recent years, due to detection systems such as R-CNN, complicated processing processes have appeared to be partially insufficient to mimic the human visual system.

Examples include slow processing speeds, optimization methods, and structural improvements in current systems. The representative object detection and classification methods include SSD (Single Shot Detector) and YOLO (You Only Look Once).

YOLO divides each image into S x S grids and calculates the reliability of the grid. Reliability reflects the accuracy when recognizing objects in the grid.

This initially sets the boundary grid away from object recognition, but by calculating the reliability and adjusting the position of the boundary grid, the position of the highest boundary grid is obtained, and when it is recognized, it has the accuracy of the object.

For this reason, the bounding box and class probability in the image are regarded as a single regression problem, and the type and position of the object are estimated when the image is viewed once. This method is structured to calculate class probability for multiple bounding boxes through a single convolutional network.

Compared to the existing object detection method, the characteristic of YOLO is a very simple process, and the speed is very fast, and it shows twice the memory performance compared to other real-time detection systems. As you can see the whole thing at once, it improves the understanding of the class.

On the other hand, it is characterized by showing relatively low accuracy. Since the number of false positives is relatively small considering the context for the entire image using the full-connected layer, there is no restriction on the size of the bounding box to be output. There are also disadvantages.

On the other hand, there is a single shot detector (SSD) method with a structure that has a balanced element (processing speed and accuracy) that is relatively different from the YOLO method.

The SSD runs the CNN on the input image only once and computes the feature map. In order to predict the boundary grid and object classification probability, CNN is performed on this feature map with a size of 3 × 3.

The SSD predicts the boundary grid after CNN processing and can detect objects of various scales. Since the final result is output through convolution layers in consideration of multi-scale features, false positives are generated only for regional information through at least convolution layers, and the generalization error is somewhat higher than YOLO.

In this way, by applying the artificial intelligence and deep learning techniques beyond the existing method in the image, the reliability and recognition rate of the system is high, and as a result of this, along with advances in technology, it is applied to application services for vehicles. This trend is being introduced, and it is used in various systems to recognize the contents of video, natural language, voice, control, and communication.

Furthermore, it is used to analyze complex data and various types of data, and in particular, it is used to increase the accuracy of vehicle detection in various industries such as illegal parking control, bus driving vehicle control, parking control and autonomous driving.

In the present specification, a vehicle determined as a vehicle through an object detection method and a vehicle determined as parking of an abnormal behavior among the parked vehicles is recognized and identified from the result.

In addition, in the object detection step for determining the vehicle according to the present invention, it is determined as an initial object (vehicle) based on CNN, and by determining whether or not the vehicle is a normal parking vehicle within the result, vehicle recognition for abnormal behavior parking is applied. Is done.

Here, in the case of abnormal parking or abnormal parking, the vehicle is detected based on the license plate, which is unique information (characteristic information) for determining the vehicle from the vehicle appearance.

In other words, in the case of a vehicle determined to be a vehicle, it is judged whether the identification of the vehicle-specific information (characteristic information) is inevitable due to the evasion of the license plate according to the object of parking, and due to objects or surroundings (including objects and people). It is also possible to recognize whether or not it is a chafe.

18 shows an example of an overall system configuration in connection with the present invention.

Referring to FIG. 18, first, a step S10 in which the camera 100 receives a streaming image is performed.

Thereafter, the control unit 700 selectively recognizes a vehicle using DSSD, SSD, or the like (S20).

In addition, the controller 700 extracts a single region feature of the vehicle (S30).

At this time, the controller 700 determines whether the license plate extraction of the vehicle was successful or whether the vehicle number recognition in the license plate was successful (S40).

If it is successful in step S40, it is determined as a vehicle (S50), and a subsequent procedure is performed. If it is unsuccessful, the identified vehicle is re-identified (S60).

In step S60, it is possible to utilize the added features in the existing accumulated database (S70).

Through the step S60, a trajectory may be accumulated, and re-identification may be attempted while attempting to change a region to be detected based on the trajectory (S80).

If the object moves, the procedure is performed again from step S40 described above.

Here, the following characteristics may be included in the case of the abnormal parking.

(1) Opening of the trunk, covering the number plate due to objects in the open state of the trunk (box, vinyl, newspaper, box, etc.)

(2) Covering license plates (all, part, specific location, plastic packaging, tea using terrain features), blanking plates (no parking, prohibited construction sites, etc.)

(3) A person artificially roams or obscures the license plate position

(4) In the case of trucks, truck tailgate falls and avoidance using terrain features

(5) Dodge due to size and cover part or all of the entire license plate depending on the vehicle interval

(6) Covering acts such as telephone poles, street lights, and cargo (load)

Input image and vehicle detection

Each step will be described in detail with reference to the technology flow described through FIG. 18.

First, input images and vehicle detection, which are steps S10 and S20, will be described.

In step S10, the input of the image is digitized data acquired from the camera.

At this time, the video is a target object, and each frame (image) is composed of two-dimensional data.

In relation to step S20, the object proposed in the present invention is to identify and judge a vehicle other than a normal parking vehicle, and the definition of the normal parking vehicle is visually identified by the type of the vehicle from the input camera position and field of view. Define vehicles that must be within the possible range and are license plate identifiable.

At this time, the definition of the abnormal parking vehicle refers to a vehicle existing in the screen in a state in which the vehicle license plate is not discriminated visually differently from the definition of the normal parking vehicle.

And objects present in the image, that is, vehicles, people, objects (objects that can damage or obscure part or all of the license plate such as newspapers, boxes, vinyls, tapes), street lights, parking prohibited signs, construction guide signs, surrounding parking The vehicle is different from the direction of the vehicle, and it is defined as an abnormal vehicle.

In the method proposed in the present invention, the object property of the detection target is a vehicle, and in order to determine a normal parking vehicle and an abnormal parking vehicle, the characteristics of people, objects, and even a normal vehicle are analyzed and identified by a continuous parking vehicle. And judge.

In addition, when it is judged and identified as an abnormal parking vehicle, the parking vehicle is defined as an object of interest and reallocated to be continuously monitored, and the vehicle number is identified and recognized when the movement of the vehicle occurs.

Normal parking vehicle detection and recognition

Next, the normal parking vehicle detection and recognition will be described in detail.

An object detection and recognition method of the present invention for detecting a desired target object (vehicle) from input data will be described.

Recently, SSD (Single Shot Detector) and YOLO (You only Look Once) methods are used according to the system environment as object detection and classification methods.

This system applies and proposes the DSSD (Deconvolutional Single Shot Detector) method to improve the problem of small object detection in the SSD (Single Shot Detector) method and to improve the performance of fast processing speed and object detection.

19 shows an example of a network structure of a single shot detector (SSD) in connection with the present invention.

At this time, in the existing method of single shot detector (SSD), a deconvolution layer is added, and an improvement method using a feed forward connection is used to improve processing speed through skip and improve object detection performance.

In order to apply the convolution kernel considering the aspect ratio of the default region (width and height of the region, aspect ratio), the shape of the default region includes a rectangle as well as a square, so the SSD structure Set to perform only 3x3 convolution in the process of obtaining the coordinate and class score of the bounding region detected in.

The reason for this is a structure for compensating for errors in detection of a single shot detector (SSD) 300 by making it similar to the default region type as an effective receptive field for effective detection in each feature map. It is an improvement.

20 shows an example of a network structure of a DSSD (Deconvolution Single Shot Detector) in connection with the present invention.

20, the dictionary data for learning is utilized for learning as VOC Pascal 2007, VOC Pascal 2012, MS COCO, Caltech pedestrian dataset, ImageNet 2012, 2014, MIO-TCD dataset.

In addition, FIG. 21 shows an example of an improved structure in the SSD and DSSD structures in the entire system in relation to the present invention.

Referring to FIG. 21, the improved structure of the present invention is applied to an area partitioned by a dotted block.

Extract feature of detected vehicle

Next, a method of extracting the characteristics of the detected vehicle will be described.

Vehicles existing in the image were detected using the improved method in the structure of SSD (Single Shot Detector), and the features are analyzed and extracted for normal parking and abnormal parking in the area.

Feature extraction calculates the middle coordinates of the lower left-right coordinates according to the aspect ratio size in the detected region. At this time, the intermediate coordinates can be identified with the naked eye differently depending on the assumption that the vehicle license plate exists in the corresponding position coordinates according to the detection result and the assumption that the camera position is in the top view in most cases. Since the license plate position is located in the lower part of the morphological element of the vehicle, the calculated intermediate coordinate point below is used.

The coordinate setting for one point as the intermediate coordinate point in the detection area is calculated from the values of the four coordinates that are the edge positions (coordinates) of the detection area as the coordinates for the two points below.

In addition, the detection area due to the four coordinates was stored as one single object and set to a value that can be reassigned due to the corresponding coordinate. That is, four coordinate value positions are determined in the detection step.

Calculation of the middle coordinates for the bottom two coordinates from the object detection results is as follows, and the coordinates are used as points to analyze the eigenvalues and moving components that identify the object and use them as trajectory information. Is the value.

The calculation of the detected area and the aspect ratio is as shown in Equation 6 below.

Equation 6

은 0.3이고

는 0.95이며,

은 multi-feature maps의 개수를 나타낸다. In Equation 6 above

Is 0.3

Is 0.95,

Indicates the number of multi-feature maps.

At this time, the size of the default box is obtained from each feature map, and the aspect ratio is calculated.

는 검출 영역 개수를 말하며, 이때 종횡비(aspect ratio)가 1일 때 가장 최적의 default box를 말한다.Also, in Equation 6

Is the number of detection areas, and this is the most optimal default box when the aspect ratio is 1.

On the other hand, the object detection feature point for a point in the corresponding default box is as shown in Equation 7.

Equation 7

Equation 7 is an expression expressed as a feature point for a point in the region.

Next, FIG. 22 shows an example of extracting the object trajectory from the detection result in relation to the present invention.

22 (a) shows the result of the detected object area.

In addition, Fig. 22 (b) shows a point obtained from the result.

That is, based on the object trajectory information, the object of the main vehicle is judged as the license plate location information, and it is determined, such as the distance (distance) between a specific object or a neighboring vehicle, and the occlusion (partial occlusion) due to a specific fixed object.

At this time, the determination of the occlusion due to a specific or fixed object is determined from the result of the detection of an object existing in a different region from the already detected object.

Detection of different areas (including different objects in an image or all objects with the same properties) was set to have two meanings.

First, object detection and classification in other areas are detected and recognized from additional learning data.

This detection and classification is the result of the proposed method, DSSD, and the data of the vehicle having an abnormal form is not included in the classification category as a learning model in addition to the unique data of a person, a specific object, a fixed object, or a vehicle. Accordingly, an object for an object other than the detected vehicle is determined.

Second, if the size of the detected object area (boundary box) falls within the size of the standardized vehicle, the value for the extraction of trajectory information of one point is small, or when calculated as one point, the value is small, resulting in a standardized vehicle. It was judged to be inaccurate data to judge.

Equation 8 below is a condition for determining the occlusion between two objects, and is set to recognize the overlapping part of the region of interest in the region between the object A and the object B.

In addition, the condition of the region of interest between object A and object B in Equation 8 is also described below.

Equation 8

On the other hand, Figure 23 shows an example of determining the occlusion of the license plate from the results of different objects in relation to the present invention.

23 (a) shows the result of trajectory extraction from the detected vehicle result related to object A.

In addition, FIG. 23B shows the result of object detection in other areas related to object B.

In addition, FIG. 24 shows an example of determining the occlusion of the vehicle license plate when the data value of the trajectory extraction result is small in relation to the present invention.

24 (a) shows the result of detecting the detected vehicle object (a), and FIG. 24 (b) shows the result of trajectory extraction from the detected result.

As described above, in the present invention, in order to increase the accuracy of the occlusion determination, number recognition is performed in the detected area, and if the recognized result is unrecognized or partially recognized, it is determined as occlusion.

In addition, in the case of a vehicle judged to be occluded (abnormal parking), it is possible to divide the vehicle into a case in which a person around the vehicle or a person who has stopped the vehicle has not dropped off.

That is, in the case of abnormal parking, if a movement occurs from the vehicle, it may be assumed that a person is identified in the vicinity of the vehicle in which the abnormal parking is abnormal, and that there is no person in the vicinity in the step of identifying the abnormal parking.

At this time, in the first case, if it is judged to be a person, number recognition is performed from continuous image data, and in the second case, number recognition is performed due to the motion vector of the abnormal position coordinate (moving trajectory component).

Equation 9 below calculates a moving component vector for a feature point (a point of an abnormal parking vehicle) in a region.

Equation 9

In Equation 9, R is the coordinate components of the detected area, and P represents the coordinates of the feature point. At this time, R is the coordinates of the detected area of the continuous image data.

Equation (9) represents the coordinates of the moving component targeting a point by region coordinates from the object detected from the continuous image data, and is used as a data value that leads to the number recognition step as a criterion for determining whether the coordinates exist in the image. .

는 이전 영상에서 검출된 객체와 현재 영상에서 검출된 객체 사이의 거리 추정값이고

는 이전 영상에서 검출된 특징점의 좌표이고

는 현재 영상에서 검출된 특징점의 좌표를 말한다.Also, in Equation 9

Is an estimate of the distance between the object detected in the previous image and the object detected in the current image.

Is the coordinates of the feature points detected in the previous image

Is the coordinates of the feature points detected in the current image.

25 is a flowchart illustrating a proposed abnormal vehicle detection and classification system module in connection with the present invention.

Referring to FIG. 25, the controller 700 extracts a single region feature of the vehicle (S100).

At this time, the controller 700 allocates a one-dimensional trajectory (S200), and attempts to recognize the object occlusion and the vehicle number in the license plate (S300).

Based on the step S300, it may be determined whether the vehicle is an abnormally parked vehicle (S400), and attempts to re-identify while attempting to accumulate a trajectory and change a region to be detected (S500).

Finally, the vehicle number may be retried to succeed (S600).

Experiment result

The present invention is a method of detecting a vehicle and using the same as a discrimination method for determining the occlusion due to an unspecified object in the center of the vehicle license plate. If the detected vehicle moves continuously, it is determined as the vehicle to be detected first through re-detection and identification and recognition of the license plate.

At this time, in the case of the detected vehicle, the moving component is analyzed and re-detected with the detected vehicle, and the vector of the moving component, that is, the one-dimensional (1D) trajectory information is calculated as a criterion for determining the same vehicle.

26 is a view illustrating the characteristics of experimental images and images in relation to the present invention.

26 (a) includes a vehicle and an obstacle object, FIG. 26 (b) includes a vehicle, an obstacle object, and a person, and FIG. 26 (c) includes a vehicle and a person.

In addition, FIG. 26 (d) includes a vehicle and an obstruction object, FIG. 26 (e) includes a plurality of vehicles and an obstruction object, and FIG. 26 (f) includes only a vehicle.

FIG. 27 is a view showing the experimental results (processing speed) and image characteristics of the input image in FIG. 26.

That is, FIGS. 27 (a) to 27 (f) are used as input images in FIGS. 26 (a) to 26 (f), and the processing results are shown as images.

In FIG. 27, the image on the left is the input image in FIG. 26, and the image on the right is the result of processing.

Referring to FIG. 27 (a), the following results were obtained with respect to the feature and the detection processing speed.

Image size: Full HD / Image format: JPEG / Processing speed: 18.18 fps (1/1000 sec) / Image feature: Vehicle, Object

Referring to (b) of FIG. 27, the following results were obtained with respect to the feature and the detection processing speed.

Image size: Full HD / Image format: JPEG / Processing speed: 18.18 fps (1/1000 sec) / Image feature: Vehicle, Object, Human

Referring to FIG. 27 (c), the following results were obtained with respect to the feature and the detection processing speed.

Image size: Full HD / Image format: JPEG / Processing speed: 14.08 fps (1/1000 sec) / Image feature: Vehicle, Human

Referring to FIG. 27 (d), the following results were obtained with respect to the feature and the detection processing speed.

Image size: Full HD / Image format: JPEG / Processing speed: 26.32 fps (1/1000 sec) / Image feature: Vehicle, Object

Referring to FIG. 27 (e), the following results were obtained with respect to the feature and the detection processing speed.

Image size: Full HD / Image format: JPEG / Processing speed: 43.48 fps (1/1000 sec) / Image feature: Vehicle's, Object

Referring to FIG. 27 (f), the following results were obtained with respect to the feature and the detection processing speed.

Image size: Full HD / Image format: JPEG / Processing speed: 41.67 fps (1/1000 sec) / Image feature: Vehicle

In the case of crackdown on existing illegally parked vehicles, crackdown is avoided by using tools that can avoid crackdown or cover some or all around the license plate depending on the surrounding environment, fixed objects, or human actions.

In addition, a vehicle that can identify a license plate is used as a target for crackdowns, and in a real road environment, it is difficult to operate due to vehicles subject to avoidance.

These problems can be overcome through the methods and systems of the present invention described above.

That is, the proposed system utilizes the DSSD method that effectively solves the problems of SSDs applied to vehicle detection and classification methods and configures the layer layer structure in the existing network structure to improve the accuracy of classification rather than processing speed. It was applied so that only a specific layer structure could be used.

In addition, when the avoidance vehicle of an object to be controlled has a movement or when an object or a mask used in a screening phenomenon is removed, the vehicle of the object to be avoided has been applied to enable control through re-identification and license plate recognition. As a result, a method to classify the vehicles to be covered and avoided and to improve the recognition rate for detection by analyzing moving components was proposed.

Meanwhile, the above-described embodiments of the present invention may be implemented through various means. For example, embodiments of the present invention may be implemented by hardware, firmware, software, or a combination thereof.

For implementation by hardware, the method according to embodiments of the present invention includes one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs) , Field Programmable Gate Arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and the like.

In the case of implementation by firmware or software, the method according to embodiments of the present invention may be implemented in the form of a module, procedure, or function that performs the functions or operations described above. The software code can be stored in a memory unit and driven by a processor. The memory unit is located inside or outside the processor, and can exchange data with the processor by various known means.

The detailed description of preferred embodiments of the present invention disclosed as described above has been provided to enable those skilled in the art to implement and practice the present invention. Although described above with reference to preferred embodiments of the present invention, those skilled in the art will understand that various modifications and changes can be made to the present invention without departing from the scope of the present invention. For example, those skilled in the art can use each of the configurations described in the above-described embodiments in a manner of combining with each other. Accordingly, the present invention is not intended to be limited to the embodiments presented herein, but to give the broadest scope consistent with the principles and novel features disclosed herein.

The present invention may be embodied in other specific forms without departing from the spirit and essential features of the present invention. Accordingly, the above detailed description should not be construed as limiting in all respects and should be considered illustrative. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention. The present invention is not intended to be limited to the embodiments presented herein, but is intended to give the broadest scope consistent with the principles and novel features disclosed herein. In addition, claims that do not have an explicit citation relationship in the claims can be combined to form an embodiment or included as a new claim by amendment after filing.

Claims (11)

A first step of acquiring an image through a camera;
A second step of dividing the entire region of the image into a short-distance region and a long-distance region based on a predetermined reference line in the image;
A third step of dividing the image into a plurality of preset block units;
A fourth step of deriving a feature map applying the plurality of block units using at least one of a plurality of convolutional layers of a convolutional neural network (CNN);
A fifth step of determining whether an object included in the image is at least one vehicle using the feature map;
A sixth step of determining whether a license plate identification of the at least one vehicle and identification of a vehicle number in the license plate are possible when the object is at least one vehicle;
A seventh step of monitoring the first vehicle when there is a first vehicle in which at least one of identification of the license plate and identification of the vehicle number is unavailable among the at least one vehicle; And
Including the eighth step of retrying the identification of the license plate of the first vehicle and the identification of the vehicle number in the license plate when the movement of the first vehicle occurs during the monitoring operation;
When the object is included in the remote area, a feature map for the area including the object is derived using the plurality of convolutional layers,
When the object is included in the short-range region, a feature map for the region including the object is derived by using only a predetermined number of layers among the plurality of convolution layers,

When the object is included in the remote area in the second step,
Between the second step and the third step, the 2-1 step of converting the scale of the remote area; further comprising,

The fifth step,
Step 5-1 of comparing the feature map associated with the object for each of the plurality of block units;
A 5-2 step of determining a max pooing area in which the object exists most frequently among the plurality of block units; And
And a 5-3 step of determining whether the object is a vehicle using the feature value of the vehicle based on the max polling area.

The 5-3 step,
Step 5-4 of setting a region of interest (ROI) based on the max polling region;
Step 5-5 of setting a bounding box based on the set ROI; And
And a step 5-6 of comparing the feature value of the object in the bounding box with the feature value of the vehicle to determine whether the object is the vehicle.

The fifth step is performed by a DSSD (Deconvolutional Single Shot Detector) method,
The DSSD method,
As a single shot detector (SSD) method that additionally uses a deconvolution layer, a feed forward connection is used to improve identification speed.
In the process of obtaining the coordinates and class score of the object region (bounding region) detected through the SSD method, only 3x3 convolution is performed,

The fifth step,
It is performed using coordinates derived according to an aspect ratio size in a region of the at least one vehicle detected through the DSSD method,

Based on the arrangement information of the camera and the position information of the license plate of the at least one vehicle obtained in advance, the coordinates are derived,

Trajectory information is generated using the derived coordinates,
In the sixth step,
Vehicle identification method characterized in that it is determined whether the identification of the license plate of the at least one vehicle and identification of the vehicle number in the license plate is possible using the generated trajectory information.
According to claim 1,
In the seventh step,
The first vehicle is a vehicle identification method characterized in that at least one of identification of the license plate and identification of the vehicle number by at least one of a person and an object other than the at least one vehicle is impossible. delete delete delete delete delete delete delete delete delete

Images