KR20220153721A - system and method of clamping down on vehicles that violate traffic laws - Google Patents

Abstract

The present invention provides a system and method of cracking down on a law violation in an intersection multi-lane. The system comprises: a camera unit installed facing an intersection multi-lane and photographing one or more vehicles driving in the intersection multi-lane; an image signal processing unit for recognizing the sign signal of a traffic light and the license plate of each of one or more vehicles driving and intersecting the intersection multi-lane by receiving the image information photographed through the camera unit and tracking the motion of the vehicle corresponding to the motion of the license plate; a law violation determination unit for determining a law violation based on the sign signal of the traffic light and the movement of the one or more vehicles; and a transceiver for generating and transmitting law violation information about a vehicle determined to be in violation of the law. Accordingly, it is possible to detect and crack down on abnormal driving conditions of multiple vehicles driving in an intersection multi-lane.

Description

System and method of clamping down on vehicles that violate traffic laws}

The present invention relates to vehicle number recognition technology, and more particularly, recognizes vehicle numbers of multiple vehicles operating on the intersection multi-lane from image information obtained by capturing the intersection multi-lane with a single wide-angle camera, and recognizes the vehicle number The present invention relates to a law enforcement system and method for detecting and regulating abnormal driving conditions of multiple vehicles operating on intersection multi-lanes by tracking movement in an area.

As the social structure becomes complex and diversified, the area of individual activity gradually widens, while the use of automobiles increases in response to the demand for rapid movement. In addition, due to the development of logistics, the number of vehicles for delivery has increased rapidly. As the social and economic scale expands and advances, the demand for vehicles and transportation is explosively increasing.

With the rapid increase in the number of vehicles and the generalization of vehicle use, criminal acts using vehicles are also increasing, and the number of vehicles subject to crackdowns, such as speeding vehicles and hit-and-run vehicles, is also increasing.

In order to solve this problem, conventionally, a license plate recognition technology has been developed and commercialized so that a vehicle driving on a road can be photographed and the license plate number of the vehicle can be recognized.

The license plate recognition is widely used for purposes such as managing unmanned parking, detecting illegally parked vehicles, automatically imposing fines for speed violations on highways, detecting illegal license plates of stolen vehicles, and detecting overloaded vehicles.

A commercial system implementing the license plate recognition above employs a single-lane license plate recognition method of a loop sensor method. This has a configuration in which a loop-type detection sensor is provided in a specific section of a specific lane on the road, and a camera operated according to a signal detected by the loop-type detection sensor is installed.

Since such a license plate recognition system requires a loop-type detection sensor to be installed on the road, its installation cost is high and it is likely to be damaged when used for a long period of time, so maintenance is not easy. In addition, since the loop-type detection sensor can be installed only on a specific road section and lane, there is a limit in that there is no choice but to crack down on vehicles in a very limited way.

In addition, in the case of a camera operating in conjunction with the loop-type detection sensor, it is common to use a CCD camera having a high pixel count to recognize a vehicle number, and FIG. 1 (a) illustrates a conventional license plate recognition system. it did The above-described conventional vehicle license plate recognition system has a disadvantage in that installation cost is high, and also has a problem in that a vehicle passing through a lane in which a loop-type detection sensor is not installed cannot be photographed.

Furthermore, in the case of crackdown on only a specific road section or lane, it is difficult to clearly detect the abnormal driving condition of the vehicle. In other words, there was a problem that it could not detect illegal U-turns, lane violations, and parking/stopping.

Moreover, there was also a problem of difficulty in recognizing license plates for new vehicle license plates that have recently been used, such as blue electric and hydrogen vehicle license plates with a low contrast ratio. (b) of FIG. 1 illustrates a license plate assigned to an electric vehicle. Since the license plate assigned to the electric vehicle employs a reflective license plate, there is a problem in that it is difficult to recognize the license plate by reflecting light as shown in FIG. 1(c).

Accordingly, in the prior art, it has been urgently desired to develop a technology capable of detecting abnormal driving conditions of multiple vehicles operating on intersecting multi-lanes and recognizing license plate numbers of multiple vehicles.

Republic of Korea Patent Registration No. 10-0852683 Korean Patent Publication No. 10-2016-0040036 Republic of Korea Patent Registration No. 10-0703956

The present invention recognizes license plates of multiple vehicles operating on the intersection multi-lane from image information captured by a single wide-angle camera, and tracks the movement of the intersection multi-lane to drive the intersection multi-lane The purpose of the present invention is to provide a system for regulating violations of laws and regulations in multi-lane intersections that detects and regulates abnormal driving conditions of a large number of vehicles and a method therefor.

In order to achieve the above object, a law violation enforcement system in intersection multi-lanes according to the present invention includes a camera unit installed opposite to intersections and capturing images of one or more vehicles traveling through intersections; An image for receiving image information captured through the camera unit, recognizing the license plate of each of one or more vehicles driving by crossing the sign signal of a traffic light and intersection multi-lane, and tracking the movement of the vehicle corresponding to the movement of the license plate signal processing unit; a law violation determination unit for determining a law violation based on the display signal of the traffic light and the movement of the one or more vehicles; and a transmission/reception unit generating and transmitting law violation information on the vehicle determined to be in violation of the law to the outside.

The present invention recognizes license plates of multiple vehicles operating on the intersection multi-lane from image information captured by a single wide-angle camera, and tracks the movement of the intersection multi-lane to drive the intersection multi-lane It provides the effect of detecting and regulating abnormal driving conditions of a number of vehicles.

1 is a diagram illustrating a conventional vehicle number recognition process;
2 is a block diagram of a system for enforcing violations of laws in multi-lane intersections according to a preferred embodiment of the present invention;
3 is a configuration diagram of the video signal processing unit of FIG. 2;
4 is a configuration diagram of the image and vehicle information storage unit of FIG. 2;
5 to 7 are procedures of a law violation enforcement method in intersection multi-lane according to a preferred embodiment of the present invention;
8 is a diagram illustrating an ROI of image information of an intersection multi-lane according to a preferred embodiment of the present invention;
9 and 10 are diagrams illustrating a vehicle number recognition process according to a preferred embodiment of the present invention.

The present invention recognizes license plates of multiple vehicles operating on the intersection multi-lane from image information captured by a single wide-angle camera, and tracks the movement of the intersection multi-lane to drive the intersection multi-lane It is possible to detect and crack down on abnormal driving conditions of a large number of vehicles.

In addition, the present invention improves vehicle number recognition performance using deep learning, enabling vehicle numbers for electric or hydrogen vehicles as well as vehicle numbers for multiple vehicles operating on intersecting multi-lane to be recognized, thereby enabling vehicle number recognition rate can be increased.

A law violation enforcement system and a method therefor in a crossroads according to a preferred embodiment of the present invention described above will be described in detail with reference to the drawings.

<Configuration of the system for cracking down on law violations at intersections>

Figure 2 shows the configuration of a law violation enforcement system in intersection multi-lane according to a preferred embodiment of the present invention.

The law violation enforcement system 100 in the intersection multi-lane includes a camera unit 102, a video signal processing unit 104, a law violation determination unit 106, an image and vehicle information storage unit 108, and a transmission and reception unit 110 consists of

The camera unit 102 is a high-definition, wide-angle camera, installed facing the multi-lane intersection, capturing an image of a partial area of the multi-lane intersection, and providing image information based on the image to the video signal processing unit 104.

The video signal processing unit 104 receives the video signal and generates traffic light information output from the traffic light and vehicle tracking information that tracks the movement of a plurality of vehicles located on the multi-lane intersection to determine the violation of the law (106). provided by

The law violation determination unit 106 detects whether there is a vehicle violating the traffic law based on the traffic light information and vehicle tracking information, and also detects the law violation information consisting of video information and vehicle information about the vehicle violating the traffic law. It is generated and provided to the image and vehicle information storage unit 108 and the transceiver 110.

The video and vehicle information storage unit 108 receives and stores video information from the video signal processing unit 104, traffic light information and vehicle tracking information, and enforcement information provided by the law violation determination unit 106.

The transmission/reception unit 110 transmits the law violation information from the law violation determination unit 106 and stored image information, traffic light information, and vehicle tracking information corresponding to the law violation information from the image and vehicle information storage unit 108. It is read and transmitted to the police station server or police center server through the network.

<Detailed configuration of the video signal processing unit 104>

3 is a detailed configuration diagram of the video signal processing unit 104. Referring to FIG. 3, the image signal processing unit 104 includes an image distribution processing unit 200, a traffic light recognition unit 202, a license plate recognition unit 204, a vehicle tracking unit 206, an image combining unit 208, It consists of a data store (210).

The image distribution processing unit 200 provides the image information provided from the camera unit 102 to the traffic light recognition unit 202, license plate recognition unit 204, vehicle tracking unit 206, and image combining unit 208. do.

The traffic light recognizing unit 202 recognizes traffic lights and sign signals from the image information and provides the resulting sign signal information to the image combiner 208.

The license plate recognition unit 204 detects a license plate area for each of a plurality of vehicles intersecting in the image information, recognizes the license plate number indicated in the corresponding license plate area, and converts the corresponding license plate recognition information to the image. It is provided as a coupling part 208. In particular, the license plate recognition unit 204 improves vehicle number recognition performance using deep learning, and can recognize vehicle numbers for electric vehicles or hydrogen vehicles as well as vehicle numbers for multiple vehicles operating on intersections. By doing so, the vehicle number recognition rate is increased.

The data storage 210 stores learning data for license plate recognition.

The vehicle tracking unit 206 estimates the position or tracks the motion of the license plate area detected by the license plate recognition unit 204, and transmits the location estimation information or motion tracking information to the image combining unit 208. provided by

The image combiner 208 combines the sign signal information, license plate recognition information, location estimation information, or motion tracking information with image information and provides it to the law violation determination unit 106.

<Detailed configuration of image and vehicle information storage unit 208>

4 illustrates data stored in the image and vehicle information storage unit 208 . Referring to FIG. 4 , the image and vehicle information storage unit 208 stores location information, license plate recognition information, location estimation information or motion tracking information of a vehicle with license plate recognition, and recorded image information. The location information is location information where the law violation enforcement system 100 is installed in the intersection multi-lane, and can be used as identification information of the law violation enforcement system 100 in the intersection multi-lane. The license plate recognition information is result information obtained by recognizing a license plate area and a license plate number from image information. The location estimation information or motion tracking information is information for estimating the location of the vehicle from which the license plate area is recognized, or information for tracking motion. The recorded image information is information in which image information from the camera unit 102 is stored.

Now, a procedure of a law violation enforcement method in intersection multi-lane applicable to the above-described law violation enforcement system in intersection multi-lane will be described in detail with reference to the drawings.

<Procedures for cracking down on law violations at cross-traffic multi-lanes - overall flow>

Figure 5 is a diagram showing the overall procedure of the law violation enforcement method according to a preferred embodiment of the present invention.

Referring to FIG. 5 , the system 100 for enforcing violation of the law in the intersection multi-lane captures an image of the intersection multi-lane through the camera unit 102 and acquires image information accordingly (step 300). Thereafter, the law violation enforcement system 100 detects lane information from the image information of the intersection multi-lane (step 302). Further, the law violation enforcement system 100 detects signal signal information of a traffic light from the image information of the intersection multi-lane (Step 304). Further, the law violation enforcement system 100 recognizes the license plate and license plate number from the intersection multi-lane image information (step 306).

Thereafter, the law violation enforcement system 100 tracks the movement of each vehicle based on the position change of the recognized license plates, and determines whether the law is violated by matching the movement tracking information and sign signal information accordingly (308, 310). step).

The law violation enforcement system 100 checks whether there is a law violation vehicle determined based on the motion tracking information and traffic light information for each of the license plates (step 312), and if there is a law violation vehicle, generates law violation information After storing in the image and vehicle information storage unit 108, it is transmitted to the National Police Agency server or police sensor server (step 314). The law violation information may be composed of vehicle number and traffic light signal information recognized from the license plate, location information, time information, image information during the motion tracking period, and the like.

<Procedures on how to crack down on law violations at multi-lane intersections - Cracking down on crosswalk trespassing or stop line violations>

6 is a diagram illustrating a procedure for cracking down on crosswalk trespassing or stopping line violations among law violation crackdowns according to a preferred embodiment of the present invention.

Referring to FIG. 6 , the law violation enforcement system 100 in the multi-lane intersection captures an image of the multi-lane intersection through the camera unit 102 and acquires image information accordingly (step 400). Thereafter, the law violation enforcement system 100 detects a first region of interest (ROI) area, that is, a crosswalk area, from the image information of the intersection multi-lane (step 402). The first ROI area may be set as a crosswalk area, and FIG. 8(a) illustrates this.

In addition, the law violation enforcement system 100 detects signal signal information of a traffic light from the image information of the intersection multi-lane (step 404). Further, the law violation enforcement system 100 recognizes the vehicle license plate and license plate number from the intersection multi-lane image information (step 406).

Thereafter, the law violation enforcement system 100 estimates the position where each of the recognized license plates is located on the intersection multi-lane (step 408).

The law violation enforcement system 100 matches the location information of each of the recognized license plates with traffic light signal information to determine whether the law has been violated, that is, whether the crosswalk has been entered at the time of a red light (step 410). ).

The law violation enforcement system 100 determines whether the law is violated by matching the location information of each of the recognized license plates with the traffic light signal information, and checks whether there is a law-violating vehicle (step 412), If there is a violating vehicle, law violation information is generated and stored in the image and vehicle information storage unit 108, and then transmitted to the National Police Agency server or police sensor server (step 414). The law violation information may be composed of vehicle number and traffic light signal information recognized from license plate, location information, time information, image information, and the like.

<Procedures on how to crack down on law violations at cross-traffic multi-lanes - Illegal U-turn or illegal parking enforcement>

7 is a diagram illustrating a procedure for cracking down on illegal U-turn or illegal parking among crackdowns on violation of laws according to a preferred embodiment of the present invention.

Referring to FIG. 7 , the law violation enforcement system 100 in the intersection multi-lane captures an image of the intersection multi-lane through the camera unit 102 and acquires image information accordingly (step 500). Thereafter, the law violation enforcement system 100 detects the second and third ROI areas, that is, the left road area and the right road area, from the image information of the intersection multi-lane (step 502). The second ROI area may be set as a left road area, and the third ROI area may be set as a right road area, which is exemplified in (b) of FIG.

In addition, the law violation enforcement system 100 detects signal signal information of a traffic light from the image information of the intersection multi-lane (step 504). In addition, the law violation enforcement system 100 recognizes the license plate and license plate number from the intersection multi-lane image information (step 506).

Thereafter, the law violation enforcement system 100 tracks the change in position of each of the recognized license plates (step 510).

The law violation enforcement system 100 matches the location tracking information for each of the recognized license plates with traffic light signal information to determine whether the law is violated, that is, illegal U-turn, illegal left turn, illegal lane change, illegal parking and stopping, etc. (510 steps).

The law violation enforcement system 100 determines whether the law is violated by matching the location tracking information for each of the recognized license plates with the traffic light sign signal information, and checks whether there is a law violating vehicle (step 512), If there is a law-violating vehicle, law-violating information is generated, stored in the image and vehicle information storage unit 108, and then transmitted to the National Police Agency server or police sensor server (step 514). The law violation information may be composed of vehicle number and traffic light signal information recognized from the license plate, location information, time information, image information during the movement tracking period, and the like.

<Vehicle number recognition process>

As described above, the present invention detects a license plate, recognizes the license plate number, generates vehicle identification information, estimates the location of the vehicle, and tracks the movement of the vehicle.

Therefore, the present invention improves vehicle number recognition performance using deep learning, enabling vehicle number recognition of electric vehicles or hydrogen vehicles as well as vehicle numbers of multiple vehicles operating on intersecting multi-lane roads, thereby increasing the vehicle number recognition rate. raise

The above vehicle and plate number recognition process will be described in more detail. When recognizing the vehicle and vehicle numbers, object detection and recognition are performed.

For the object detection and recognition described above, the present invention may use a method of predicting a boundary including the position and size of the object.

In addition, according to the sliding window method, the sliding ROI image may be passed to the object class prediction function, and objects of various sizes may be generated and detected as pyramids.

Then, according to the object recognition method using HOG (Histogram of Oriented Gradients) FEATURES and SVM (support vector machine), pyramid sliding windows are executed to calculate HOG FEATUREs for object classification in each window, and then recognize objects using SVM You may.

In addition, deep learning techniques can be applied, CNN (convolutional neural network) can extract objects with high accuracy, R-CNN (REGIONS WITH CNN) can reduce the number of ROIs required for object classification, texture, Use local information such as intensity and color. FAST RCNN is calculated based on SPP-NET (spatial pyramid pooling) and computes CNN once for all images.

In addition, YOLO (YOU ONLY LOOK ONCE) can be applied. YOLO reflects the accuracy of the S*S grid for each image and predicts N entities in a lead, most of which have low reliability. In order to increase reliability, neighboring grids can be merged, and unnecessary parts can be removed by setting a threshold.

The YOLO is fast compared to other real-time vision technologies and enables class discrimination by viewing the entire image at once.

The procedure according to the vehicle number recognition method according to the above preferred embodiment of the present invention will be described in detail with reference to the drawings.

9 illustrates a procedure of a vehicle number recognition method according to a preferred embodiment of the present invention.

The law violation enforcement system 100 in the multi-lane crossing checks whether one operation mode of learning or license plate recognition is set. If it is set to any one of the operation mode of learning or license plate recognition, the law violation enforcement system 100 in the intersection multi-lane checks whether the operation mode is a learning mode (step 600).

If the operation mode is a learning mode, the law violation enforcement system 100 in the intersection multi-lane implements deep learning for object detection and recognition rate (step 602). The above deep learning is for vehicle detection and plate number recognition, and any one or more of CNN (convolutional neural network), R-CNN (REGIONS WITH CNN), and FAST RCNN may be selected. The above vehicle detection and license plate recognition may employ a sliding window method or an object recognition method using Histogram of Oriented Gradients (HOG) FEATURES and a support vector machine (SVM). In addition, the deep learning described above is based on deep learning data, which is metadata for learning and testing generated for vehicle detection and plate number recognition, CNN (convolutional neural network), R-CNN (REGIONS WITH CNN) and FAST RCNN It is performed by implementing one or more of the following.

If the operation mode is the license plate recognition mode, the law violation enforcement system 100 in the intersecting multi-lane recognizes the front and rear license plates of the vehicles traveling on the multi-lane according to the deep-learned vehicle detection and license plate recognition method (step 604).

<Detailed procedure of vehicle number recognition step>

The license plate recognition step according to the above preferred embodiment of the present invention will be described in more detail with reference to FIG. 10 .

When the operation mode for license plate recognition is set, the system 100 for enforcing violation of the law on the intersection multi-lane drives the wide-angle camera unit 102 to capture images of vehicles traveling on the intersection multi-lane (step 700). The system 100 for enforcing violations of the law on the multi-lane intersection improves the image information obtained by the imaging (step 702). As the image information enhancement method, single scale retinex (SSR), multi scale retinex (MSR), and BILATERAL FILTER are employed. Through this image information improvement, it is possible to recognize the vehicle license plate even from low-quality image information, and it is also possible to recognize the license plate of the electric vehicle formed on the reflector.

After the image information is improved, the law violation enforcement system 100 in the intersection detects the vehicle from the image information (step 704). When a vehicle is detected from the above image information, the law violation enforcement system 100 in the multi-lane intersection recognizes the vehicle number of the license plate for the vehicle detection area (step 708).

When the vehicle number of the license plate is recognized from the above image information, the law violation enforcement system 100 in the intersection multi-lane stores the image information and license plate recognition information if the vehicle is detected and the license plate number is recognized ( Step 712).

Unlike this, if the vehicle is detected but the license plate number is not recognized, the image processing device 100 returns to step 700 to re-recognize the image information and license plate number (step 714).

In the above description, the present invention has been shown and described in relation to specific embodiments, but it is common knowledge in the art that various modifications and changes are possible without departing from the spirit and scope of the invention indicated by the claims. Anyone who has it will be able to easily understand.

100: Regulations violation enforcement system in intersection multi-lane
102: camera unit
104: video signal processing unit
106: Law Violation Judgment Department
108: image and vehicle information storage unit
110: transceiver

Claims (8)

In the law violation enforcement system in intersection multi-lane,
a camera unit that is installed facing the intersection multi-lane and captures an image of one or more vehicles traveling on the intersection multi-lane;
An image for receiving image information captured through the camera unit, recognizing the license plate of each of one or more vehicles driving by crossing the sign signal of a traffic light and intersection multi-lane, and tracking the movement of the vehicle corresponding to the movement of the license plate signal processing unit;
a law violation determination unit for determining a law violation based on the display signal of the traffic light and the movement of the one or more vehicles;
a transceiver for generating and transmitting law violation information about the vehicle determined to be in violation of the law; and
An image and vehicle information storage unit for storing the video information, the recognition result of the license plate number and the sign signal, and the motion tracking information of the vehicle. According to claim 1,
The video signal processing unit,
Estimating the location of the license plate,
The law violation determination department,
A system for cracking down on laws and regulations at intersections, characterized in that it determines violations of laws based on the display signal of the traffic light and the location of the license plate. According to claim 1,
The video signal processing unit,
Deep learning data storage for storing deep learning data for deep learning of vehicle detection and license plate recognition; further comprising,
The video signal processing unit,
Performs deep learning for vehicle detection and license plate recognition according to the deep learning data, detects the vehicle by receiving the image information, and recognizes the license plate number of the detected vehicle,
The deep learning is one of CNN (convolutional neural network), R-CNN (REGIONS WITH CNN) and FAST RCNN based on deep learning data, which is meta data for learning and testing generated for vehicle detection and license plate number recognition. A system for cracking down on laws and regulations in multi-lane intersections characterized by fulfilling the above. According to claim 3,
The video signal processing unit,
Receiving the image information, improving the image information by implementing one or more of single scale retinex (SSR) and multi scale retinex (MSR);
A law violation enforcement system in multi-lane intersections, characterized in that for performing anisotropic filtering (BILATERAL FILTER) on the improved image information. In the method of cracking down on law violations in cross-traffic lanes,
A camera unit installed opposite to the intersection multi-lane is provided with image information obtained by capturing images of one or more vehicles traveling through the intersection multi-lane, and the sign signal of the traffic light and one or more vehicles traveling crossing the intersection multi-lane Each vehicle Recognizing the license plate and tracking the movement of the vehicle corresponding to the movement of the license plate;
determining a law violation based on the display signal of the traffic light and the movement of the one or more vehicles;
Generating law violation information about the vehicle determined to be in violation of the law and transmitting it to the outside; and
The video information, the recognition result of the license plate number and the sign signal, and the step of storing the motion tracking information of the vehicle. According to claim 5,
estimating the location of the license plate from the image information; and
The law violation enforcement method in intersection multi-lane, characterized in that it further comprises; determining the violation of the law based on the display signal of the traffic light and the position of the license plate. According to claim 5,
Deep learning data storage for storing deep learning data for deep learning of vehicle detection and license plate recognition; further comprising,
When recognizing the vehicle number,
Performs deep learning for vehicle detection and license plate recognition according to the deep learning data, detects the vehicle by receiving the image information, and recognizes the license plate number of the detected vehicle,
The deep learning is one of CNN (convolutional neural network), R-CNN (REGIONS WITH CNN) and FAST RCNN based on deep learning data, which is meta data for learning and testing generated for vehicle detection and license plate number recognition. A method for cracking down on violations of laws in multi-lane intersections, characterized by fulfilling the above. According to claim 7,
Improving the image information by performing at least one of single scale retinex (SSR) and multi scale retinex (MSR) on the acquired image information;
The law violation enforcement method in intersection multi-lane, characterized in that it further comprises the step of implementing anisotropic filtering (BILATERAL FILTER) on the improved image information.

Images