KR102418823B1 - Deep learning-based illegal parking enforcement system using wide-area detection images - Google Patents

Abstract

The present invention relates to a system for quickly and efficiently detecting a vehicle parked or stopped in an illegal parking zone by using a wide area detection image obtained by photographing a wide range of an enforcement zone in cracking down on vehicles entering an illegal parking zone. According to the present invention, a vehicle area is detected by applying a deep learning detection model built for an image of the vehicle area and a license plate area included in the wide area detection image taken by an imaging unit. Also, a vehicle number is immediately recognized or a target expansion ratio of the vehicle area is determined according to a size of the detected vehicle area. When the target expansion ratio is below a certain size, super-resolution image restoration is applied to the vehicle area image to recognize the vehicle number. When the target expansion ratio exceeds a certain size and vehicle number recognition is impossible, it is determined whether the vehicle is a vehicle that has been cracked down. In the case of a cracked vehicle, detection operation for the vehicle area is stopped, and in the case of a new vehicle, a new enforcement operation is notified such that it is possible to effectively detect a vehicle in violation. In addition, a deep learning-based illegal parking enforcement system using a wide area image of the present invention comprises: a field part (100); a control room part (300); and a wired and wireless network (200) connecting the field part and the control room part through a wired and wireless communication means.

Description

Deep learning-based illegal parking enforcement system using wide-area detection images}

The present invention relates to a system for quickly and efficiently detecting violating vehicles parked or stopped in an illegal parking area by using a wide-area detection image taken of the enforcement area in cracking down on a vehicle entering an illegal parking and stopping area.

The present invention detects the vehicle area by applying a deep learning method to the images of the vehicle area and the license plate area included in the wide area detection image taken by the imaging unit, and if the size of the detected vehicle area is greater than a certain size, the vehicle immediately The number is recognized, and if the size of the detected vehicle area is greater than or equal to a certain size, a target enlargement ratio of the corresponding vehicle area is determined. If it is determined that the target magnification ratio is less than a certain size, super-resolution image restoration is applied to the image of the vehicle area to enlarge the image, and then the vehicle number is recognized. If it is determined that this is impossible, the vehicle number recognition operation is stopped and it is determined whether the vehicle is a previously intercepted vehicle. When it is determined that the corresponding vehicle area is for a pre-enforcement vehicle, the detection operation for the vehicle area is stopped, and when it is determined that the corresponding vehicle area is for a new vehicle, the system performs a new enforcement operation. It relates to an illegal parking enforcement system that can quickly and efficiently detect

Even before the present application, various devices or methods for cracking down on illegally parked and stopped vehicles have been proposed.

For example, as one of the prior art, Korean Patent Registration No. 10-1769790 (2017.08.14.) is a prior art titled 'Illegal Parking and Stopping Enforcement Device and Method', which detects a vehicle and quickly acquires a vehicle number. Accordingly, the present invention is to provide an illegal parking and stopping enforcement device and method capable of performing illegal parking and stopping enforcement. The prior art includes a detection unit for receiving a first captured image from a photographing device to detect the vehicle, at least one learning area, and an information management unit for managing a PTZ value of the photographing device corresponding to the learning area; A control unit for controlling the photographing device by using a PTZ value of a learning area overlapping the vehicle area for the detected vehicle among the areas, and a recognition unit for recognizing a vehicle number by receiving a second photographed image from the controlled photographing apparatus By using the previously learned PTZ information to accurately move the rotating camera to the vehicle and recognizing the license plate number, the detection position It is to quickly recognize the license plate number for the vehicle. In particular, this prior art relates to an invention that can minimize the movement error and trial and error of a rotary camera that occurs by repeatedly moving to find the detection point after detection by obtaining the optimal PTZ value by using the learning area information. it's about This prior art uses a photographing device including two cameras, such as a first camera for photographing the first photographed image, and a second camera for photographing the second photographed image closer than the first photographed image to perform the above operation. be able to perform

Another prior art, Korean Registered Patent No. 10-1888381 (Aug. 8. 2018), is a prior art named 'detection area deep learning-based vehicle control system' applied by the applicant of the present invention, and It relates to a deep learning-based vehicle enforcement system that detects the presence or absence of a stopped vehicle through deep learning and controls the vehicle by PTZ control of an enforcement camera to the detection area by a lookup table for PTZ location.

This prior art is a scan method or background method problems of the prior art, such as an increase in the interception time required, an increase in the failure rate of the camera due to excessive movement, an increase in the incidence of false detection of the camera, or the amount of computation when mapping the reference point of a pixel in the indexed image to the PTZ position value. This is to solve problems such as increasing.

In order to solve the problems of the prior art as described above, this prior art does not recognize the vehicle number of the license plate when detecting a vehicle, but first detects the presence or absence of a vehicle, and controls corresponding to the portion corresponding to the detection area in the detection image Provides a vehicle enforcement system that detects a vehicle by generating an image, and also generates a PTZ position value of the enforcement camera corresponding to the detection area during vehicle enforcement existing in the detection area and stores it in a lookup table, referring to the lookup table By performing parking and stopping enforcement by the PTZ control of the enforcement camera for the detection area, the vehicle in the detection area can be detected in real time, and by detecting the presence of a vehicle in the detection area, the amount of deep learning calculation and detection required for vehicle detection Time can be shortened, and by mapping the index coordinate value where the intermittent image is located on the index image and the PTZ position value of the PTZ-controlled intermittent camera with each other, the mapping setting between the detection coordinate value and the PTZ position value is simple, and a lookup table The time it takes to configure can be reduced, and by PTZ control of the enforcement camera with the detection area by referring to the lookup table to perform parking and stopping enforcement, the mechanical movement of the enforcement camera is reduced, thereby reducing the failure rate and false detection rate of the enforcement camera. to make it happen

Another prior art, Korean Registered Patent No. 10-2073971 (2020.01.30.), is also a prior art called 'deep learning-based vehicle control system and method' as a prior art filed by the applicant of the present invention. This prior art builds a vehicle detection model and a vehicle magnification model together by deep learning based on a pre-prepared learning image, and detects or identifies a vehicle from an inspection image taken at a crackdown site using the vehicle detection model and the enlarged model. , relates to a deep learning-based vehicle enforcement system and method for updating a vehicle detection model and a vehicle magnification model by using an inspection image labeled with a vehicle candidate area.

This prior art relates to a vehicle enforcement system that cracks down on illegally parked and parked vehicles using real-time images of a photographing device. It is a technology related to a deep learning-based vehicle enforcement system and method to crack down. This prior art builds a vehicle detection model by deep learning a learning image in which a vehicle part region including a license plate and only a front or rear portion of a vehicle is labeled, thereby constructing a vehicle detection model and time required for vehicle detection can be reduced, and the enforcement efficiency can be improved by detecting only the candidate vehicle area in which the license plate can be identified, and if the number of pixels in the candidate vehicle area is smaller than the set number, the vehicle candidate area is displayed using the vehicle enlargement model as a super-resolution image By converting to , it is possible to secure the number of pixels capable of recognizing a vehicle number, and by securing the number of pixels capable of recognizing a vehicle number, it is possible to widen the area that can be controlled on one screen, It can provide enforcement, and includes contents such as reducing misrecognition of vehicle number by obtaining the effect of improving image quality.

Domestic Registered Patent No. 10-1769790 (2017.08.14.) Domestic Registered Patent No. 10-1888381 (2018.08.08.) Domestic Registered Patent No. 10-2073971 (2020.01.30.)

The present invention is to provide a technology capable of quickly and efficiently detecting a vehicle parked and stopped in an illegal parking area, and in particular, to quickly and efficiently detect a violating vehicle using a wide-area detection image.

The present invention recognizes the vehicle number immediately when the size of the vehicle area detected from the wide-area detection image is greater than or equal to a certain size, and when the target magnification ratio is less than or equal to a certain size, super-resolution image restoration is applied to the vehicle area. By recognizing the license plate number after enlargement, it is intended to accurately recognize the number of the offending vehicle within a wide range of enforcement zones.

An object of the present invention is to detect a violating vehicle quickly and efficiently by using a wide-area detection image obtained by photographing a wide range of enforcement zones among illegal parking and stopping zones (enforcement zones) through a rotating camera provided in an imaging unit.

The present invention uses a wide area index image, but overlaps the upper part of the wide area index image to detect an object that appears very small in the index image, and applies a deep learning detection model to the object included in the divided area, This is to make it possible to easily detect whether an infringing vehicle exists even for an object that appears very small in the index image.

The present invention divides the selected range into the vehicle area by applying a deep learning detection model from the rear or front area of the vehicle containing the license plate of each vehicle included in the wide area detection image, and uses the image of the area to To increase the efficiency of vehicle detection by recognizing the number of the vehicle or detecting whether the vehicle is a vehicle under control, a vehicle for which enforcement has been completed, or a new vehicle subject to enforcement, and by operating the system according to the situation of each vehicle it is to do

In particular, it is recommended not to perform overlapping enforcement actions on previously intercepted vehicles, and in the case of a vehicle for which it is impossible to detect a vehicle number, the vehicle number is not detected. Calculate the structural similarity of the corresponding vehicle image, determine whether the matching target vehicle is the same vehicle according to the calculated similarity value, and perform the existing enforcement procedure And, when it is determined that the matching target vehicle is a different vehicle, it is determined that the vehicle is a new enforcement target vehicle, and the system starts a new enforcement operation to effectively perform the enforcement operation according to the situation of the vehicle.

The 'deep learning-based illegal parking enforcement system using a wide area image' of the present invention is a field unit 100 installed at the enforcement site, a situation room unit 300 installed at a remote location from the field unit, and a wired/wireless communication means between the field unit and the situation room unit. It is configured to include a wired/wireless network 200 that is connected through the .

The field unit 100 of the present invention is configured to include an imaging unit 110 that is installed at the enforcement site to photograph a vehicle, and a communication unit 130 that communicates with the situation room unit through a wired/wireless network. The imaging unit 110 is a means for acquiring an image of a vehicle subject to enforcement, and is composed of a rotating CCTV camera or the like. The imaging unit 110 moves a wide-area detection image for detecting a violating vehicle within the enforcement zone, and a rotary camera to the location of the detected vehicle area to take a regulating image and situational photo for the violating vehicle, and the management server 310 will be sent to

The management server 310, which is a central component of the situation room 300, includes a camera remote control unit 311, a vehicle detection unit 312, a vehicle enforcement unit 313, a control unit 314, and an enforcement information storage unit 315. and may be configured to further include a display unit 316 and a learning image storage unit 317 .

In the present invention, the operation for vehicle detection is mainly performed by the vehicle detection unit 312 . The vehicle detection unit determines whether the image of the object included in the wide-area detection image captured by the imaging unit 110 and transmitted through the wired/wireless network 200 is for the vehicle, and at the time of determination, the learning image storage unit 317 It is determined whether the object in the wide-area detection image is for the vehicle by using the vehicle area detection model built by deep learning the stored learning image.

The present invention detects whether the object included in the wide-area detection image is for a vehicle, and when the object is determined to be a vehicle, recognizes the vehicle number from the image of the vehicle area including the license plate of the vehicle.

The present invention divides the upper part of the wide area detection image by overlapping when the object is not detected as an offending vehicle because the size of the object displayed in the wide area detection image is very small during the operation of detecting the vehicle area (S120 No), and the By applying a vehicle area detection model built by deep learning to an object included in the segmented image, it is possible to easily detect whether a vehicle violates even an object that appears very small in a wide area detection image.

As a result of performing the above detection operation, when it is detected that the object photographed in the wide-area detection image is an offending vehicle, the vehicle detection unit 312 performs an operation for recognizing the number of the vehicle.

In the operation for the vehicle number recognition performed by the vehicle detection unit 312 of the present invention, different steps are performed depending on whether the size of the vehicle area included in the wide-area detection image is greater than or less than a certain size.

If the size of the vehicle area shown in the wide-area detection image is over a certain size, it is determined whether the vehicle number recognition is possible without enlarging the image of the vehicle area (S150), and if the vehicle number recognition is possible (S150 Yes) Using the recognized number, it is determined whether or not the vehicle is a previously-controlled vehicle (S180), and if the vehicle number matches the previously-controlled vehicle (S180 Yes), a new enforcement procedure for the vehicle is not performed, and the rotation type The camera is moved to another vehicle or another control zone to perform a detection operation for the vehicle or control zone (S100). The specific intermittent operation for the air speed vehicle is to be performed separately from the contents of the present invention.

On the other hand, if it is determined that the vehicle number recognition is not possible despite the size of the vehicle area included in the wide area detection image being over a certain size (S150 No), the vehicle number recognition is not performed, and instead the vehicle area (new vehicle) Area) calculates the structural similarity of the image and the corresponding vehicle area (existing vehicle area) image included in the wide-area detection image taken immediately before (S160), and according to the calculated image similarity value, the two (existing and new) vehicle areas are separated It is determined whether it is for the same vehicle (S170). When the similarity of the calculated image exceeds a set threshold (S170 No), it is determined that the two vehicle areas relate to the same vehicle, and when the similarity is less than or equal to the threshold (S170 Yes), it is determined that the two vehicle areas relate to different vehicles do.

As a result of the previous determination, if the similarity of the image exceeds the threshold and it is determined that the existing vehicle area and the new vehicle area relate to the same vehicle (S170 No), the vehicle including the new vehicle area is determined as an air raid vehicle, The model of the vehicle area is updated with the model of the new vehicle area (S175).

The 'vehicle area model' used in the present invention is a concept including the number of the enforcement zone in which the vehicle is located, the vehicle number, the location value of the vehicle, the image of the vehicle area, and the like.

When the vehicle detection unit 312 determines that the existing vehicle area and the new vehicle area are different from each other (Yes in S170), the vehicle is added to the new vehicle area model (S190), and the vehicle enforcement unit 313 is An initial enforcement operation is performed on the vehicle (S200).

The present invention can detect a violating vehicle quickly and efficiently by using a wide-area detection image taken for a range in which the enforcement zone is divided wider than in the prior art in detecting a vehicle entering an illegal parking and stopping zone. The wide-area detection image applied to the present invention is to take a wide range of enforcement zones compared to the prior art, and it is possible to effectively detect the offending vehicle using the wide-area detection images photographed in the wide enforcement zone.

The present invention recognizes the vehicle number directly according to the size of the vehicle area detected in the wide area detection image, or by applying the super-resolution image restoration to the image of the vehicle area to recognize the vehicle number and then to recognize the vehicle number. Violation vehicles located inside can be recognized quickly.

The present invention detects a vehicle using a wide-area detection image obtained by photographing a relatively wide range of enforcement zones through a rotary camera provided in an imaging unit, thereby reducing the number of operations of a rotary camera for vehicle detection to detect violation vehicles can be detected quickly.

The present invention overlaps the upper part of the image to detect an object that appears small in the upper part of the wide area detection image, and applies a deep learning vehicle area detection model to the object included in the divided area, during the wide area detection image It is possible to easily detect whether a vehicle violates even a small object.

The present invention divides the range set by the system among the rear or front areas of the vehicle including the number of each vehicle among the wide-area detection images into the vehicle area, and uses the image of the vehicle area to recognize the vehicle number or the vehicle By performing a detection operation according to the vehicle situation, such as detecting whether the vehicle is under the control, whether the vehicle is a vehicle for which the control has been completed, or whether the vehicle is a vehicle subject to new control, the efficiency of vehicle control can be increased.

In particular, according to the present invention, when the detected vehicle is determined to be a pre-registration vehicle, only a necessary operation may be performed according to the control state of the vehicle. That is, in the case of a vehicle on which the enforcement has been completed, the enforcement operation is stopped, and in the case of a vehicle in progress, only the other operations related to the enforcement are performed, so that the previously performed enforcement operation is not repeated, thereby increasing the vehicle enforcement efficiency. On the other hand, even in the case of a vehicle whose enforcement has been completed according to the enforcement policy set for the target enforcement zone, the vehicle may be re-enforced after the set time has elapsed for a vehicle that has been parked for a long time exceeding the allowable time for one enforcement. .

The present invention does not detect the vehicle number in the case of a vehicle in which the vehicle number detection is impossible, but calculates the structural similarity of the image in the wide area detection image taken immediately before and newly for the corresponding enforcement area, and matching target according to the result It is determined whether the vehicle is the same vehicle, and when it is determined that the vehicle is a previously intercepted vehicle, the intermittent operation is performed according to the previously performed procedure, and a new intermittent operation is started for the vehicle determined as a new vehicle. It is possible to perform the intermittent action according to the situation.

1 is a block diagram of the overall configuration of the present invention.
2 is a view showing an example of the configuration of the field part of the present invention.
3 is a diagram showing the flow of operations performed in the system of the present invention.
Figure 4 shows an example of a wide-area detection image used in the present invention.
5 is a view showing a case in which the vehicle is not detected in the wide-area detection image of the present invention.
Figure 6 shows an image obtained by dividing the top of the wide area detection image of the present invention.
7 shows an image of detecting a vehicle not detected in FIG. 5 by mapping the image obtained by overlapping the wide area detection image of the present invention and the top thereof.
8 shows an image displayed by dividing a vehicle capable of recognizing a vehicle number without expanding the vehicle area among the wide-area detection images and a vehicle to be enlarged in the vehicle area.
9 is an image for showing an example of calculating the structural similarity of vehicle images.
10 is a view showing a case of cropping the vehicle area in the wide area detection image.
11 is a view showing the original brightness map prepared for the vehicle area cropped in FIG.
12 shows a brightness comparison map created by dividing the original brightness map prepared in FIG. 11 into patches based on the size of the IoU area.
13 is a view showing a case in which the system of the present invention displays the similarity calculation result of the brightness value pattern for the existing area and the new area on the screen of the wide area detection image.
14 is an example of visually showing a result of calculating a license plate number detection result and a structural similarity of an image for a vehicle area detected in a wide area detection image.
15 is another example of visually showing the result of calculating the structural similarity of the license plate number detection result for the vehicle area detected in the wide area detection image and the image.

The 'deep learning-based illegal parking enforcement system using wide-area images' of the present invention relates to a system for accurately and efficiently detecting a vehicle entering an illegal parking area using a single rotating camera installed in the field. . In particular, it relates to a system for quickly detecting a vehicle entering an illegal parking and stopping area using a wide area detection image obtained by photographing an image of a wide range of enforcement areas.

As a whole, as shown in FIG. 1, the system of the present invention generally includes a field unit 100 installed at the enforcement site, a situation room unit 300 installed at a remote location from the field unit, and a wired/wireless communication between the field unit and the situation room unit. It is configured to include a wired/wireless network 200 that connects through means.

The system of the present invention may be configured to include one or a plurality of field units 100 , as shown by a dotted line in FIG. 1 .

The field unit 100 of the present invention is configured to include an imaging unit 110 that is installed at the enforcement site to photograph a vehicle, and a communication unit 130 that communicates with the situation room unit through a wired/wireless network. The imaging unit 110 includes a rotating CCTV camera as a means for acquiring a wide area detection image and an image of a vehicle subject to enforcement. The imaging unit 110 is a wide-area detection image for detecting a violating vehicle within the enforcement zone, and an enforcement image for recognizing a vehicle number by moving the rotary camera to the position of the detected vehicle, and the circumstances of the vehicle determined as a violating vehicle A picture is taken and transmitted to the management server 310 . In detail, the wide-area detection image is transmitted to the vehicle detection unit 312 of the management server 310 , and the intermittent image and situational photos are transmitted to the vehicle enforcement unit 313 .

The situation room unit 300 is provided with a management server 310, which is a central component of detecting and intercepting vehicles in violation, and the management server 310 includes a camera remote control unit 311, a vehicle detection unit 312, and a vehicle enforcement unit. 313 , a control unit 314 , and a control information storage unit 315 , and may further include a display unit 316 and a learning image storage unit 317 .

The present invention applies a vehicle area detection model acquired through deep learning to a vehicle area, which is an image of a vehicle included in a wide area detection image taken of an illegal parking area, to detect a vehicle entering an illegal parking area, that is, a violating vehicle. perform the action 'Illegal parking and stopping area', 'vehicle control area', or 'regulation area' used in the description of the present invention are all names with the same meaning indicating the range of the area subject to parking and stopping violations.

In the present invention, it is determined whether the object included in the wide-area detection image is a vehicle, and when the object is determined to be a vehicle, that is, a violation vehicle, the number of the vehicle is recognized. If it is difficult to recognize the vehicle number from the image of the wide area detection image, the operation for recognizing the vehicle number is performed by enlarging the vehicle area. If the license plate cannot be recognized due to the cover of the license plate, IoU for the image of the previously detected vehicle area (existing vehicle area or existing area) and the image of the newly detected vehicle area (new vehicle area or new area) (Intersection over Union) value is calculated to calculate the structural similarity of the images of the existing and new vehicle areas, and it is determined whether the vehicle is a new enforcement target vehicle. If the similarity of the image is less than the threshold, the vehicle is determined as a new enforcement target vehicle, and the enforcement procedure for the new enforcement vehicle is performed.

The 'vehicle area' used in the present invention is not an image such as a photograph that targets the entire vehicle, but a part of the image of the rear or front of the vehicle that includes a license plate and a partial range that can recognize the characteristics of the vehicle. This means the selected part. A portion demarcated by a green rectangle in FIG. 5 and a portion demarcated by a green or light blue (hereinafter referred to as 'blue') rectangle in FIG. 8 correspond to examples of the vehicle area of the present invention.

In addition, in the present invention, only the license plate portion of the vehicle is selected and the partitioned license plate area is used. The license plate area is an area set by separately dividing only the license plate part of the vehicle area, and the vehicle detection unit 312 detects the vehicle number from the license plate area. In FIG. 5 of the present invention, a rectangle demarcated by a green line is a vehicle area, and a small rectangle demarcated by a red line inside the vehicle area in FIG. 8 corresponds to a license plate area.

The field unit 100 of the present invention may be formed in the form shown in FIG. 2 , and includes an imaging unit 110 and a communication unit 130 . When there are a plurality of field units 100 , an imaging unit 110 and a communication unit 130 are provided for each field unit.

The imaging unit 110 is a means for acquiring an image of the vehicle subject to enforcement, and is configured to include a rotational camera capable of photographing a wide-area detection image for the enforcement zone, such as a rotational CCTV camera. The rotary camera 111 of the present invention is a camera capable of taking a plurality of situational photos with different zoom values for each vehicle and an image for a wide range of enforcement zones through PTZ control, and is a general-purpose rotary CCTV camera (hereinafter referred to as 'time referred to as a 'typical camera') can be used.

As in the example of FIG. 2 , the field unit 100 may be configured in a form including a rotating camera, a vertical post and a horizontal support supporting the rotating camera, and a housing provided on one side of the vertical post. A power supply means for the imaging unit 110 and means constituting the communication unit 130 may be included inside the housing provided on one side of the vertical post.

The present invention detects whether a vehicle enters the enforcement zone and the image of the vehicle entering the enforcement zone using a wide-area detection image taken of the illegal parking zone, that is, the enforcement zone, and uses the detection result to detect the vehicle provided for use in the control of If the enforcement section to be monitored spans a very long or wide range, the enforcement section can be divided into a plurality of enforcement zones, and then a wide-area detection image for each enforcement zone can be taken and used. The enforcement zone used in the present invention refers to a range in which a wide-area detection image can be filmed using a rotary camera, and the enforcement zone is a range set for enforcement of illegal parking and stopping including one or a plurality of enforcement zones.

As a result of the vehicle detection unit 312 performing a detection operation on the object included in the wide-area detection image, the intermittent image is determined that the object is an offending vehicle, the vehicle enforcement unit 313 uses the rotating camera of the imaging unit several times. It includes a situation photo of the vehicle to be taken while changing the zoom magnification of the step, and a photo taken of the vehicle area and the license plate area. Circumstance photos among the crackdown videos are used for vehicles whose violations have been confirmed, and are used as a basis for violations in administrative actions such as imposing fines on vehicles in violation. In the present description, detailed descriptions related to intermittent operations such as shooting of intermittent images will be omitted.

The situation room unit 300 of the present invention is installed in a situation room located at a remote location from the field unit 100, and through the operation of the management server 310, the system automatically detects illegally parked and stopped vehicles. will be. When the field unit 100 is composed of a plurality, the management server 310 controls the rotary camera 111 installed in each field unit through the components to shoot a wide-area detection image and an intermittent image, and , to transmit the captured image back to the management server through the communication unit 130, and the like.

In the situation room unit 300 of the present invention, a management server 310 that controls the overall operation of the system, and transmission/reception of control signals and image data between the situation room unit 300 and the field unit 100 through a wired/wireless network 200 A communication unit 330 for performing functions such as transmission is provided.

The management server 310, which is an element performing a key role in the configuration of the situation room 300, includes a camera remote control unit 311, a vehicle detection unit 312, a vehicle enforcement unit 313, and a control unit 314, It is configured to include an enforcement information storage unit 315 , and may further include a display unit 316 and a learning image storage unit 317 .

The camera remote control unit 311 is a means for controlling the operation of the rotary camera 111 installed in the field. The rotary camera installed in the field unit is configured to be automatically operated under the control of the camera remote control unit 311 , and an administrator may operate the camera remote control unit 311 by manually operating the camera remote control unit 311 if necessary.

The system of the present invention is configured to automatically perform shooting of a wide-area detection image through a rotating camera, recognition of a license plate number, and securing of an interception image for an offending vehicle. On the other hand, it is configured so that it can be operated manually during initial setting or when an error occurs in the system and the administrator needs to manually reset the system. That is, in the system to which the present invention is applied, after all necessary matters for operation are set, the camera control by the camera remote control unit 311 and the detection of the offending vehicle through the vehicle detection unit 312 and the vehicle enforcement unit 313 or It automatically performs actions such as vehicle enforcement.

The vehicle detection unit 312 is a key means for performing an operation for detecting a vehicle illegally parked and stopped in an enforcement area in the present invention. The vehicle detection unit 312 determines whether the image of the object included in the wide-area detection image photographed and transmitted by the imaging unit of the field unit is for a vehicle, and if the object is a vehicle, whether it is possible to recognize the number of the vehicle By determining whether the vehicle number recognition is possible, whether it is a vehicle under control, a vehicle that has been cracked down, or a vehicle subject to new enforcement, and for a vehicle for which number recognition is not possible, the vehicle area included in the wide-area detection image taken immediately before Calculate the structural similarity between the (existing area) and the image of the vehicle area (new area) included in the newly captured wide area detection image, and depending on the calculation result, determine whether the vehicle included in the new area is a previously intercepted vehicle. It detects whether the vehicle is subject to enforcement.

When it is determined that the vehicle detected by the vehicle detection unit 312 is a new enforcement target vehicle, that is, a new violation vehicle, the vehicle enforcement unit 313 transmits the rotary camera of the imaging unit 110 through the camera remote control unit 311 to the corresponding vehicle. After recognizing the license plate number by moving it enlarged to the location of 315) to store it.

The control unit 314 is a means for controlling the operation of various components (refer to FIG. 1) included in the management server 310. That is, the control unit controls the vehicle detection unit 312 to perform the operation of detecting the offending vehicle while operating according to the sequence shown in FIG. 3 . The controller 314 controls the overall operation of the present invention for detecting an illegally parked or stopped vehicle using data (information) stored or set in the system. The control unit 314 may be configured in the same form as an MCU, or may be configured in the form of a program that is stored in each component of the management server and performs a related function.

The enforcement information storage unit 315 is a component in which data such as situation photos of the offending vehicle secured by the vehicle enforcement unit 313 in the enforcement process are stored. In the case of the parking violation enforcement system, if it is determined that the detected vehicle is a parking violation vehicle, it should be configured to present an objective basis to prove the violation of the vehicle to the owner of the vehicle. The enforcement information storage unit 315 of the present invention stores the vehicle number of the offending vehicle, information related to the enforcement date and time and the location of the enforcement, and a picture of the situation. The data stored in the enforcement information storage unit 315 is maintained for a certain period of time, and is used when necessary according to related administrative measures.

The display unit 316 is a means for displaying the wide-area detection image transmitted by the imaging unit on the screen, and also visually outputting the process or result of the vehicle detection unit 312 detecting the offending vehicle. The display unit 316 visually displays the result of determining whether the vehicle detected in the vehicle detection process is a vehicle under control, a vehicle for which enforcement is completed, or a new vehicle to be intercepted, etc. It is a means of indicating The display unit 316 is installed in the situation room unit 300 and visually displays whether the object entering the enforcement zone is a violation vehicle, the similarity calculation result between the existing vehicle area and the new vehicle area image, etc., thereby operating the system Make it easy to check the status. One or a plurality of display units may be installed.

The learning image storage unit 317 is a means for storing numerous learning images labeled with images of the vehicle area and license plate area used as a model for detecting illegally parked vehicles in the system of the present invention. The system of the present invention detects the vehicle area by applying the vehicle area detection model and the license plate area detection model built through deep learning in order to accurately detect the image of the vehicle area and the license plate area stored in the learning image storage unit 317. And it is possible to facilitate recognition of the vehicle number.

The present invention allows the system to learn the images of the vehicle area and the license plate area in advance through deep learning for the images of the vehicle area and the license plate area stored in the learning image storage unit 317, so that the image is taken from the imaging unit during system operation. If an image that is the same as or similar to the vehicle area or license plate area is included in the transmitted wide area detection image or the image enlarged by the super-resolution image restoration method, the image is recognized as for the vehicle area or the license plate area Controlling the vehicle is performed.

The deep learning applied to the present invention is to be performed in advance during system setting, so the image stored in the learning image storage unit 317 is not directly used in the process of the system performing vehicle detection and license plate recognition operations.

As described above, the vehicle area used in the present invention relates to the rear or front area of the vehicle including the license plate. The present invention detects the vehicle region from the wide-area detection image transmitted by the imaging unit, and then directly recognizes the vehicle number from the image of the license plate included in the vehicle region, that is, from the license plate region, or restores the image of the vehicle region to a super-resolution image Recognize the vehicle number after enlarging it by applying technology such as, and if the vehicle number cannot be recognized, the structural similarity of the image of the vehicle area corresponding to each other in the wide area detection image taken just before and the newly photographed wide area detection image By calculating, the operation of determining whether the vehicle including the new vehicle area is a previously intercepted vehicle or a new vehicle to be intercepted is performed.

Vehicle detection and vehicle number recognition according to the present invention are performed through the following process. Hereinafter, the contents of the operations performed in the present invention will be described in detail with reference to FIG. 3 and other related drawings.

The first operation performed in the system of the present invention is to move the rotary camera of the imaging unit 110 to the intermittent zone to take a wide-area detection image for the selected intermittent zone (S100).

The vehicle detection unit 312 controls the imaging unit 110 of the on-site unit 100 through the camera remote control unit 311 in order to detect an offending vehicle to photograph an illegal parking area subject to enforcement (enforcement area), and the corresponding enforcement A wide-area detection image for the area is acquired.

The enforcement zone may be divided into one or a plurality of settings according to the situation of the corresponding regulation zone. The situation of the enforcement section is related to the enforcement policy, such as the scope or length of the enforcement zone, whether the front and rear of the road are all set as enforcement zones with a rotating camera, and how many enforcement zones are set in crossroads, etc. will be.

In the system of the present invention, the vehicle detection unit 312 operates the imaging unit through the camera remote control unit 311 in order to detect whether a violation vehicle exists within each enforcement zone set according to the situation of the enforcement zone as described above, and a specific selected enforcement zone The operation (S100) of obtaining an image (wide area detection image) for the intermittent zone is performed by taking a wide-area detection image and transmitting the image to the vehicle detection unit 312 again. The wide-area detection image is scanned while traversing each enforcement zone set in the system with a certain time difference, or when it is detected that a violating vehicle has entered within a specific enforcement zone, a rotating camera is installed in the enforcement zone. You can set it to move and shoot.

Figure 4 shows an example of a wide area detection image applied to the present invention. The system of the present invention sets an intermittent section with a length of, for example, 250 m from a rotating camera by dividing it into 2 to 3 intermittent sections, and an image for each intermittent section (in the present invention, this image is called a 'wide area detection image') is taken and used to detect the vehicle area and license plate area. The system of the present invention can take a wide-area detection image at once when the length of the intermittent section is relatively short, and when the length of the section is long, the intermittent section is divided into a plurality of intermittent sections to obtain a wide-area detection image for each intermittent section It can be used for shooting. The wide-area detection image of FIG. 4 includes even a vehicle at a location quite far away from a vehicle in close proximity.

The vehicle detection unit 312 that has received the wide-area detection image as shown in FIG. 4 from the imaging unit 110 performs the deep learning-based offending vehicle detection step (S110 and S120) of detecting the offending vehicle using the entire image as an input. will do

The detection of the offending vehicle made by the vehicle detection unit 312 of the present invention is a vehicle area detection built by deep learning an image of a vehicle including a license plate, that is, a learning image for the vehicle area among the wide area detection images transmitted from the imaging unit. This is done using the model. Figure 5 is for a case of detecting a vehicle area in the wide-area detection image, and a portion divided by a green rectangle at the rear of the vehicle corresponds to the detected vehicle area.

On the other hand, in the case of a white vehicle located at the top of the wide area detection image in FIG. 5, it is not recognized as a vehicle, and thus is not detected as a vehicle area.

The system of the present invention builds a vehicle area detection model and a license plate area detection model by deep learning a number of learning images to detect the vehicle area and the license plate area during initial setting, and then starts the operation for detecting the offending vehicle. That is, after extracting the image of the object from the wide-area detection image, by applying a deep learning-based detection model to the extracted image, it is determined whether the image of the object corresponds to the vehicle area. If it is determined that the image of the object is a vehicle area as a result of the determination, the license plate area included in the vehicle area is extracted again and a deep learning-based detection model is applied to the license plate area to recognize the vehicle number.

In the present invention, when it is determined that the image of the target object corresponds to the vehicle area, that is, when it is determined that the vehicle is an offending vehicle, steps for more accurate detection, such as an operation for recognizing a vehicle number, are further performed. The steps further performed in the present invention include a step for recognizing a vehicle number and a step of calculating the structural similarity of the vehicle area image when the vehicle number recognition is not possible to determine whether the vehicle is a previously cracked vehicle or a new enforcement target vehicle, etc. to be.

When a vehicle region including a license plate is detected in the wide area detection image received from the imaging unit, the vehicle detection unit 312 starts an operation for number recognition of the corresponding vehicle. At this time, when the size of the detected vehicle area is greater than or equal to a certain size, the license plate area is immediately detected, and the vehicle number is recognized from the area (S130, S150).

The present invention is configured to accurately recognize the vehicle number by applying the detection model of the deep learning method when detecting the vehicle area performed above, and applying the detection model of the deep learning method in the vehicle number recognition process. When the size of the vehicle area included in the wide area detection image is greater than or equal to a certain size (S130), the vehicle detection unit 312 detects the license plate area and then performs an operation for recognizing the vehicle number. That is, when the vehicle area is larger than a certain size, the license plate area is separately detected and the vehicle number is recognized from the detected number plate area. In FIG. 5 , a rectangle delimited by a green line is a vehicle area, and a small rectangle demarcated by a red line inside the vehicle area is a license plate area.

The vehicle detection unit 312 of the present invention is configured to detect a vehicle area including a license plate, and detect a violation vehicle by applying a detection model of the deep learning vehicle area, The recognition operation is not performed.

On the other hand, when it is determined that the images of the vehicle area and the license plate area detected in the preceding process are for a vehicle subject to new enforcement, the vehicle detection unit 312 provides information on the corresponding offending vehicle to the vehicle enforcement unit 313 and , the enforcement information obtained in the process of the vehicle enforcement unit 313 receiving this information performing an enforcement operation on the corresponding vehicle is stored in the enforcement information storage unit 315 .

That is, when the vehicle detection unit 312 detects a new offending vehicle, information on the vehicle is provided to the vehicle enforcement unit 313 , and the vehicle enforcement unit 313 receiving this information is the camera remote control unit 311 . ) to enlarge and move the rotary camera of the imaging unit 110 to the location of the vehicle to recognize the vehicle number, then adjust the zoom magnification of the rotary camera to take a plurality of situational photos with different magnifications and transmit them again. And, after securing the situation photo of the new violation vehicle received, enforcement information such as the vehicle number for the vehicle, the detected vehicle area and the image of the license plate area, and the situation picture of the vehicle are stored in the enforcement information storage unit 315 to be saved in

In the above detection process, when it is difficult to detect the object because the size of the object included in the image of the wide area detection image received by the vehicle detection unit 312 is too small, the system of the present invention overlaps the upper part of the wide area detection image and (S110), to detect the offending vehicle by applying the deep-learning vehicle domain learning model to the divided images (S120).

For example, in FIG. 5 , in the case of a vehicle at the lower end of the image, the vehicle area is relatively large, so it is easy to detect the vehicle area and the vehicle number can also be easily recognized from the license plate area. On the other hand, in the case of a vehicle at the upper end of the image, the vehicle area is formed to be very small, so it is difficult to detect the vehicle area.

As in the case of the vehicle in the upper part of FIG. 5 , when the size of the object to be detected is relatively small in the entire image, it is difficult to detect the object and it is difficult to apply the deep learning method. Therefore, in the present invention, the image of the upper part in which the size of the object appears particularly small among the wide-area detection images is divided into a plurality, so that the size of the object (vehicle) in the image of the divided image, that is, the size occupied by the vehicle area is relatively large, By applying a deep learning detection model for detecting the vehicle area and the license plate area to the object included in the divided image, the vehicle detection unit 312 can easily detect the vehicle area or the license plate area.

On the other hand, in the present invention, in dividing the upper end of the wide area detection image, in order to prevent the problem that the object (vehicle) located near the dividing boundary is cut and the vehicle area or the license plate area is not detected properly, a part of the adjacent image is The images are segmented so that they overlap each other.

6 shows an example in which the upper part of an image is overlapped and divided in the present invention. Figure 6 is an example of a wide area detection image, the upper part of Figure 5 is divided into three images, it can be seen that the adjacent images among these images are divided so that the adjacent parts overlap each other partly. As such, the present invention separates and uses the upper part of the wide-area detection image to accurately detect a vehicle at a long distance, but partially overlaps the divided images, so that the vehicle area can be detected more safely.

The present invention performs deep learning-based infraction vehicle detection on the wide-area detection image obtained through the above process, and performs deep learning-based infraction vehicle detection on the image obtained by dividing the upper part of the image by overlapping. In detecting the vehicle area using the image and the superimposed segmented image in the wide area detection image as input, coordinate mapping is performed on the objects included in the two images, and if there is a duplicate object (vehicle), one of them is removed and collects the data of the vehicle area subject to enforcement.

In the present invention, in the overlapping division of the upper part of the wide area detection image, the size of the object itself contained in the divided image is not enlarged, and at the same time the position value of each object included in the wide area detection image and the divided image (image) Again, the same value is used without changing it. Therefore, when the same type of object exists in the same position in the wide-area detection image and the overlapped segmented image, it is determined that they are the same object, and only one is selected when selecting the final enforcement target to perform the operation for detecting the offending vehicle.

7 shows the result of performing overlapping division and deep learning-based offending vehicle detection, and this figure shows that even a white vehicle at the top of the image that was not detected in FIG. 5 was detected as a vehicle area (yellow rectangle) .

If the object included in the wide area detection image is not detected as a violating vehicle even after the above steps have been performed, the present invention stops the violating vehicle detection operation for the object (S120 No), detects another vehicle, or an enforcement zone is moved to perform a vehicle detection operation for other control zones (S100).

On the other hand, after going through the above steps, if it is determined that the object included in the wide area detection image is an offending vehicle (S120 yes), the vehicle number is obtained using the image of the vehicle area included in the previously acquired wide area detection image. It recognizes and performs an operation for determining whether the vehicle is already intercepted (S130 to S200).

As a result of performing the preceding steps (S100 to S120), when it is determined that the object included in the wide-area detection image is a vehicle (Yes in S120), that is, when a violation vehicle is detected, specific information about the vehicle is obtained To do this, the following steps are performed:

First, when the size of the detected vehicle area is greater than or equal to a certain size (Yes in S130), that is, when the size of the object is greater than or equal to a certain size, the license plate area is detected from among the vehicle area without performing the operation of enlarging the vehicle area and the vehicle number An operation for recognizing is performed (S150).

For example, in the case of the wide-area detection image of FIG. 8, vehicles 1 and 3 can directly recognize the vehicle number from the license plate without magnifying the image. That is, when the size of the vehicle area is large enough to easily recognize the number, such as vehicles 1 and 3, and is larger than a certain size, the vehicle number is recognized immediately without enlarging the image of the vehicle area.

As above, if it is determined that the vehicle area in the wide area detection image is large enough to easily recognize the vehicle number (Yes in S130), after detecting the license plate area, whether or not the vehicle number can be recognized from the license plate area It is determined (S150).

When it is determined that the vehicle number can be recognized from the license plate area (Yes in S150), the vehicle detection unit 312 recognizes the vehicle number and determines whether the vehicle of the number is a previously intercepted vehicle (S180). As a result of the determination, if the number of the recognized vehicle is already in progress or matches the number of the vehicle for which the enforcement has been completed (Yes in S180), the vehicle detection unit 312 determines that the vehicle is a previously intercepted vehicle and the vehicle Stops the detection operation for , and performs the detection operation for another vehicle, or moves the intermittent zone to perform the vehicle detection operation for the other zone (S100).

Next, if the size of the detected vehicle area is not larger than a certain size (No in S130), the system of the present invention determines how much the vehicle area can be enlarged to detect the vehicle area using the deep learning detection model. This magnification ratio is called the target magnification ratio. After the target magnification ratio is determined, it is determined whether the determined target magnification ratio is equal to or less than a predetermined size (S140).

If the license plate image is enlarged within the target magnification ratio, if it is determined that the license plate image can be recognized (S140 Yes), the vehicle detection unit 312 performs deep learning-based super-resolution image restoration on the vehicle area to restore the image of the vehicle area. is enlarged (S145), the license plate area is detected from the enlarged image, and the vehicle number is recognized from the license plate area (S150).

When magnifying an image, in general, if the image is magnified by more than 3 times, accurate recognition is difficult even if super-resolution image restoration is applied. Therefore, the target magnification ratio applied to the present invention is set to be within 3 times. In this case, in determining the target magnification ratio for enlarging the vehicle area in S140, when the size of the license plate is about 150 pixels in the horizontal direction to be accurately recognized, if the size of the detected license plate area is 80 pixels, the target magnification ratio becomes 2, and in this case, the vehicle number can be recognized by magnifying the image twice. On the other hand, if the size of the detected license plate area is 40 pixels, the target magnification ratio should be 4, so in the latter case, the vehicle number recognition becomes ineffective. Therefore, the target magnification ratio becomes one reference factor for determining whether the vehicle number can be recognized.

If it is determined that the target magnification ratio for the vehicle area is less than or equal to a certain size (S140 Yes), the vehicle detection unit 312 performs deep learning-based super-resolution image restoration on the target vehicle area to adjust the size of the vehicle area by the target magnification ratio. After the enlargement (S145), it is determined whether the license plate area detection and vehicle number recognition are possible from the enlarged image (S150). As a result, when it is determined that the vehicle number recognition is possible (S150 Yes), the vehicle detection unit 312 determines whether the vehicle is a previously intercepted vehicle after recognizing the vehicle number (S180). As used in the present invention, the term 'vehicle that has been cracked down' includes both a vehicle in progress and a vehicle in which the crackdown has been completed.

If it is determined that the vehicle number recognized through the preceding steps matches the number of the previously intercepted vehicle (Yes in S180), the vehicle detection unit 312 stops the detection operation for the corresponding vehicle, and detects another vehicle The operation is performed or the control area is moved to perform the vehicle detection operation for the other control area (S100).

On the other hand, if it is determined that the number of the vehicle detected in step S180 does not match the number of the previously intercepted vehicle (No in S180), since the vehicle is a new control target vehicle, the vehicle detection unit 312 controls the new vehicle. It is determined as a target vehicle, and a new vehicle area model related to the vehicle is added to the new control target vehicle list (S190). The information on the newly added vehicle subject to enforcement is provided to the vehicle enforcement unit 313 , and the enforcement procedure for the corresponding vehicle is performed by the vehicle enforcement unit 313 . The new vehicle area model includes the number of the enforcement area in which the new enforcement target vehicle is located, the location value of the vehicle, and the image of the vehicle area.

On the other hand, if the vehicle detection unit 312 determines that the vehicle number cannot be recognized even if the vehicle area is enlarged to a predetermined size, that is, when it is determined that the target enlargement ratio exceeds the predetermined size (S140 No), the vehicle The detection unit 312 stops the operation of recognizing the number of the vehicle and performs an operation of calculating the structural similarity of the image of the vehicle area (S160).

As described above, the vehicle detection unit 312 of the present invention determines that the size of the vehicle area in the wide-area detection image is greater than or equal to a certain size and the target magnification ratio is less than or equal to a certain size (S140 Yes). After enlarging the size of the vehicle area by executing the running-based super-resolution image restoration, the histogram smoothing (CLAHE) process is performed on the license plate area image of the vehicle to accurately recognize the vehicle number. A histogram expresses the brightness distribution (contrast ratio) of an image in the form of a bar graph, and by using this, it is possible to know the brightness distribution and saturation (brightness and darkness of color) of the entire image.

The present invention improves the accuracy of vehicle number recognition by applying the Contrast Limited Adaptive Histogram Equalization (CLAHE; 'Image Histogram Equalization' or 'Image Histogram Equalization') technique to the license plate image (license plate area) of the vehicle. making it higher In case of performing CLAHE processing to limit the contrast ratio, it is possible to increase the accuracy of number recognition by adjusting the abnormal illuminance of the image. In the present invention, the 'contrast limit histogram' will be referred to as a 'brightness histogram'.

The vehicle detection unit 312 of the present invention calculates a brightness histogram for the detected image of each vehicle area, and then, when the bias of the brightness histogram exceeds a threshold, further includes an adjustment unit that adjusts brightness balance to see the vehicle number effectively recognized.

On the other hand, if the vehicle number is not recognized even after magnification of the wide area detection image and image histogram smoothing (CLAHE) processing (S150 No), the operation for vehicle number recognition is stopped and the structural similarity to the image of the vehicle An operation of calculating ? is performed (S160).

The structural similarity of images is used to determine how similar two images are, and in the present invention, IoU (Intersection over Union) is applied to determine the similarity of images between vehicle regions. That is, by calculating the IoU for the image of the same vehicle or the same location included in the wide area detection image and the newly photographed wide area detection image for the same intermittent zone, the structural similarity of the two images is compared.

In the present invention, when the target enlargement ratio exceeds a certain size and when the license plate is covered, the operation for detecting the license plate number is not performed. That is, even in the case of a vehicle in which the license plate region image cannot be detected from among the vehicle region images because the license plate is covered, the vehicle detection unit 312 does not perform an operation for detecting the license plate number.

When the vehicle number cannot be recognized for the reasons described above, the vehicle detection unit 312 of the present invention analyzes the structural similarity between pixels forming the image of the vehicle area to determine whether the vehicle is an intermittent vehicle (S160, S170).

9 is a view showing an example of calculating the structural similarity of the vehicle image using a wide area detection image taken with a parallax. The picture on the left in the drawing is an image taken immediately before, and the picture on the right is a newly shot image.

In this drawing, in the case of vehicles partitioned by a blue rectangle, it is determined that license plate number recognition is possible, so structural similarity analysis is not performed. On the other hand, in the case of vehicles partitioned by a green rectangle at the upper end of the left and right photos, it is determined that the vehicle number is not recognized, and in this case, the present invention performs the steps as described below, It is determined whether the vehicle is a vehicle under speed control or a vehicle subject to new enforcement.

That is, when it is determined that the vehicle number recognition is impossible through the preceding steps (S130 and S140), the vehicle detection unit 312 analyzes the structural similarity between the pixels of the vehicle area image included in the wide area image. It is determined whether the object is an air-controlled vehicle.

The image structural similarity analysis of the present invention is performed through a method of calculating IoU (Intersection over Union) between a previously detected vehicle area (existing area) and a newly detected vehicle area (new area), in particular, the brightness of two vehicle areas By calculating the IoU using the value, the structural similarity of the two regions is calculated. For reference, Intersection over Union (IoU) is a method used to determine how accurately the position of an object is detected, and it is to calculate the degree of overlap between two pictures or pictures. IoU can be expressed in % units, and when the areas of two pictures or photos of the same size overlap each other by 2/3, IoU becomes 50%.

The IoU calculation for the vehicle area is to calculate the IoU between the existing area, that is, the vehicle area where the enforcement is already in progress or the enforcement has been completed, and the new area, that is, the newly detected vehicle area. For this analysis, the image of the existing area is and the image of the new area will be used. These images are images obtained by cropping the corresponding vehicle area from among the two wide-area detection images transmitted by the imaging unit with a time difference.

In the present invention, the image of the vehicle area obtained by cropping as above is called the original map. Separately, create a brightness original map.

Of the accompanying drawings, FIG. 10 is an example of an original map in which a vehicle area is cropped from a wide-area detection image, and FIG. 11 is an example of an original brightness map prepared by extracting brightness values from the original map of FIG.

Here, the reason why IoU is calculated by using the original brightness map prepared separately instead of using the original map as it is, is because the accuracy of comparison increases when the original map created based on the brightness value is used. That is, the original map taken with a rotating camera includes elements such as hue, saturation, and brightness. In particular, in the case of a vehicle exposed to the outside air, the color tone, saturation, etc. This is subject to change. Therefore, when calculating the IoU based on the hue or saturation value, it is often difficult to obtain an accurate IoU value for the vehicle area.

After creating the original brightness map for the existing and new areas as above, each original brightness map is divided into patches according to the size of the IoU area, and then the brightness comparison including each patch corresponding to each other in the existing and new areas is compared. Create a map.

The brightness comparison map prepared in the form shown in FIG. 12 is for comparing the brightness values of patches located at corresponding positions in the images of two vehicle regions divided into the same number of patches, respectively. 12 shows an example of a brightness comparison map created by dividing the original brightness map of FIG. 11 into patches according to the size of the IoU area. For reference, the original map, the original brightness map, or the brightness comparison map illustrated in FIGS. 10 to 12 are diagrams for reference for explaining the method of calculating the structural similarity of images. When another vehicle enters and the image of the vehicle area included in the picture taken immediately before is different from the image of the vehicle area included in the newly taken picture, the left and right sides of the original map, the original brightness map, and the brightness comparison map images are different from each other.

Meanwhile, after calculating the average brightness value for each patch divided in the brightness comparison map of the two vehicle regions to be compared, the calculated average brightness value is set as the representative brightness value of the corresponding patch. Therefore, as the size of the divided patch decreases, IoU for the original brightness map of the two vehicle regions can be accurately calculated. In the present invention, the representative brightness value and the average brightness value are actually the same value, and the representative brightness value is a comparison value of brightness used in calculating the IoU for the original brightness map of the two vehicle areas.

After the representative brightness values for each patch in the existing and new areas are calculated as above, the covariance and standard deviation of the representative brightness values for each patch are calculated, and the correlation coefficient obtained by dividing the covariance by the standard deviation is calculated to determine the structural similarity of the image. will yield

The time required to scan the entire enforcement zone using the system of the present invention may vary depending on the range of the enforcement zone, but takes about several minutes. On the other hand, when the scanning time of the enforcement zone is a time period in which the amount and direction of sunlight are rapidly changed, such as dawn or evening, the color tone of the vehicle changes even if the same vehicle is photographed with a short time difference. On the other hand, despite the time difference or color tone change, the difference in the brightness values of each of the divided patches compared to the average brightness values of the entire vehicle region is maintained.

The present invention is included in the brightness value pattern of the vehicle area (existing area) included in the wide-area detection image taken immediately before and the newly captured image by using the point that the difference in brightness value between patches is maintained the same in the same vehicle. How similar the brightness value patterns of the existing vehicle area (new area) are, that is, the structural similarity of images is calculated to determine whether the two vehicle areas relate to the same vehicle.

In Fig. 9, which is a drawing to show an example of calculating the structural similarity of the vehicle image, the left vehicle number 1 and the right vehicle number 2 are vehicles capable of recognizing a vehicle number, and matching is completed because it is determined that the vehicle number is the same, Vehicle number 3 on the left and vehicle number 3 on the right can also be recognized, and the vehicle number is determined to be the same vehicle, so matching is completed. In the present invention, image similarity is not calculated for these vehicles capable of recognizing license plates.

On the other hand, in the case of a vehicle whose number is not recognized among the vehicles of FIG. 9 (cars 4, 2 and 5 on the left and vehicles 5, 1, 4, and 6 on the right), the structural similarity of the vehicle image is calculated. Among vehicles, left vehicle 2 and right vehicle 4, left vehicle 4 and right vehicle 1, left vehicle 5 and right vehicle 6 are determined to be the same vehicle because the structural similarity of images is greater than or equal to the threshold, and 5 on the right Vehicle No. is a case in which there is no vehicle to prepare for in the photo on the left, so it is judged to be a new vehicle.

That is, in the present invention, when the structural similarity of the image exceeds a threshold value, the vehicle is determined as an air raid vehicle, and when the structural similarity of the image exceeds the threshold value, it is determined as a new enforcement target vehicle.

13 is a diagram showing a case in which the system of the present invention calculates the similarity of the brightness value pattern for the existing area and the new area, and then displays the result on the screen of the wide area detection image.

For example, in FIG. 13 , a vehicle area partitioned by a red square is determined as a new vehicle, a vehicle area partitioned by an orange square is determined as a vehicle in progress, and a vehicle area partitioned by a blue square is a vehicle that has been intercepted. It may indicate that it is determined as . In other words, the vehicle area partitioned in orange and blue in the above screen corresponds to the vehicle that has been identified as a previously intercepted vehicle, and is displayed in different colors so that the vehicle can be classified as either in progress or completed according to the enforcement status of the vehicle. example shown.

In the present invention, as a result of calculating the similarity between the existing area (existing vehicle area) and the new area (new vehicle area) through the preceding steps (S160 and 170), when the calculated similarity exceeds a set threshold (S170) No), the vehicle including the new area is classified as a pre-enforcement vehicle, that is, a vehicle in progress or for which the control has been completed, and the model of the existing vehicle area is updated using the image of the new vehicle area (S175). That is, when it is determined that the new vehicle area used for calculating the similarity of the image is for the vehicle under control, the vehicle detection unit 312 of the present invention updates the image of the existing area with the image of the new area, and the vehicle area of the corresponding vehicle. to update the image taken at the most recent point in time (S175).

On the other hand, as a result of calculating the structural similarity of the image, if the similarity is less than or equal to the threshold (Yes in S170), the vehicle including the vehicle area is classified as a new enforcement target vehicle. In this case, since another vehicle is entering the same location, the vehicle detection unit 312 adds this vehicle to the new vehicle area model (S190), and then transfers the information of the added new vehicle area model to the vehicle enforcement unit ( 313), so that the vehicle enforcement unit 313 performs an operation for cracking down on the offending vehicle (S200). In addition, when the new area is detected as a new enforcement target vehicle as described above, the vehicle enforcement unit 313 after a certain time has elapsed for the existing vehicle area model including the image of the existing vehicle area stored in the enforcement information storage unit 315 . ) moves the rotary camera to the location of the vehicle to confirm that the vehicle does not exist, and then deletes the vehicle area model related information for the existing vehicle from the enforcement information storage unit 315 .

On the other hand, when the vehicle detection unit 312 provides information on the new vehicle to be cracked down, that is, information on the new vehicle area model to the vehicle enforcement unit 313 , the vehicle enforcement unit 313 controls the field through the camera remote control unit 311 . Control the imaging unit 110 to move the rotating camera to the location of the vehicle to take pictures related to enforcement information including images of the vehicle area and license plate area, and a plurality of circumstantial pictures with different zoom magnifications, The enforcement information including the above photos received from the unit 110 is stored in the enforcement information storage unit 315 .

The present invention relates to a technique derived for quickly and efficiently detecting a parking and stopping violation vehicle, and therefore a detailed description of the details of the enforcement action performed on the vehicle detected as a violation vehicle will be omitted.

Most of the operations for detecting a violation vehicle in the present invention are performed by the vehicle detection unit 312 . That is, the vehicle detection unit 312 of the present invention determines whether the object in the wide area detection image transmitted to the vehicle detection unit 312 by photographing the image of the object entering the enforcement zone by the imaging unit 110 is for the vehicle. And, if the object is a vehicle, an image of the vehicle area including the license plate of the vehicle is obtained, and a deep learning detection model is applied to the image of the vehicle area to determine whether the vehicle is an air raid vehicle or a new vehicle subject to enforcement cognition is detected.

The vehicle detection unit 312 includes an object in the wide-area detection image, but when it cannot detect the object because the size is very small, it performs overlapping division on the upper part of the image, and a deep learning detection model for the divided image. It is applied to detect whether the object is an image of the vehicle area.

As a result of the above steps, if it is determined that the object is for the offending vehicle, the present invention performs the following operation to recognize the vehicle number.

First, the step for recognizing the vehicle number performed in the system of the present invention is to determine whether the size of the vehicle area included in the wide-area detection image is greater than or equal to a certain size, and steps of different contents are performed according to the result. .

Next, if the size of the vehicle area included in the wide area detection image is over a certain size, it is determined whether the vehicle number recognition is possible without enlarging the image of the vehicle area, and if the vehicle number recognition is possible, the recognized vehicle Using the number, it is determined whether the vehicle is a previously-controlled vehicle, and if the vehicle number matches the previously-controlled vehicle, the additional detection operation for the vehicle is stopped, and the detection operation for another vehicle is performed or a rotary type The camera is moved to another intermittent area and the detection operation is performed for that area.

On the other hand, if the vehicle number is not recognized even though the size of the vehicle area included in the wide area detection image is over a certain size, the vehicle number is not recognized, and instead, the vehicle area (new vehicle area) and immediately before the recognition are taken. Structural similarity of the images to the converted vehicle area (existing vehicle area) is calculated, and it is determined whether the two images are of the same vehicle according to the calculated image similarity value. If the similarity of the calculated images exceeds a set threshold, it is determined that the two vehicle areas relate to the same vehicle.

As a result of the previous determination, if it is determined that the existing vehicle area and the new vehicle area relate to the same vehicle, the vehicle is determined as an air-control vehicle, and if it is determined that the existing vehicle area and the new vehicle area relate to different vehicles, the vehicle is determined as a new vehicle to be cracked down, and the vehicle enforcement unit 313 performs a crackdown procedure on the vehicle.

Meanwhile, the configuration of the present invention has been described above with reference to FIG. 1 . Among the configurations of the present invention, the management server 310 may include one or more display units 316 . The display unit 316 provided in the present invention is configured to display wide-area detection images as shown in FIGS. 4 and 13, or images of existing and new vehicle regions and images of both regions as shown in FIGS. 8, 9, 14 and 15. It may be configured to visually express a vehicle detection related situation performed by the vehicle detection unit 312, such as a structural similarity calculation result.

14 and 15 are examples of visually showing the results of vehicle number recognition and structural similarity calculation of vehicle area images through the system of the present invention through the display unit. The display unit of the present invention is a means that can be provided in the situation room, and can visually show the vehicle detection result through the image.

In the case of FIG. 14, the wide area detection image and the newly photographed wide area detection image are shown on the upper left and right, respectively, and a portion in which the vehicle area is detected is divided into a rectangle in each image. In the table formed long left and right in the middle of FIG. 14, after detecting vehicle areas (referred to as 'matching target vehicle areas') corresponding to each other in the upper left and right images, detection performed on the entire matching target vehicle area The results are expressed in specific characters or numerical values. In the lower left and right of FIG. 14, a brightness comparison map obtained by dividing each matching target vehicle area by patch is shown, and in the inner center of these brightness comparison maps, a brightness structure map created using the representative brightness values for each vehicle area for each vehicle area is shown. . The vehicle area patch division brightness comparison map on both sides of FIG. 14 and the brightness structure map displayed therein show the result values for the selected specific matching target vehicle, and the selected matching target vehicle area is displayed in the table at the middle.

In Figure 14, the vehicle area partitioned by the blue rectangle is for a vehicle that can recognize the vehicle number directly from the wide area detection image or by enlarging the image of the wide area detection image to recognize the vehicle number, and the vehicle divided by the green rectangle The area is a case in which the vehicle number recognition is not possible even after magnifying the wide area detection image itself and the image of the enlarged wide area detection image, and additional steps such as calculating the structural similarity of the image must be performed.

In Fig. 14, the vehicle areas detected by the system are 4 in the case of the left and 5 in the case of the right. Also, among the detected vehicle areas, matching target vehicle areas are, respectively, the first vehicle on the left and the second vehicle on the right, the third vehicle on the left and the first vehicle on the right, the fourth vehicle on the left and the fifth vehicle on the right, and the second vehicle on the left and the right vehicle on the right, respectively. This is for vehicle 3, and in the case of vehicle 4 on the right, there is no matching target because there is no vehicle corresponding to the left.

In FIG. 14, the vehicle number 1 on the left and vehicle number 2 on the right side have a vehicle area larger than a certain size and the vehicle number can also be recognized. because the number is the same. This is a case where two vehicle areas are detected as being for the same vehicle and matching is completed. In the case of this vehicle, since it corresponds to an air raid vehicle, the vehicle detection unit 312 does not perform an additional detection operation for the vehicle.

In Fig. 14, in the case of the 3rd vehicle on the left and the 1st on the right vehicle, and the 4th vehicle on the left and the 5th vehicle on the right side, it was determined that it was difficult to recognize the license plate number from the wide area detection image, and the structural similarity of the two vehicle area images was obtained. This is a calculated case, and in this case, the structural similarity of the images calculated for the corresponding vehicle area is equal to or greater than the threshold, so that the matching target vehicle is determined to be the same vehicle and matching is completed. In the case of these vehicles as well, since they correspond to pre-registration vehicles, the vehicle detection unit 312 does not perform an additional detection operation for these vehicles, and the vehicle enforcement unit 313 performs a necessary interception operation on these vehicles.

On the other hand, although it is assumed that the second vehicle on the left and the third vehicle on the right are in the same location and have similar appearances in FIG. 14, they are actually different vehicles, and the vehicle area brightness shown on the left and right sides at the bottom of FIG. 14, respectively The comparison map and the brightness structure map are for these vehicles. This is a case in which the structural similarity of the image for the vehicle area is calculated because the vehicle number cannot be recognized from the wide-area detection image for these vehicles. In this case, it is determined that there is no matching vehicle because the similarity of the image calculated by the vehicle detection unit 312 is less than a threshold (eg, 50%), and the third vehicle on the right is determined as a new enforcement target vehicle, and the vehicle enforcement unit (313) to perform the intermittent operation for the new vehicle. The brightness structure map for the vehicle areas of the left and right vehicles in the center at the bottom of FIG. 14 also shows that there is a significant difference in the shape of the two vehicle areas. In the case of these vehicle areas, since the similarity of the images is less than or equal to the threshold, matching between the two vehicles is unsuccessful.

In addition, in the case of vehicle No. 4 on the right in FIG. 14, since it is a vehicle that was not in the previous wide area detection image (photo on the left), the vehicle detection unit 312 determines that there is no vehicle matching the vehicle No. 4 on the right, and analyzes the structural similarity. Instead, it is immediately determined as a new enforcement vehicle, and information related to enforcement is provided to the vehicle enforcement unit.

In the case of the present invention, in detecting the offending vehicle, even if it is estimated that the same vehicle exists in the existing area and the new area and there appears to be no position movement, the vehicle area is detected by covering the license plate of the vehicle by another vehicle entering later. If this is not possible, an additional detection operation for the corresponding vehicle cannot be performed. In this case, the vehicle enforcement unit 313 stops the crackdown on the vehicle, and when the vehicle covering the license plate moves and the license plate is detected in the subsequent scanning process, it is determined as a new enforcement target vehicle and a new enforcement operation is performed. .

In the present invention, for the vehicle in which the image of the vehicle area appears very small because the vehicle photographed with the wide-area detection image is too far away, the image enlargement and brightness comparison map using the super-resolution image restoration and the overlapping division of the upper part as described above. By applying the image structural similarity analysis, etc., it is possible to calculate the similarity of the image for the area that could not be distinguished in the wide-area detection image, and depending on the calculation result, it is possible to detect whether the two vehicles are for the same vehicle or a new vehicle. be able to

Meanwhile, unlike the case of FIG. 14 , FIG. 15 is a case in which the detection result of a vehicle determined to be the same vehicle as the structural similarity of the image exceeds a threshold value.

Also in FIG. 15, in the case of a pair of corresponding vehicles (left vehicle No. 2 and right vehicle No. 1) that are close to the rotating camera, the detected vehicle area is larger than a certain size, so the vehicle number is recognized directly from the wide area detection image. As a vehicle capable of this, matching is completed.

On the other hand, in the case of vehicles 1, 4, and 3 on the left and vehicles 3, 4 and 2 on the right, it is determined that the vehicle number cannot be recognized directly from the wide-area detection image. In this case, the target expansion for the vehicle area If the ratio is calculated and the ratio is less than a certain size, an operation is performed to recognize the license plate number Structural similarity of the image to the image of the vehicle area and the new vehicle area is calculated.

In the case of left vehicle 1 and right vehicle 3, left vehicle 3 and right vehicle 2, and left vehicle 4 and right vehicle 4 in FIG. 15 , the structural similarity of images is a threshold (eg, 50%) This is a case where it is determined that it is for the same vehicle in excess of , and matching is completed.

In Fig. 15, the brightness comparison map respectively displayed on both sides of the lower part and the two brightness structure maps displayed in the center are prepared for the vehicle areas of the 3rd vehicle on the left and the 2nd vehicle on the right, respectively. It can be seen that the brightness structure map has a very similar shape.

Since the third vehicle on the left and the second vehicle on the right in FIG. 15 are determined to be the same vehicle, the vehicle detection unit 312 stops the detection operation for this vehicle and displays the image of the newly captured image (the second vehicle on the right). It is saved (updated) as an image of the relevant vehicle area. This vehicle corresponds to a vehicle that has already been cracked down or has been cracked down through the vehicle enforcement unit 313 .

The effects that can be expected through the system of the present invention configured and operated as above are as follows.

First, the present invention can increase the operating efficiency of the system by detecting a vehicle entering an illegal parking and stopping area using a wide-area detection image, so that the violating vehicle can be quickly detected in a widely divided enforcement area compared to the prior art. have.

The present invention uses a wide-area detection image obtained by photographing a wide range of intermittent zones compared to the prior art, and after enlarging the image by applying a super-resolution image restoration technology to the image of an object included in this image, the vehicle area By detecting it, it is possible to increase the distance of the infringing vehicle enforcement area.

The present invention recognizes the vehicle number in the vehicle area detected in a small size by directly recognizing the vehicle number according to the size of the vehicle area detected in the wide area detection image, or by applying deep learning-based super-resolution image restoration to the vehicle area. This increases the accuracy and speed of enforcement. In particular, the present invention overlaps the upper part of the image to detect an object that appears too small in the wide area detection image, and super-resolution image restoration technology and a deep learning detection model for the image of the object included in the divided area. By applying it, it is possible to accurately detect whether a vehicle violates even an object that is difficult to detect in the prior art.

The present invention divides the range set by the system into the vehicle area among the rear or front areas of the vehicle containing the license plate of each vehicle included in the wide area detection image, and uses the image of the area to recognize the number of the vehicle or Alternatively, it is possible to increase the efficiency of vehicle enforcement by determining whether the corresponding vehicle is a vehicle under enforcement, a vehicle for which enforcement has been completed, or a vehicle subject to new enforcement, and allowing the system to operate according to the vehicle situation.

The present invention can increase the accuracy of vehicle detection by using the brightness value of the image when calculating the structural similarity of the existing and new vehicle region images.

In particular, the present invention prevents the duplicate detection operation from being performed on the previously intercepted vehicle, and does not detect the vehicle number in the case of a vehicle for which identification of the vehicle number is not possible, but instead of detecting the vehicle area immediately and newly photographed for the vehicle. It is determined whether the corresponding vehicle (matching target vehicle) is the same vehicle by calculating the structural similarity of the image. By starting a new crackdown operation only for the identified vehicle, it is possible to increase the speed of the crackdown.

As described above in detail with respect to the content of the present invention, the technical spirit of the present invention is not limited to the above description, but those of ordinary skill in the art to which the present invention pertains are the same as the technical spirit of the present invention It should be construed as being included in the scope of the present invention, including those that are similar or applicable without special technical changes, and similar or equivalent to the technical spirit of the present invention described in the claims.

100: field department
110: imaging unit
111: rotating camera
130: communication department
200: wired and wireless network
300: situation room
310: management server
311: camera remote control unit
312: vehicle detection unit
313: vehicle enforcement unit
314: control unit
315: enforcement information storage unit
316: display unit
317: learning image storage unit
330: communication department

Claims (5)

In the illegal parking enforcement system comprising a field unit installed at a crackdown site, a situation room at a remote location, and a wired/wireless network connecting the field unit and the situation room by wire or wireless,
The on-site unit provides a wide-area detection image for detecting a violation vehicle within the enforcement area and an enforcement image for recognizing a vehicle number by moving to the location of the detected vehicle, and taking a picture of the situation for the violation vehicle and transmitting it to the management server Consists of an imaging unit,
The situation room unit is configured to include the management server,
The management server includes a camera remote control unit for remotely controlling the imaging unit, a vehicle detection unit receiving a wide-area detection image from the imaging unit to detect a violating vehicle, and information on the violating vehicle detected by the vehicle detection unit After receiving the information It is configured to include a vehicle enforcement unit that obtains a situational picture of the offending vehicle from the imaging unit, and an enforcement information storage unit that stores information on the enforcement performed by the vehicle enforcement unit,
The vehicle detection unit detects an offending vehicle by applying a deep learning detection model to an object in the wide area detection image obtained from the imaging unit, and determines that the detected vehicle is a vehicle in progress or completion of enforcement. If the size is larger than a certain size, the vehicle number is recognized immediately, and if the detected vehicle area is smaller than a certain size and the target magnification ratio is less than a certain size, deep learning-based super-resolution image restoration is performed for the vehicle area. After enlarging the , the vehicle number is recognized, and if vehicle number recognition is not possible, the structural similarity between the image of the existing vehicle area and the image of the new vehicle area is calculated, and based on the result of the calculation, it is determined whether the new vehicle area is being cracked down or the enforcement is complete. Or to determine whether the vehicle is a new enforcement target,
When the vehicle enforcement unit determines that the detected vehicle is a new vehicle subject to enforcement, the camera remote control unit moves the imaging unit to the location of the vehicle to recognize the vehicle number, then adjusts the zoom magnification of the imaging unit to adjust the magnification It operates to take and transmit a plurality of different situational photos, and secure the received situational photos for the offending vehicle,
The enforcement information storage unit stores the enforcement information and situation photos secured by the vehicle enforcement unit,
To calculate the structural similarity of the image, the overlapping area is calculated through IoU (Intersection over Union) of the existing vehicle area and the new vehicle area, and cropping is performed on the overlapping area of each vehicle area, and then the existing vehicle area of the cropping area After creating the original brightness map for the new vehicle area and calculating the size of the patch based on the size of the cropped area, the brightness comparison map composed of the divided patches is created, and the average brightness value for each patch in the brightness comparison map. After calculating the calculated average brightness value as the representative brightness value, the representative brightness value for each patch in the brightness comparison map of the existing vehicle area and the representative brightness value for each patch in the brightness comparison map of the new vehicle area are covariance and standard deviation , and a correlation coefficient obtained by dividing the covariance by the standard deviation as a similarity value, a deep learning-based illegal parking enforcement system using a wide-area detection image.
According to claim 1,
The vehicle detection unit performs overlapping division for the upper end of the wide area detection image, and performing the step of applying a deep learning-based vehicle area detection model to the object detected in the divided image, characterized in that the wide area detection image A deep learning-based illegal parking enforcement system.
According to claim 1,
The vehicle detection unit Deep learning-based using a wide area detection image, characterized in that it further comprises an adjustment unit for adjusting the brightness balance when the bias exceeds the threshold after calculating the brightness histogram for the image of each vehicle area detected in the wide area detection image Illegal parking enforcement system.
delete According to claim 1,
The management server is a deep learning-based illegal parking enforcement system using a wide-area detection image, characterized in that it further comprises a display that visually outputs a situation in which the wide-area detection image and the vehicle detection unit 312 detects the offending vehicle.

Images