KR20240084863A - Deep Learning Technology-Based Traffic Law Violation Video Analysis Method And Computer-Readable Recording Medium Recording The Program That Performs It - Google Patents

Abstract

The present invention relates to a deep learning technology-based method of analyzing images of traffic law violation public interest reports and a computer-readable recording medium on which a program for performing the same is recorded. More specifically, it relates to a black box installed in any vehicle by at least one processor. A public interest report image acquisition step in which a public interest report image captured from a camera is obtained, and a first deep learning model that recognizes a vehicle object and a second deep learning model that recognizes a lane object in the public interest report image by the at least one processor. About a deep learning technology-based traffic law violation public interest report video analysis method including a violation output step in which the violation of the course change of the vehicle object is output by inputting it into the computer-readable recording medium on which the program for performing the same is recorded. will be.

Description

Deep Learning Technology-Based Traffic Law Violation Video Analysis Method And Computer-Readable Recording Medium Recording The Program That Performs It }

The present invention relates to the field of computer vision, and relates to a method of analyzing images of traffic law violation public interest reports based on deep learning technology and a computer-readable recording medium on which a program for performing the same is recorded.

According to the recent status of public interest reports received and processed by public institutions, the number of public interest reports has increased significantly. This rapid increase is due to the increase in citizens' voluntary public interest reporting of violations of road traffic laws using high-definition black boxes, and the simplifying of the reporting process with the introduction of mobile applications such as Smart National Report and Safety Report Center.

However, the personnel in charge of public reporting of traffic violations are busy with other duties and are insufficient to respond to the ever-increasing number of public interest reports.

Therefore, in this field of technology, there is an urgent need for technology that can reduce simple repetitive tasks through automation based on high accuracy.

Republic of Korea Patent Publication No. 10-1845459 Republic of Korea Patent Publication No. 10-2021-0058293

The present invention is intended to solve the above problems and is a first deep learning model that recognizes vehicle objects in public interest report images to quickly respond to public interest reports, which are exponentially increasing due to the development of camera technology and the convenience of public interest reporting. and a method of analyzing images of public interest reports of traffic law violations based on deep learning technology, which outputs whether the vehicle object's course change violation is output by being input to a second deep learning model that recognizes lane objects, and a computer-readable computer-readable program on which the program for performing the method is recorded. The purpose is to obtain recording media.

The technical problems to be achieved by the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description of the present invention.

In order to achieve the above purpose, the deep learning technology-based traffic law violation public interest report image analysis method of the present invention is a public interest report image captured by a black box camera installed in an arbitrary vehicle by at least one processor. Acquisition stage; And by the at least one processor, the public interest report image is input to a first deep learning model that recognizes a vehicle object and a second deep learning model that recognizes a lane object, so that whether the vehicle object has a course change violation is output. Provides a violation output step.

In order to achieve the above object, the present invention provides a computer-readable recording medium recording a program that performs a deep learning technology-based traffic law violation public interest report video analysis method.

As described above, according to the present invention, the public interest report image is input to the first deep learning model that recognizes the vehicle object and the second deep learning model that recognizes the lane object, so that whether the vehicle object's course change violation is output and the camera technology is used. It has the effect of being able to quickly respond to public interest reports, which are exponentially increasing due to development and convenience of public interest reporting.

In addition, the present invention has the effect of increasing the efficiency of public interest reporting work without recruiting insufficient public interest reporting manpower.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the detailed description and claims.

Figure 1 is a flow chart of a video analysis method for public interest reporting of traffic law violations based on deep learning technology of the present invention.
Figure 2 is a diagram showing an arbitrary image in an external environment dataset according to an embodiment of the present invention.
Figure 3 is a diagram showing an arbitrary image in the lane data set according to an embodiment of the present invention.
Figure 4 is a diagram showing an RGB-based image (a), a binarized image (b), and an image (c) displaying lane colors according to brightness and darkness according to an embodiment of the present invention.
Figure 5 is a diagram showing a plurality of vehicle objects and lane objects individually displayed from a first deep learning model and a second deep learning model according to an embodiment of the present invention.
Figure 6 is a diagram showing a case where a course change violation is determined from the violation output step according to an embodiment of the present invention.
Figure 7 is a diagram showing a case where a course change violation is excluded from the violation output step according to an embodiment of the present invention.

The terms used in this specification are general terms that are currently widely used as much as possible while considering the function in the present invention, but this may vary depending on the intention or precedent of a person working in the art, the emergence of new technology, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the relevant invention. Therefore, the terms used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than simply the name of the term.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the technical field to which the present invention pertains. Terms defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in the present application, should not be interpreted in an ideal or excessively formal sense. No.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Figure 1 is a flow chart of a video analysis method for public interest reporting of traffic law violations based on deep learning technology of the present invention. Figure 2 is a diagram showing an arbitrary image in an external environment dataset according to an embodiment of the present invention. Figure 3 is a diagram showing an arbitrary image in a lane data set according to an embodiment of the present invention. Figure 4 is a diagram showing an RGB-based image (a), a binarized image (b), and an image (c) displaying lane colors according to brightness and darkness according to an embodiment of the present invention. Figure 5 is a diagram illustrating a plurality of vehicle objects and lane objects individually displayed from a first deep learning model and a second deep learning model according to an embodiment of the present invention. Figure 6 is a diagram showing a case where a course change violation is determined from the violation output step (S600) according to an embodiment of the present invention. Figure 7 is a diagram showing a case where a course change violation is excluded from the violation output step (S600) according to an embodiment of the present invention.

First, the present invention includes a recording medium 120 that can be read by a computer device 100 that records a program that performs a deep learning technology-based traffic law violation public interest report video analysis method. The recording medium 120 may be, for example, a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, etc. In addition, the deep learning technology-based traffic law violation public interest report video analysis method of the present invention can be implemented by at least one processor 110 in the computer device 100 reading the recording medium 120.

Referring to Figure 1, the deep learning technology-based traffic law violation public interest report image analysis method includes a public interest report image acquisition step in which a public interest report image captured from a black box camera installed in an arbitrary vehicle is acquired by at least one processor 110. (S500) and the at least one processor 110, the public interest report image is input to a first deep learning model that recognizes a vehicle object and a second deep learning model that recognizes a lane object, thereby determining the path of the vehicle object It includes a violation output step (S600) in which change violation is output.

Meanwhile, the present invention provides an external environment image and/or image captured by the at least one processor 110 of the external environment including buildings, running vehicles, parked vehicles, pedestrians, etc. that can be viewed from inside the vehicle and outside the vehicle. An external environment data set acquisition step (S100) acquired as an external environment data set and a first learning step in which the first deep learning model is learned by using the external environment data set by the at least one processor 110. (S200) may be further included.

For example, the external environment dataset may be tens of hours of video and/or millions of images in which the external environment is continuously or sporadically captured, and all captured vehicles may be labeled. 2, the external environment dataset displays a bounding box for all vehicles captured in each video and/or image, and 2 for the four vertices of the bounding box. Dimensional coordinates and each bounding box may be labeled as 'general vehicle' or 'SUV/van.'

Next, the first learning step (S200) is characterized by having the first deep learning model including the YOLOv4 model, which is easy for real-time object recognition with rapid processing speed and improved accuracy. At this time, the first learning step (S200) may further include a labeling format conversion step (S210) in which the labeling format of the external environment dataset is converted to the labeling format of the YOLOv4 model.

According to one embodiment of the present invention, in the labeling format conversion step (S210), if the labeling format of each video and/or image of the external environment dataset is 'ID, two-dimensional coordinates of the bounding box, and label', 'class number , can be converted to the x-coordinate of the center point, y-coordinate of the center point, area, and height. And in the first learning step (S200), the hyperparameter values can be modified as the first deep learning model is learned using the external environment dataset whose labeling format has been converted, and the weight with the highest recognition rate is applied to enable learning. A completed first deep learning model may be created.

At this time, the first deep learning model recognizes only the object called ‘vehicle’ in all videos and/or images in the external environment dataset without distinguishing between vehicle objects. That is, the first learning step (S200) is characterized by having a first deep learning model that further includes a deep sort algorithm to individually recognize vehicle objects.

The deep sort algorithm mentioned in the present invention is based on IoU (Intersection over Union), which is the overlap between bounding boxes, so that vehicle objects can be tracked. Therefore, there is a significant effect of solving the problem of not being able to continuously track the original target ID and assigning a different ID.

Meanwhile, the present invention is a lane dataset in which a lane dataset including a plurality of lane images and/or images in which at least one of the lane, crosswalk, and stop line is photographed while driving is obtained by the at least one processor 110. It may further include an acquisition step (S300) and a second learning step (S400) in which a second deep learning model is learned using the suboptimal dataset by the at least one processor 110.

3, the lane dataset is labeled with 'lane color, lane type, and two-dimensional coordinates of each of the two lane end points' for all lanes captured in each image and/or image. It may be a state. The lane color may be one of white, yellow, and blue, and the lane type may be one of general lanes, crosswalks, and stop lines, and further includes one of solid lines and dotted lines. can do. And the recognized lane is a finite line created by connecting dots. Therefore, the lane includes endpoints at both ends, and can be labeled with two-dimensional coordinates for one endpoint and two-dimensional coordinates for the other endpoint.

Next, the second learning step (S400) is characterized by having a second deep learning model including the Lanenet model in an end-to-end learning method that is easy when the number of features is small and the number of labeled data is large.

Because solid lines, like lanes, are not object recognized through bounding boxes, there are technical limitations in using the YOLO model. To solve this problem, the lane can be learned through a second deep learning model including the Lanenet model of the end to end learning method.

At this time, the second learning step (S400) may include a lane dataset preprocessing step (S410) in which a lane dataset including a plurality of lane images and/or images is preprocessed. According to one embodiment of FIG. 4, the lane data set preprocessing step (S410) is binarized so that lanes can be extracted from each image and/or image as shown in (a) of FIG. 4, and (b) of FIG. 4. Likewise, a binary segmentation image can be created. In addition, in the preprocessing step (S410), an instance segmentation image in which lanes are classified, as shown in (c) of FIG. 4, can be generated so that the extracted lanes can be individually recognized.

And in the second learning step (S400), the second deep learning model can be learned by using preprocessed, that is, lane extraction through binarization and instance segmentation images in which the extracted lanes are individually recognized. Then, in the second learning step (S400), each image and/or image in the preprocessed lane dataset is used to learn the second deep learning model, and the hyperparameter values can be modified, and the weight with the highest recognition rate is used. can be applied to create a second deep learning model that has completed learning.

That is, the violation output step (S600) of the present invention includes a first deep learning model learned from the external environment dataset acquisition step (S100) and the first learning step (S200), and the lane dataset acquisition step ( The second deep learning model learned from S300) and the second learning step (S400) can be used to output whether the vehicle object has a course change violation.

Vehicles are generally equipped with black box cameras at the front and rear for purposes such as checking the circumstances of an accident. The black box camera includes a memory so that images captured for a preset period of time can be stored. And if the owner of the vehicle wishes to report it to a public institution after observing a vehicle that violates traffic laws, the owner's terminal uploads some of the images stored in the memory to platforms, web pages, and applications provided by the public institution through the Internet network. can do.

In other words, the public interest report video may be a video uploaded from the vehicle owner's terminal and may be a video captured from a black box camera installed in the vehicle. In addition, the public interest report video was filmed from inside the car to the outside of the car, and was filmed in a complex manner with other vehicles, lanes, buildings, obstacles, objects, traffic lights, etc. outside the car.

In order to analyze whether or not traffic laws are violated in the public interest report video obtained from the public interest report video acquisition step (S500), the violation output step (S600) may proceed with the following detailed process.

First, the violation output step (S600) includes a vehicle object recognition step (S610) in which multiple vehicle objects are recognized from the first deep learning model without distinguishing between multiple vehicle objects in the public interest report video. Do it as

At this time, in the vehicle object recognition step (S610), multiple vehicle objects may be recognized through the YOLOv4 model in the previously learned first deep learning model, and at this time, the multiple vehicle objects cannot be distinguished from each other.

Next, in the violation output step (S600), the similarity between the plurality of vehicle objects is determined from the first deep learning model to prevent switching between the plurality of vehicle objects, and the vehicle objects are individually tracked. It is characterized in that it further includes an individual vehicle object tracking step (S620).

That is, as mentioned above, in the vehicle object recognition step (S610), multiple vehicle objects in the public interest report video may be recognized, but each vehicle object cannot be individually distinguished, so the initially recognized vehicle object is tracked. Instead, it can be converted to a nearby vehicle object. Therefore, there are technical limitations that make it impossible to continuously track whether traffic laws have been violated or not as the video continues.

In order to solve this problem, the present invention tracks each vehicle object individually for a plurality of vehicle objects through the deep sort algorithm within the previously learned first deep learning model in the vehicle object individual tracking step (S620). It can be. In other words, the deep sort algorithm has the remarkable effect of being able to restore the identity of vehicle objects even after they are occluded for a long time in a continuous image by determining the similarity between the plurality of vehicle objects.

Therefore, in the vehicle object recognition step (S610) and the vehicle object individual tracking step (S620) of the present invention, a plurality of vehicle objects are recognized from the public interest report image, and the multiple vehicle objects are individually recognized, so that each vehicle object It has a notable effect in making analysis possible.

Next, the violation output step (S600) is a lane extraction step (S630) in which the public interest report image is binarized from the second deep learning model and a plurality of lanes are extracted, and the first lane is extracted from the public interest report image from which the lanes are extracted. 2 It is characterized in that it further includes a lane individual display step (S640) in which a plurality of lanes are individually recognized from the deep learning model and then displayed in different colors.

Types of lanes may include general lanes, crosswalks, and stop lines, and general lanes may be solid or dotted lines. And the lanes can be white, yellow and blue.

That is, in the lane extraction step (S630), all lanes in the public interest report image can be recognized through the Lanenet model in the pre-learned second deep learning model, and the lanes in the RGB-based public interest report image are white and lane By converting the outer part to black, a lane can be extracted.

And, in the lane individual display step (S640), the lane may be a lane of the same color or a lane of a different color from the extracted public interest report image. In general, white lanes are used to separate the flow of vehicles moving in the same direction, yellow lanes are used to separate the flow of vehicles moving in the opposite direction, or to indicate road use restrictions and instructions, and blue lanes are used to separate the flow of vehicles moving in the opposite direction. A bus-only lane is a lane that marks the boundary of a road section where only buses or vehicles designated according to the number of passengers can pass.

In other words, distinguishing the color of the lane is very important in determining whether traffic laws are violated. Accordingly, in the lane individual display step (S640), each lane can be displayed by varying the brightness based on the conventional color in the public interest report image from which the lane was extracted through the Lanenet model in the pre-learned second deep learning model. there is.

Therefore, in the lane extraction step (S630) and the individual lane display step (S640) of the present invention, a plurality of lanes are extracted from the public interest report image, the plurality of lanes are individually recognized, and then displayed in different colors according to brightness. , it has the notable effect of enabling analysis of each lane.

Looking at an embodiment of Figure 5, the violation output step (S600) is a pre-trained first deep learning model that recognizes vehicle objects in the public interest report image and a pre-trained second deep learning model that recognizes lane objects. Through this, you can individually check multiple vehicles and lanes filmed in the above public interest report video.

Then, in the violation output step (S600), if the lower vertex of one side touches a solid line and then touches the other vertex of the bounding box containing a random vehicle object in the public interest report video, the arbitrary vehicle object If it is determined that the vehicle object has violated the course change, and the lower vertex of the bounding box touches the solid line and then touches the other vertex again, it is determined that the arbitrary vehicle object has not violated the course change.

Looking at the example of Figure 6, the public interest report video is composed of multiple frames (Frame1, Frame2, Frame3), that is, multiple images. In the violation status output step (S600), it can be confirmed that the lower vertex B of the bounding box containing an arbitrary vehicle object in Frame 1 of the public interest report video touches the white solid line. And in the violation status output step (S600), it can be confirmed that the edge between the bottom vertices A and B of the bounding box in Frame 2 of the public interest report video intersects the white solid line. And in the violation status output step (S600), it can be confirmed that the lower vertex A of the bounding box in Frame 3 of the public interest report video touches the white solid line.

That is, in the violation output step (S600), as the frame changes, the same white solid line touches vertex B and then touches vertex A, so that the vehicle object crosses the solid line from the solid line where lane changes are not possible and changes course to another lane. It may be judged as a violation of the law.

Looking at one embodiment of FIG. 7, in the violation status output step (S600), it can be confirmed that the lower vertex B of the bounding box containing an arbitrary vehicle object in Frame 1 of the public interest report video is in contact with a white solid line. And in the violation status output step (S600), it can be confirmed that the edge between the bottom vertices A and B of the bounding box in Frame 2 of the public interest report video intersects the white solid line. And in the violation output step (S600), it can be confirmed that the bottom vertex B of the bounding box in Frame 3 of the public interest report video touches the white solid line again.

In other words, the vehicle in question briefly moved to the solid white line, but returned to its original driving lane rather than changing course to a completely different lane. Therefore, the violation output step (S600) determines that it did not violate the traffic law for changing course. It can be.

Therefore, according to the present invention, by inputting the public interest report image into a first deep learning model that recognizes a vehicle object and a second deep learning model that recognizes a lane object, whether the vehicle object violates the course change can be output. Accordingly, despite the exponential increase in public interest reports due to the development of camera technology and the convenience of public interest reporting, the present invention quickly and highly accurately determines whether traffic laws are violated for each public interest reporting video, especially whether a change of course is violated. There is a remarkable effect that can be achieved.

In addition, there is a notable effect of eliminating the need to recruit personnel in charge of public interest reporting and increasing work efficiency by reducing repetitive tasks.

Embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented as software, firmware, middleware, or microcode, program code or code segments that perform necessary tasks may be stored in a computer-readable storage medium and executed by one or more processors.

And aspects of the subject matter described herein may be described in the general context of computer-executable instructions, such as program modules or components that are executed by a computer. Typically, program modules or components include routines, programs, objects, and data structures that perform specific tasks or implement specific data types. Aspects of the subject matter described herein may be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including memory storage devices.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in an order different from the described method, and/or the components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or in a different configuration. Appropriate results may be achieved by substitution or substitution of elements or equivalents.

Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

100.. Computer devices
110.. At least one processor
120.. Recording media

Claims (5)

A public interest report image acquisition step in which a public interest report image captured from a black box camera installed in an arbitrary vehicle is acquired by at least one processor; and
By the at least one processor, the public interest report image is input to a first deep learning model that recognizes a vehicle object and a second deep learning model that recognizes a lane object, so that whether the vehicle object's course change violation is output is output. Deep learning technology-based traffic law violation public interest report video analysis method including an output step. According to clause 1,
The violation output step is,
A vehicle object recognition step in which multiple vehicle objects are recognized from the first deep learning model without distinction between multiple vehicle objects in the public interest report image; and
A vehicle object individual tracking step in which similarity between the plurality of vehicle objects is determined from the first deep learning model to prevent switching between the plurality of vehicle objects and the vehicle objects are individually tracked. A deep learning technology-based method of analyzing traffic law violation public interest report videos. According to clause 1,
The violation output step is,
A lane extraction step in which the public interest report image is binarized and a plurality of lanes are extracted from the second deep learning model; and
Traffic laws based on deep learning technology, further comprising a lane individual display step in which a plurality of lanes are individually recognized from the second deep learning model in the public interest report image from which lanes are extracted and then displayed in different colors. Video analysis method for public interest reporting violations. According to clause 1,
The violation output step is,
When one lower vertex of a bounding box containing a random vehicle object in the public interest report video touches a solid line and then touches the other vertex, it is determined that the random vehicle object has violated the course change,
A deep learning technology-based traffic law violation public interest report video analysis method, characterized in that if one lower vertex of the bounding box touches a solid line and then touches one vertex again, it is determined that a random vehicle object has not violated the course change. . A computer-readable recording medium that records a program that performs the deep learning technology-based traffic law violation public interest report video analysis method according to any one of paragraphs 1 to 4.