KR20240079248A - Road danger guidance system and its method - Google Patents

Abstract

The present invention relates to a road hazard guidance system and method, comprising: an information collection vehicle controller that provides real-time vehicle control data on a driving road; A road information provision server that provides road weather information and road driving-related information; And after learning a deep learning model using a reference road risk dataset, matching according to time and space when the real-time vehicle control data is transmitted, and receiving and optimizing the road weather information and road driving-related information, By inputting the optimized object recognition image into the deep learning model to detect road hazards, and a road hazard information server that guides the detected road hazards, sensor data collected based on time and high-precision location are included. By complementing the shortcomings of image classification through data, not only can road hazards be effectively detected, but accidents due to road hazards can be prevented in advance.

Description

Road hazard guidance system and method {ROAD DANGER GUIDANCE SYSTEM AND ITS METHOD}

The present invention trains a deep learning model using a standard road risk data set from a road risk information server, then matches it according to time and space when real-time vehicle control data is transmitted from an information collection vehicle controller, and transmits it from a road information provision server. After receiving and optimizing road weather information and road driving-related information, the optimized object recognition image is input into a deep learning model to detect road hazards, and by guiding the detected road hazards, the data collected based on time It relates to a road hazard guidance system and method that can not only effectively detect road hazards by complementing the shortcomings of image classification through sensor data and high-precision location data, but also prevent accidents due to road hazards in advance. .

As is well known, as the computer vision field is growing rapidly, the demand for video control in fields such as construction, maritime, public safety, disaster, and safety is increasing.

In relation to this, research is being actively conducted to distinguish and analyze objects in images using computer vision technology. For example, cameras can be used to detect signs and signs on the road, and can be used to update the relevant road, or to detect cracks and potholes. It is used to detect etc.

However, when controlling using CCTV, it is difficult to collect data in areas other than the area where CCTV is installed and filmed, and it is even more difficult to control in areas where CCTV is difficult to install, such as the coast.

In order to solve the above-mentioned problems, there are increasing attempts to build terminals and collection systems that can not only capture images but also collect sensor data like a vehicle's black box, but these studies only distinguish images through image classification. There is a problem in that image classification efficiency is reduced due to overfitting to specific images.

Accordingly, there is a need to develop a system to compensate for the shortcomings of image classification by collecting more accurate data through sensor data and high-precision location data collected based on time rather than image classification.

1. Korean Patent No. 10-1850286 (registered on April 13, 2018)

The present invention trains a deep learning model using a standard road risk data set from a road risk information server, then matches it according to time and space when real-time vehicle control data is transmitted from an information collection vehicle controller, and transmits it from a road information provision server. After receiving and optimizing road weather information and road driving-related information, the optimized object recognition image is input into a deep learning model to detect road hazards, and by guiding the detected road hazards, the data collected based on time We aim to provide a road hazard guidance system and method that can not only effectively detect road hazards by complementing the shortcomings of image classification through sensor data and high-precision location data, but also prevent accidents caused by road hazards. do.

The purposes of the embodiments of the present invention are not limited to the purposes mentioned above, and other purposes not mentioned will be clearly understood by those skilled in the art from the description below. .

According to one aspect of the present invention, an information collection vehicle controller that provides real-time vehicle control data on a driving road; A road information provision server that provides road weather information and road driving-related information; And after learning a deep learning model using a reference road risk dataset, matching according to time and space when the real-time vehicle control data is transmitted, and receiving and optimizing the road weather information and road driving-related information, A road hazard guidance system that includes a road hazard information server that inputs the optimized object recognition image into the deep learning model to detect road hazard elements and guides the detected road hazard elements.

In addition, according to one aspect of the present invention, the information collection vehicle controller is a road hazard guidance system that transmits the real-time vehicle control data, including real-time image data and real-time sensor data, to the road hazard information server in a collected time series. This can be provided.

Additionally, according to one aspect of the present invention, the road hazard guidance server may be provided with a road hazard guidance system that trains the deep learning model including a darkflow-based YOLOv3 detector model.

In addition, according to one aspect of the present invention, after calculating the image quality of the real-time image data, the road hazard information server calculates the image quality of the real-time image data according to the existing collected data, the real-time sensor data, road weather information, and road driving-related information. A road hazard guidance system optimized with images for object recognition can be provided.

Additionally, according to one aspect of the present invention, the road risk information server may be provided with a road risk information system that references the existing collected data including shoulder vehicle statistics for each day of the week and time zone.

In addition, according to one aspect of the present invention, a road hazard guidance system may be provided in which the road hazard factors include at least one selected from road damage, road icing, road water, road encroaching bushes, and road falling objects.

According to another aspect of the present invention, providing real-time vehicle control data on a driving road from an information collection vehicle controller; Providing road weather information and road driving-related information from a road information provision server; Learning a deep learning model using a standard road risk data set in a road risk information server; When the real-time vehicle control data is transmitted from the road hazard information server, matching according to time and space; Receiving and optimizing the road weather information and road driving-related information from the road risk information server; Detecting road hazard factors by inputting an image for object recognition optimized from the road hazard information server into the deep learning model; A road hazard guidance method including a step of guiding the detected road hazard elements may be provided.

In addition, according to another aspect of the present invention, the step of providing the real-time vehicle control data includes collecting the real-time vehicle control data including real-time image data and real-time sensor data from the information collection vehicle controller. A road hazard information method transmitted to a risk information server may be provided.

In addition, according to another aspect of the present invention, the step of learning the deep learning model includes training the deep learning model including a darkflow-based YOLOv3 detector model in the road risk information server. Risk guidance methods may be provided.

In addition, according to another aspect of the present invention, the step of receiving and optimizing the road weather information and road driving-related information includes calculating the image quality of the real-time image data in the road risk information server, combining existing collected data, A road hazard guidance method that optimizes the object recognition image according to the real-time sensor data, road weather information, and road driving-related information may be provided.

In addition, according to another aspect of the present invention, the step of receiving and optimizing the road weather information and road driving-related information refers to the existing collected data including shoulder vehicle statistics by day and time in the road risk information server. Road hazard guidance methods may be provided.

In addition, according to another aspect of the present invention, a road hazard guidance method may be provided wherein the road hazard factors include at least one selected from road damage, road icing, road water, road encroaching bushes, and road falling objects.

The present invention trains a deep learning model using a standard road risk data set from a road risk information server, then matches it according to time and space when real-time vehicle control data is transmitted from an information collection vehicle controller, and transmits it from a road information provision server. After receiving and optimizing road weather information and road driving-related information, the optimized object recognition image is input into a deep learning model to detect road hazards, and by guiding the detected road hazards, the data collected based on time By complementing the shortcomings of image classification through sensor data and high-precision location data, not only can road hazards be effectively detected, but accidents due to road hazards can be prevented in advance.

In addition, the present invention utilizes vehicle black box images, GNSS/IR, and various communication technologies to update road hazard situations in real time and effectively prevent road hazard accidents for workers in the transportation industry.

In particular, from a technical perspective, the present invention provides a foundation for building a precise road map through data collection, allows continuous data collection using vehicles operating on fixed routes, and combines the collected data with the constructed precision road map. Not only can changes be updated by comparison, but the inspection efficiency of the inspection system that frequently checks infrastructure on the road can also be improved. In the future, a precision map construction system through continuous data collection and precise positioning can be developed for use in autonomous vehicles. A foundation can be laid.

In addition, from an economic and industrial perspective, the present invention applies the 4th industrial revolution technology to the traditional public transportation area and implements a terminal for real-time acquisition of business vehicle operation information, vehicle interior and exterior environment information, and road information, providing Korea's first real-time large-capacity commercial vehicle operation. It is a source technology that can lay the foundation for the development of a data processing and analysis system, implement a precise safe driving index calculation process through artificial intelligence (AI) comparative analysis of large-capacity control data, and develop a data collection device tailored to consumers on the road. It can be used as:

Meanwhile, from a social perspective, the present invention can secure the public's right to public movement safety in the form of risk factors on the road through real-time dynamic control, and can provide optimal artificial intelligence (AI) based on the driving patterns of exemplary drivers through continuous data accumulation. ) Road safety accidents can be prevented in advance by deriving driving patterns, improving the driving habits of transport workers and drivers, and detecting and taking action on dangerous driving behavior of drivers through real-time driving situation control.

1 is a block diagram of a road hazard guidance system according to an embodiment of the present invention,
2 to 5 are diagrams for explaining a road hazard guidance system according to an embodiment of the present invention;
Figure 6 is a flow chart showing a road hazard guidance method according to another embodiment of the present invention.

Advantages and features of the embodiments of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the disclosure of the present invention is complete and to be understood by those skilled in the art in the technical field to which the present invention pertains. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

In describing embodiments of the present invention, if a detailed description of a known function or configuration is judged to unnecessarily obscure the gist of the present invention, the detailed description will be omitted. The terms described below are defined in consideration of functions in the embodiments of the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the contents throughout this specification.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

Figure 1 is a block diagram of a road hazard guidance system according to an embodiment of the present invention, and Figures 2 to 5 are diagrams for explaining the road hazard guidance system according to an embodiment of the present invention.

Referring to Figures 1 to 5, the road risk guidance system according to an embodiment of the present invention includes an information collection vehicle controller 100, a road information provision server 200, a road risk information server 300, and a communication network 400. It may include etc.

The information collection vehicle controller 100 is a device that provides real-time vehicle control data on the road being driven,

This information collection vehicle controller 100 can transmit real-time vehicle control data, including real-time image data and real-time sensor data, to the road hazard information server 300 in a collected time series.

For example, the information collection vehicle controller 100 may be mounted on an information collection vehicle as shown in FIG. 2. The information collection vehicle includes a video camera (e.g., 6-channel CAM, HD2) that collects real-time video data. channel/D14 channel) can be provided and can include front video, driver video, left and right side video, rear video, etc.

Additionally, the information collection vehicle can collect high-precision location information through RTK-GNSS (Real Time Kinematic-Global Navigation Satellite System), drive record information from M-DTG (Korea Transportation Safety Authority), and IMU. Various sensor information can be collected through (Inertial Measurement Unit) sensors, carbon dioxide detection sensors, IR sensors, etc.

The information collection vehicle controller 100 as described above includes a darkflow-based YOLOv3 detector to detect changes in road conditions from real-time image data, and an OpenCL (open universal parallel computing framework)-based It can perform acceleration and can be equipped with the BLAZE AI Kit, a PCB with an MCU core that can be mounted on a vehicle.

The road information provision server 200 is a server that provides road weather information and road driving-related information, and may include servers operated by the Korea Meteorological Administration, road control center, etc.

Here, road weather information may include, for example, weather for each road, and road driving-related information may include, for example, traffic volume for each road.

The road risk information server 300 is a server that detects road risk factors using a deep learning model and guides road risk. After learning a deep learning model using a standard road risk data set, real-time vehicle control data is provided. When transmitted, it is matched according to time and space, and after receiving and optimizing road weather information and road driving-related information, the optimized object recognition image is input into a deep learning model to detect road hazards, and road hazards are detected. If detected, it can be reflected in the existing road map and updated to a new road map, providing guidance on road hazards.

Here, the road risk factor may include at least one selected from, for example, road damage, road ice, road water, road invading bushes, and road falling objects.

This road risk information server 300 can train the deep learning model including the darkflow-based YOLOv3 detector model.

For example, when explaining the darkflow-based YOLOv3 detector model, tx, ty, tw, and th are predicted in the bounding box prediction in the image, and the predicted tx, ty are ranged through a sigmoid function. It is limited to 0??1, and is added to cx and xy (coordinates of the grid cell containing the object) to become the bounding box, and tw and th are used as exponents to create pw and ph (width and height of the previous bounding box). Stable learning can be achieved by limiting the range in which the bounding box is updated at one time using a multiplication-type normalization technique.

In addition, each box predicts the class of the bounding box, using an independent logistic classifier using binary cross-entropy loss, which helps learning more complex datasets.

In this YOLOv3 detector model, three different scales are used: several convolutional layers are added to the backbone network, darknet-53, and the added convolutional layer is a feature pyramid. It is used to generate, upsampling is performed twice on the final feature map, and three scales are used, so the final feature map and the previous two feature maps are integrated with the upsampled feature map. It can be done in this way.

The road risk information server 300 as described above is a road risk data set (e.g., a road previously built from the AI-Hub (AI integrated platform operated by the Korea Intelligence Information Society Promotion Agency) as shown in FIG. 3. After performing labeling on risk data (including image data such as road cracks and puddles and high-resolution data among self-obtained images), road hazard elements (e.g., road hazards) are detected within the image using the YOLOv3 detector model. It can be trained to detect at least one of road damage, road ice, road water curtain, road encroaching bushes, and road falling objects).

In addition, as shown in FIGS. 3 and 4, the road risk information server 300 matches the real-time vehicle control data according to time and space when real-time vehicle control data is transmitted from the information collection vehicle controller 100, and the road information provision server 200 After receiving and optimizing road weather information and road driving-related information from ), road hazards can be detected by inputting the optimized object recognition image into a deep learning model.

Here, the road hazard information server 300 can use the collected time to match real-time image data and real-time sensor data through the location and time, and road weather information related to road driving in addition to the collected real-time vehicle control data. and road driving-related information, and utilizes this information and the original source of collected real-time video data (i.e., within the original collected video data, location accuracy is high, shaking in the Z-axis is low, external brightness is appropriate, and vehicle speed is By selecting and utilizing relatively slow, high-quality images, the image can be optimized through image compression, removal of frames that cannot recognize objects, etc., and road hazard elements can be detected in the image for object recognition through the learned YOLOv3 detector model. there is.

Meanwhile, after calculating the image quality of real-time image data, the road hazard information server 300 can optimize the image for object recognition according to existing collected data, real-time sensor data, road weather information, and road driving-related information. , Existing collected data may include shoulder vehicle statistics by day of the week and time.

For example, the road hazard information server 300 can calculate image quality using real-time image data such as location accuracy, Z-axis shaking, external brightness, vehicle speed, video length, etc., and refer to the calculated image quality. The optimal image for object recognition was selected based on existing collected data including road shoulder vehicle statistics by day and time, real-time sensor data, road weather information including road-specific weather, traffic volume by road, and road driving-related information. Later, road hazards can be detected by inputting them into the learned YOLOv3 detector model.

For example, as shown in Figure 5, from the original image, 1) the corresponding road hazards can be detected through the analysis image of road damage, and 2) the corresponding road hazards for road ice/water film can be detected. 3) it is possible to detect the relevant road hazards for road-invasive bushes, and 4) it is possible to detect the relevant road hazards for fallen objects on the road.

The road risk information server 300 as described above can continuously perform the process described above when the road risk factor is not detected, and when the road risk factor is detected, the road risk factor exists along with the relevant road information. An alarm (for example, an alarm message, etc.) notifying that the vehicle is in operation can be transmitted to the registered vehicle control terminal.

The communication network 400 is a network that supports wired communication networks and wireless communication methods, and may include, for example, a wired communication network using an Internet protocol such as TCP/IP, and may include, for example, a base station, a mobile switching center, a home location register, etc. In addition to mobile communication networks including, wireless communication networks such as RF, Zigbee, Bluetooth, Wi-Fi, Wibro, NFC (near field communication), and BLE (bluetooth low energy) It can be included, and through this, a smooth wired and wireless communication environment can be provided between the information collection vehicle controller 100, the road information provision server 200, and the road risk information server 300.

Therefore, according to one embodiment of the present invention, after learning a deep learning model using a reference road risk data set in the road risk information server, when real-time vehicle control data is transmitted from the information collection vehicle controller according to time and space After matching, receiving and optimizing road weather information and road driving-related information from the road information provision server, the optimized object recognition image is input into a deep learning model to detect road hazards and guide the detected road hazards. By doing so, it is possible to not only effectively detect road hazards by complementing the shortcomings of image classification through sensor data and high-precision location data collected based on time, but also prevent accidents caused by road hazards in advance.

Figure 6 is a flow chart showing a road hazard guidance method according to another embodiment of the present invention.

Referring to FIG. 6, the information collection vehicle controller 100 can provide real-time vehicle control data on the road on which the vehicle is driven (step 610).

In the step 610 of providing the real-time vehicle control data, real-time vehicle control data including real-time image data and real-time sensor data is collected from the information collection vehicle controller 100 and sent to the road hazard information server 300 in a time-serial manner. Can be transmitted.

Additionally, the road information provision server 200 can provide road weather information and road driving-related information (step 620).

Here, road weather information may include, for example, weather for each road, and road driving-related information may include, for example, traffic volume for each road.

Meanwhile, the road risk information server 300 can learn a deep learning model using the standard road risk data set (step 630).

In the step 630 of training the deep learning model, the road risk information server 300 can train a deep learning model including a dark flow-based YOLOv3 detector model.

For example, in the road risk information server 300, the standard road risk data set (e.g., road risk data previously constructed from AI-Hub (AI integrated platform operated by the Korea Intelligence Information Society Promotion Agency) includes road cracks, After performing labeling on image data such as puddles and high-resolution data among self-obtained images, the YOLOv3 detector model is used to detect road hazards (e.g., road damage, road icing, road hazards) in the image. It can be trained to detect at least one of water curtains, road encroaching bushes, and road falling objects).

Next, when real-time vehicle control data is transmitted from the information collection vehicle controller 100 to the road hazard information server 300, it can be matched according to time and space (step 640).

Here, the road risk information server 300 can use the collected time to match real-time image data and real-time sensor data through the location and time.

Additionally, the road risk information server 300 can receive and optimize road weather information and road driving-related information transmitted from the road information provision server 200 (step 650).

In the step 650 of receiving and optimizing the road weather information and road driving-related information, the road risk information server 300 calculates the image quality of the real-time image data, and then collects the existing collected data, real-time sensor data, and road weather. It can be optimized as an image for object recognition according to information and road driving-related information, and the road risk information server 300 can refer to existing collected data including shoulder vehicle statistics by day and time.

For example, the road hazard information server 300 utilizes road weather information and road driving-related information related to road driving in addition to the collected real-time vehicle control data, and the original collected real-time video data (i.e., the original collected video data). By selecting and utilizing high-quality images with high positional accuracy, low Z-axis shaking, appropriate external brightness, and relatively slow vehicle speeds, the images can be optimized through image compression, removal of frames where object recognition is impossible, etc. , video quality can be calculated using real-time video data such as location accuracy, Z-axis shake, external brightness, vehicle speed, video length, etc., and includes road shoulder vehicle statistics by day and time while referring to the calculated video quality. The optimal image for object recognition can be selected according to existing collected data, real-time sensor data, road weather information including road-specific weather and road-specific traffic volume, and road driving-related information.

Meanwhile, road risk factors can be detected by inputting the optimized object recognition image from the road risk information server 300 into a deep learning model (step 660).

Here, the road risk factor may include at least one selected from, for example, road damage, road ice, road water, road invading bushes, and road falling objects.

For example, the road hazard information server 300 can detect road hazard elements by inputting the optimal object recognition image selected in step 650 into the learned YOLOv3 detector model.

Next, if a road hazard is detected in the road hazard information server 300, the road hazard may be guided (step 670).

The road risk information server 300 as described above can continuously perform the process described above when the road risk factor is not detected, and when the road risk factor is detected, the road risk factor exists along with the relevant road information. An alarm (for example, an alarm message, etc.) notifying that the vehicle is in operation can be transmitted to the registered vehicle control terminal.

Therefore, according to another embodiment of the present invention, after learning a deep learning model using the standard road risk data set in the road risk information server, when real-time vehicle control data is transmitted from the information collection vehicle controller according to time and space After matching, receiving and optimizing road weather information and road driving-related information from the road information provision server, the optimized object recognition image is input into a deep learning model to detect road hazards and guide the detected road hazards. By doing so, it is possible to not only effectively detect road hazards by complementing the shortcomings of image classification through sensor data and high-precision location data collected based on time, but also prevent accidents caused by road hazards in advance.

In the above description, various embodiments of the present invention have been presented and explained, but the present invention is not necessarily limited thereto, and those skilled in the art will understand various embodiments without departing from the technical spirit of the present invention. It will be easy to see that branch substitutions, transformations, and changes are possible.

100: Information collection vehicle controller
200: Road information provision server
300: Road hazard information server
400: communication network

Claims (12)

An information collection vehicle controller that provides real-time vehicle control data on the road being driven;
A road information provision server that provides road weather information and road driving-related information; and
After learning a deep learning model using a reference road risk dataset, when the real-time vehicle control data is transmitted, it is matched according to time and space, and the road weather information and road driving-related information are received and optimized. A road hazard information server that detects road hazards by inputting the image for object recognition into the deep learning model and guides the detected road hazards;
Road hazard guidance system including.
In claim 1,
The information collection vehicle controller,
The real-time vehicle control data, including real-time video data and real-time sensor data, is collected and transmitted to the road hazard information server in a time-serial manner.
Road hazard guidance system.
In claim 2,
The road hazard information server,
Learning the deep learning model including the darkflow-based YOLOv3 detector model
Road hazard guidance system.
In claim 3,
The road hazard information server,
After calculating the image quality of the real-time image data, optimizing the image for object recognition according to the existing collected data, the real-time sensor data, road weather information, and road driving-related information.
Road hazard guidance system.
In claim 4,
The road hazard information server,
Referring to the existing collected data above, including shoulder vehicle statistics by day and time of day,
Road hazard guidance system.
The method according to any one of claims 1 to 5,
The above road hazard factors are:
Containing at least one selected from road damage, road ice, road water curtain, road invasion bushes, and road falling objects
Road hazard guidance system.
Providing real-time vehicle control data on the driving road from an information collection vehicle controller;
Providing road weather information and road driving-related information from a road information provision server;
Learning a deep learning model using a standard road risk data set in a road risk information server;
When the real-time vehicle control data is transmitted from the road hazard information server, matching according to time and space;
Receiving and optimizing the road weather information and road driving-related information from the road risk information server;
Detecting road hazard factors by inputting an image for object recognition optimized from the road hazard information server into the deep learning model; and
Guiding the detected road hazards;
Road hazard guidance method including.
In claim 7,
The step of providing the real-time vehicle control data is,
The information collection vehicle controller transmits the real-time vehicle control data, including real-time image data and real-time sensor data, to the road hazard information server in a time-series manner.
How to guide road hazards.
In claim 8,
The step of training the deep learning model is,
Learning the deep learning model including the darkflow-based YOLOv3 detector model in the road risk information server.
How to guide road hazards.
In claim 9,
The step of receiving and optimizing the road weather information and road driving-related information is,
After calculating the image quality of the real-time image data in the road hazard information server, optimizing the image for object recognition according to the existing collected data, the real-time sensor data, road weather information, and road driving-related information.
How to guide road hazards.
In claim 10,
The step of receiving and optimizing the road weather information and road driving-related information is,
The road risk information server refers to the existing collected data including shoulder vehicle statistics by day and time.
How to guide road hazards.
The method of any one of claims 7 to 11,
The above road hazard factors are:
Containing at least one selected from road damage, road ice, road water, road invasion bushes, and road falling objects
How to guide road hazards.