KR102105007B1 - Edge-cloud system for collecting and providing data of connected car - Google Patents

Abstract

An edge-cloud system for collecting connected car data and providing unexpected information is provided. The edge computing device analyzes vehicle sensing data received from the target vehicle and surrounding operation information related to the surroundings of the target vehicle received from the cloud computing system to determine whether there is an unexpected situation indication, and determines that there is an unexpected situation indication. When it is determined, the data based on the determination of the sudden situation indication is extracted from the vehicle sensing data and the surrounding driving information to generate the sudden situation indication data for the target vehicle, and the cloud computing system suddenly receives the sudden situation indication data from the edge computing device. Predicting the driving situation information of the target vehicle based on the situation indication data, and in-depth analysis of the sudden situation signs by linking the driving situation information of the target vehicle with the driving situation information of the surrounding vehicles, and then an unexpected situation including an in-depth analysis result. To generate driving information for Sent to the computing device and an edge, the edge computing device may transmit the irregularity vehicle management information to the target vehicle.

Description

Edge-cloud system for collecting and providing data of connected car

The present invention relates to an edge-cloud system for collecting connected car data and providing sudden situation information, and more specifically, it is possible to provide safe situation information by providing unexpected situation information that may occur during a vehicle driving route. It relates to an edge-cloud system for collecting connected car data and providing unexpected information.

The recently released vehicle provides the driver with a function to inform various information related to the vehicle, such as speed and engine speed, and the life of the engine oil. For example, it is possible for the driver to check information from the On-Board Diagnostics (OBD) with a smartphone. OBD is a diagnostic system made by manufacturers for self-diagnosis of vehicles.

In addition, recent vehicles provide a collision detection function for safe driving, so that the driver can prevent a traffic accident caused by a collision in advance.

However, various data generated in the existing vehicle is provided only to the driver, and the data from the OBD is limited to being provided in real time since it mainly provides history data to diagnose when a vehicle has a defect.

Therefore, there is a limit for the driver to more objectively and intuitively grasp the state of the vehicle or to determine signs related to an unexpected situation that may occur while driving.

Domestic Published Patent No. 10-2018-0063789 (2018.06.12)

The technical problem to be achieved by the present invention in order to solve the above-mentioned problems is to share various data sensed from a vehicle in real time in an edge computing system and the cloud to determine signs of an unexpected situation related to a vehicle or driving, and to determine the unexpected situation. The present invention is to provide an edge-cloud system for collecting connected car data and providing unexpected information to provide drivers with driving information about the vehicle so that the driver can more quickly and accurately respond to vehicle conditions or emergencies.

The problems of the present invention are not limited to those mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

As a means for solving the above-described technical problem, according to an embodiment of the present invention, the edge-cloud system for collecting connected car data and providing unexpected situation information, vehicle sensing data received from a target vehicle, and a cloud computing system By analyzing surrounding driving information related to the surroundings in which the target vehicle is being operated from, it is determined whether there is an unexpected situation sign, and when it is determined that there is an unexpected situation sign, the vehicle sensing data determines the basis data for determining the unexpected situation sign And an edge computing device that extracts from the surrounding driving information and generates unexpected situation indication data for the target vehicle. And when the sudden situation indication data is received from the edge computing device, predicts the driving situation information of the target vehicle based on the received sudden situation indication data, and displays the driving situation information of the target vehicle and the driving situation information of surrounding vehicles. Cloud computing that generates, in conjunction with the in-depth analysis of the sudden situation indications in connection, operation information (hereinafter, referred to as 'explosion situation operation information') for the unexpected situation including the in-depth analysis result and transmits it to the edge computing device. System; and the edge computing device may transmit sudden situation driving information received from the cloud computing system to the target vehicle.

The edge computing device analyzes vehicle sensing data of the target vehicle and the surrounding driving information to determine whether there is an unexpected situation sign including at least one of whether an accident has occurred, whether a construction is in progress, bad weather information, or an obstacle, and The vehicle sensing data sensed by the target vehicle and surrounding vehicles includes at least one of image data, speed data, acceleration data, rapid deceleration data, collision detection data, and laser transmission / reception data, and the surrounding operation information is obtained from the surrounding vehicles. It may include sensed vehicle sensing data, weather data provided by public institutions, and section information on the road.

The edge computing device, when it is determined that there is an unexpected situation indication, the at least one of location information related to the unexpected situation indication, image information, acceleration / deceleration information, collision detection information, and meteorological state information analyzed from laser transmission / reception data It can be extracted from the vehicle sensing data and the surrounding driving information as judgment basis data.

The cloud computing system, when the sudden situation indication data is received, generates driving information of the target vehicle including an object located in at least one of front and rear of the target vehicle and driving situation information of the target vehicle, The occurrence information of the unexpected situation may be in-depth analysis by linking the generated driving information of the target vehicle with the generated driving information for surrounding vehicles operating in the vicinity of the target vehicle.

The cloud computing system determines whether an accident occurs, whether construction is in progress, bad weather information, and an obstacle by linking and analyzing the result of in-depth analysis of the unexpected condition signs and external public data, and an unexpected situation including the determined result Driving information may be generated and transmitted to the edge computing device.

According to the present invention, by sharing a variety of data sensed from the vehicle in real time on the edge computing system and the cloud to determine signs of an unexpected situation related to the vehicle or driving, and providing the driver with the driving information for the determined unexpected situation, the driver can see more It can quickly and accurately respond to vehicle conditions or sudden situations.

In addition, according to the present invention, in the edge computing system and the cloud system, the state of the car is predicted based on the signs of an unexpected situation (accident, weather, etc.) analyzed in the vehicle, and continuously in an unexpected situation in connection with the state with surrounding cars. By performing the correction work for the vehicle, it is possible to generate and provide data necessary for safe operation of the vehicle.

In addition, according to the present invention, when an accident occurs in a vehicle after a sign of an unexpected situation is predicted, information necessary for handling an accident (location, type of accident, etc.) in the cloud may be provided to a police agency server, a hospital, an insurance company, and the like.

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a diagram illustrating an edge-cloud system for collecting connected car data and providing unexpected information according to an embodiment of the present invention;
FIG. 2 is a block diagram showing in detail the edge computing device and the cloud computing system shown in FIG. 1;
3 is a schematic diagram of an image recognition-based front vehicle detection algorithm,
4 is a view showing a general Tenser Flow structure,
5 is a view showing a general CNN structure,
6 is an exemplary view of ROI pooling,
7 is a diagram showing the structure of an object detection algorithm of MultiNet,
8 is a view showing an example of a center line identification result using the HSV color model and a vehicle driving direction determination result using the center line identification algorithm;
9 is a flow chart for explaining a method for providing unexpected information of an edge-cloud system according to an embodiment of the present invention, and
10 is a flowchart illustrating steps S950 to S970 in detail.

The above objects, other objects, features and advantages of the present invention will be readily understood through the following preferred embodiments related to the accompanying drawings. However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Rather, the embodiments introduced herein are provided to ensure that the disclosed contents are thorough and complete and that the spirit of the present invention is sufficiently conveyed to those skilled in the art.

Also, when it is said that the first element (or component) is operated or executed on the second element (or component), the first element (or component) is the second element (or component). It should be understood that the operation or execution is performed or executed in an environment in which the operation or execution is performed, or through an interaction directly or indirectly with the second element (or component).

When an element, component, device, or system is mentioned to include a component consisting of a program or software, the element, component, device, or system is executed or operated by the program or software, even if not explicitly stated. It should be understood that it includes hardware (for example, memory, CPU, etc.) necessary to do so, or other programs or software (for example, drivers required to drive an operating system or hardware).

In addition, it should be understood that an element (or component) may be implemented in software, hardware, or in any form of software and hardware, unless otherwise stated in the implementation of any element (or component).

In addition, the terminology used herein is for describing the embodiments and is not intended to limit the present invention. In this specification, the singular form also includes the plural form unless otherwise specified in the phrase. As used herein, 'comprises' and / or 'comprising' does not exclude the presence or addition of one or more other components.

Hereinafter, the present invention will be described in detail with reference to the drawings. In describing the following specific embodiments, various specific contents have been prepared to more specifically describe and understand the invention. However, a reader who has knowledge in this field to understand the present invention can recognize that it can be used without these various specific contents.

It should be noted that, in some cases, parts that are commonly known in describing the invention and that are not significantly related to the invention are not described in order to prevent chaos from coming into account in explaining the present invention for no apparent reason.

Each configuration of the edge-cloud system for providing connected car data collection and unexpected situation information shown in FIG. 1 is functionally and logically separated, and each configuration is divided into separate physical devices. The average expert in the technical field of the present invention can easily infer that it does not mean that it is or is written in a separate code.

In addition, in the present specification, DB may mean a functional structural combination of software and hardware that stores information corresponding to each DB. The DB may be implemented as at least one table, and may further include a separate Database Management System (DBMS) for retrieving, storing, and managing information stored in the DB. In addition, it may be implemented in various ways such as a linked list, a tree, a relational DB, and includes all data storage media and data structures capable of storing information corresponding to the DB. Edge computing is a system structure that primarily processes data before going to the cloud by constructing a computing system for processing data at the edge portion of the network, and is also called fog computing or cloudlet. There are a number of edge computing devices in the edge computing system, each of the edge computing devices 100 to 300 processes data for individual vehicles, and the cloud computing system 200 collects computing power problems and collections of multiple edge computing devices. The statistics of the data and the macros can then be automatically exposed to users or vehicles in a similar area for consumption.

The plurality of programs for the edge-cloud system for providing the unexpected information may be installed in a predetermined data processing apparatus to implement the technical idea of the present invention.

1 is a diagram illustrating an edge-cloud system for collecting connected car data and providing unexpected information according to an embodiment of the present invention.

Referring to FIG. 1, an edge-cloud system for collecting connected car data and providing unexpected information may include a plurality of edge computing devices 100, 200, and 300 and a cloud computing system 600.

First, the target vehicle 10 may receive vehicle sensing data sensing the state of the target vehicle 10 from a plurality of sensors mounted inside and outside the target vehicle 10 and transmit the received vehicle sensing data to the edge computing device 100. A communication module may be provided in each sensor so that each sensor directly transmits sensing data to the edge computing device 100.

'Vehicle sensing data' includes data of all types (vehicle status, performance, failure status, location, driving information, etc.) that can be sensed by the vehicle, for example, front or rear image data of a vehicle captured by a black box, Velocity data by speed sensor, acceleration data by acceleration sensor, sudden deceleration data, collision detection data by collision detection sensor, vehicle location information obtained from GPS, OBD data obtained from On-Board Diagnostics (OBD), laser transmission / reception It may include at least one of data and data by a rider, identification information of the target vehicle 10 and driving route information or destination information of the target vehicle 10.

Among the surrounding vehicles located around the target vehicle 10, the surrounding vehicles 20 and 30 equipped with the peripheral edge computing devices 200 and 300 also collect vehicle sensing data in the same manner as the target vehicle 10, thereby computing the surrounding edge, respectively. It transmits to the devices (200, 300).

3 ^RD PART Gateway servers (not shown) among public institution servers, such as the Meteorological Agency server 400 and the road construction server 500, are roads such as external public data, that is, weather data or section information in progress on the road. Related information.

The edge computing device 100 is mounted on the target vehicle 10 to construct an edge computing system. In an embodiment of the present invention, the edge computing device 100 analyzes vehicle sensing data received from the target vehicle 10 and surrounding driving information related to surroundings in which the target vehicle received from the cloud computing system 600 is operating. If it is determined whether there is an unexpected situation sign, and if it is determined that there is an unexpected situation sign, data for determining the unexpected situation sign may be extracted from vehicle sensing data and surrounding driving information to generate unexpected situation sign data for the target vehicle. .

The cloud computing system 600 may receive and store vehicle sensing data from the target vehicle 10 and surrounding vehicles 20 and 30. In addition, the cloud computing system 600 is a vehicle located within a predetermined radius Nkm based on the driving position of the target vehicle 10, or a driving route available to a destination based on the driving position of the target vehicle 10 A vehicle located in the front Nkm may be selected as a surrounding vehicle of the target vehicle 10, and surrounding driving information including vehicle sensing data of the selected surrounding vehicles 20 and 30 may be transmitted to the edge computing device 100. . For example, the cloud computing system 600 selects vehicles of similar locations on the GPS based on the standard node link as surrounding vehicles.

 'Ambient operation information' refers to vehicle sensing data sensed by sensors of surrounding vehicles 20 and 30, weather information of an area (or surrounding) in which the target vehicle 10 is operating, construction of a running road or a planned road. It may include all information that affects driving, such as section information.

In addition, the cloud computing system 600 predicts driving situation information of the target vehicle 10 based on the received unexpected situation indication data when the unexpected situation indication data is received from the edge computing device 100, and the target vehicle 10 ) After in-depth analysis of the unexpected situation signs determined by linking the driving situation information of the surrounding vehicles with the driving situation information of the surrounding vehicles 20 and 30, the driving information for the sudden situation including the deeply analyzed results (hereinafter, , 'Suspension situation operation information') may be generated and transmitted to the edge computing device 100.

The edge computing device 100 transmits the unexpected driving information received from the cloud computing system 600 to the target vehicle 10.

The target vehicle 10 displays the received sudden situation driving information to the outside so that the driver can recognize it, whereby the driver can recognize the location of the sudden situation indication in the driving route and focus on safe driving.

Hereinafter, embodiments of the present invention will be described in more detail with reference to FIGS. 2 to 8.

2 is a block diagram illustrating in detail the edge computing device 100 and the cloud computing system 600 shown in FIG. 1.

Referring to FIG. 2, the edge computing device 100 according to an embodiment of the present invention includes a bus 110, an edge communication unit 120, a data collection unit 130, an edge memory 140, and an edge processor 150. Includes.

The bus 110 connects the edge communication unit 120, the data collection unit 130, the edge memory 140, and the edge processor 150 to each other, and transmits various signals such as control messages, status information, and / or data It may include a circuit.

The edge communication unit 120 includes an interface circuit for communication between the target vehicle 10 and the cloud computing system 600. For example, the edge communication unit 120 receives the vehicle sensing data from the target vehicle 10, transmits the received vehicle sensing data to the cloud computing system 600, and transmits surrounding driving information from the cloud computing system 600. I can receive it.

The data collection unit 130 may collect and store vehicle sensing data received from the target vehicle 10. In addition, the data collection unit 130 may collect and store surrounding driving information of surrounding vehicles 20 and 30 received from the cloud computing system 600.

The edge memory 140 may include volatile memory and / or non-volatile memory. In order to implement and / or provide operations and functions provided by the edge computing device 100 in the edge memory 140, instructions or data related to components, one or more programs and / or software, an operating system, and the like may be stored. You can.

The program stored in the edge memory 140 analyzes the collected vehicle sensing data and surrounding driving information to determine an unexpected situation that primarily determines an unexpected situation indication that may occur on the driving route set by the driver of the target vehicle 10. It may include programs.

The edge processor 150 may execute one or more programs stored in the edge memory 140 to control the overall operation of the edge computing device 100.

To this end, the edge processor 150 may include an unexpected situation symptom determination unit 152, an unexpected situation data generation unit 154, and a normal data generation unit 156.

The unexpected situation determination unit 152 analyzes the vehicle sensing data received from the target vehicle 10 and the surrounding operation information received from the cloud computing system 600 by executing an unexpected situation determination program to determine whether an accident has occurred on the driving route. Whether or not, there is a possibility of an accident, whether there is a construction in progress, bad weather information in the surrounding area, or whether there are any obstacles, it is possible to primarily determine whether there are signs of an unexpected situation. In other words, unexpected situations include accidents, road construction, bad weather, and obstacles.

For example, the sudden situation sign determination unit 152 determines a sudden situation indication through collision detection data or sudden deceleration information among vehicle sensing data of the target vehicle 10, or operates through image data of the surrounding vehicle 20 It is possible to determine whether there is an obstacle on the path or to determine whether there is a construction or an accident from information on the road construction section provided by the Korea Highway Corporation among external public data.

If it is determined that there is a sudden situation indication in the sudden situation symptom determination unit 152, the sudden situation symptom data generation unit 154, location information, image information, acceleration / deceleration information, collision detection information, laser transmission / reception data related to the sudden situation symptom At least one of the weather condition information analyzed from may be extracted from vehicle sensing data and surrounding driving information as judgment-based data.

For example, when the sudden situation sign data generation unit 154 analyzes the vehicle sensing data of the target vehicle 10 and detects a collision after the rapid deceleration, it determines that it is an accident of the target vehicle 10 and detects the sudden deceleration data and the collision Data can be extracted as judgment basis data. In addition, as a result of analyzing the information collected through the rider and the image of the surrounding vehicle 30, the unexpected situation data generation unit 154 indicates that there is an obstacle through the image when the distance to the object in front of the rider continues to decrease. It is possible to judge and extract data and image data sensed by the rider as the basis of the judgment. In addition, the unexpected situation data generation unit 154 classifies weather conditions in the form of data transmitted when two devices that emit lasers installed in the surrounding vehicle 20 transmit and receive lasers, and transmit and receive data of the corresponding lasers. It can be extracted as the basis of judgment in bad weather. In addition, the unexpected condition data generation unit 154 determines the construction in progress on the driving route by linking the information of the Korea Highway Corporation Traffic Information Center with the GPS sensor of the target vehicle 10, and determines the location information of the construction in progress. It can be extracted as supporting data.

In addition, the sudden situation symptom data generation unit 154 includes the type of the unexpected situation indication, the judgment ground data extracting it, and the vehicle sensing data of the target vehicle 10 used to extract the judgment ground data, and the unexpected signs including surrounding driving information. Data is generated and processed so that the outbreak indication data is transmitted to the cloud computing system 600.

The normal data generating unit 156 classifies the vehicle sensing data of the target vehicle 10 and the received surrounding driving information into normal data when it is determined that there is no unexpected condition indication in the sudden condition sign determining unit 152, and the vehicle sensing data. After generating normal data including and surrounding operation information, it is processed to be transmitted to the cloud computing system 600.

Meanwhile, the cloud computing system 600 according to an embodiment of the present invention includes a bus 610, a cloud communication unit 620, a cloud DB 630, a cloud memory 640, and a cloud processor 650.

The bus 610 connects the cloud communication unit 620, the cloud DB 630, the cloud memory 640, and the cloud processor 650 to each other, and transmits various signals such as control messages, status information, and / or data. Circuitry.

The cloud communication unit 620 includes interface circuits for communication with edge computing devices 100 to 300 and public institutions 400 and 500.

The cloud DB 630 stores vehicle sensing data received from edge computing devices 100 to 300 and external public data received from public institutions 400 and 500.

The cloud memory 640 may include volatile memory and / or non-volatile memory. In order to implement and / or provide operations, functions, etc. provided by the cloud computing system 600, the cloud memory 640 may store instructions or data related to components, one or more programs and / or software, an operating system, and the like. You can.

If the program stored in the cloud memory 640 is data received from the edge computing device 100, the situation is analyzed in-depth by using the vehicle sensing data and image data included in the event data, If the received data is normal data, an in-depth analysis program that generates normal operation information may be included.

The cloud processor 650 may execute one or more programs stored in the cloud memory 640 to control the overall operation of the cloud computing system 600.

To this end, the cloud processor 650 includes a normal operation information generation unit 652, an in-depth in-depth analysis unit 654, a determination unit 656, an emergency operation information generation unit 658, and a final operation information generation unit 659 ).

The normal operation information generation unit 652 generates normal operation information from the received normal data and stores it in the memory when the data of the result of the analysis of the unexpected situation received from the edge computing device 100 is normal data, and then finalizes the normal operation information. It is transmitted to the driving information generating unit 659.

When the sudden situation in-depth analysis unit 654 receives the sudden situation symptom data classified as the sudden situation symptoms from the edge computing device 100, the target vehicle and the object located in at least one of the front and rear of the target vehicle 10 The driving information of the target vehicle 10 including the driving condition information of 10) is generated, and the driving information of the generated target vehicle 10 and surrounding vehicles 20 and 30 operating in the vicinity of the target vehicle 10 It is possible to in-depth analysis of signs of an unexpected situation by linking driving information generated for. Accordingly, the in-depth analysis unit 654 can extract whether there is an obstacle or an accident in front of the target vehicle 10 from the image data and what the weather conditions are.

First, the in-depth analysis unit 654 executes an in-depth analysis program to detect an object based on a deep learning algorithm using image data among vehicle sensing data received from the target vehicle 10 and drive by the target classification algorithm By extracting contextual information, it is possible to further analyze the signs of an unexpected situation.

In other words, the in-depth in-depth analysis unit 654, when the target vehicle 10 is firstly identified as a sign that an unexpected situation occurs while the vehicle is in operation, is based on image data captured by the target vehicle 10 and image recognition-based vehicle detection algorithm Detects an object (for example, a vehicle or a pedestrian) located in at least one of the front and rear of the target vehicle 10 using, and uses the image data captured by the target vehicle 10 and the centerline identification algorithm After detecting the driving direction of, it is possible to analyze the detected object and the driving direction of the vehicle to deeply analyze signs of an unexpected situation of the target vehicle 10.

In addition, the in-depth analysis unit 654 is based on a deep learning algorithm for the surrounding vehicles 20 and 30 even when the signs of occurrence of the unexpected situation during the operation of the surrounding vehicles 20 and 30 are primarily determined. By detecting the object, and extracting the driving situation information by the target classification algorithm, it is possible to in-depth analysis of the unexpected situation signs for each vehicle (20, 30). The obstacle recognition algorithm may refer to 'https://prezi.com/qzzqwryahnuy/opencv/' or https://blog.naver.com/windowsub0406/220503344281.

First, an operation of detecting an object based on deep learning will be described with reference to FIGS. 3 to 7.

3 is a schematic diagram of an image recognition-based front vehicle detection algorithm, FIG. 4 is a diagram showing a general Tenser Flow structure, FIG. 5 is a diagram showing a general CNN structure, FIG. 6 is an example of ROI pooling, and FIG. 7 is MultiNet It is a diagram showing the structure of an object detection algorithm.

First, in an embodiment of the present invention, object detection means detection of a front vehicle or a rear vehicle, deep learning algorithm, vehicle detection algorithm, and object detection algorithm of multinet mean the same algorithm, and may be used for convenience of explanation. have.

Referring to FIGS. 3 to 7, the deep learning algorithm includes a feature extraction step of an object, so that the image preprocessing process and the feature extraction step are not separately performed, and the deep learning algorithm calculates the feature extraction step in accordance with the learning purpose. There are advantages. The CNN (Convolution Neural Network) structure shown in FIG. 5 is a network structure of a feature expression learning algorithm in which a computer learns feature expressions and derives results, and newly designed a neural network structure to learn the topological structure of an image. It is a neural network suitable for image processing.

CNN consists of convolution and pooling layer steps to extract features, and a Fully Connected layer step to learn the extracted features and predict the results. The Convolution layer is the core layer of the CNN, and consists of a filter that extracts features from the input image and an Activiation function that changes the value of this filter to a nonlinear value. In addition, the pooling layer is a process of subsampling the features extracted by the convolution layer and belongs to the step of extracting features together with the convolution layer. The Fully Connected layer is a step of classifying the extracted feature values into an existing artificial neural network, and is a structure in which prediction result values are output through this step.It is possible to expand the diversity of classification by additionally using Activiation functions such as the Softmax function. .

Meanwhile, the MultiNet algorithm is a combination of the fast regression design, which is an advantage of the YOLO (You only look once) algorithm, and the ROI-Pooling function, which is an advantage of the Fast-Recurrent Convolution Neural Network (Fast-RCNN) algorithm.

The YOLO algorithm is an algorithm that finds the type and location of an object at a time. It considers the bounding box and class probability in the image as a single regression problem, and it can be processed in real time, but its accuracy is relatively low. The Fast-RCNN algorithm is a algorithm that extracts the feature map from the region proposal stage using the CNN algorithm, and then proceeds through the ROI pooling layer stage to perform the softmax classifier and the bounding box regressor stage. It is not suitable for real time because it is slow. ROI pooling is a method of dividing an area within an ROI so that an N * M feature that I want appears regardless of the size of the ROI, as shown in FIG. 6.

These MultiNet algorithms are algorithms for real-time classification, detection, and segmentation using deep learning algorithms, and the classification step serves to distinguish roads and side roads, and the detection step detects vehicles. In the partitioning step, it plays a role of distinguishing the empty space. In the case of the present invention, when checking the front vehicle or the rear vehicle in the black box image, the classification and division steps are unnecessary, so only the detection step can be applied. In the present invention, since the center line is identified using the center line identification algorithm and the vehicle driving direction is extracted, the classification and segmentation steps can be excluded.

On the other hand, in the existing object detection algorithm, after the region propasals step, the method of detecting objects according to the score using the convolution network was mainly used. The detection algorithm of MultiNet used in the present invention proposes an end-to-end trainable detection method, which has the advantage of having features that are adaptive to size, so that objects such as vehicles or pedestrians of various sizes can be detected in real time. have.

Referring to FIG. 7, the input image detects an object from an image through a CNN Encoder step and a Detection Decoder step. As described above, the CNN Encoder step is a step of extracting the feature map through the convolution layer of the VGG16 (Very Deep Convolution Network) and the Max pooling (pooling layer in 5 steps) step.

MultiNet's Detection decoder is called FastBox and includes 4 steps: Hidden, Prediction, Rezoom Layers, and Delta Prediction. The Hidden step is the result of 1x1 convolution of the feature map with 500 filters, and the process of finding the confidence of whether there is an object at a specific location with the 1x1 convolution and the coordinates of the bounding box around the cell is called a prediction process. ROI pooling is applied to the high dimension (VGG) to improve the accuracy of the low result of the prediction stage, feature extraction is performed at the size of the grid (39x12), and it is determined whether the corresponding bounding box is a vehicle or not through the Rezoom Layer stage that performs 1x1 convolution. In this way, the front vehicle or the rear vehicle of the vehicle) is determined.

The above-described image recognition-based object detection algorithm, that is, the vehicle detection algorithm, detects a traffic situation by detecting a vehicle located in the front or rear of the driver from the image data, and the vehicle detection size in the image is (40x40) pixels or more, and wheels If two and more than 60% of the vehicles are identifiable, they can be detected as vehicles.

Next, the operation of determining the driving direction of the vehicle by the center line identification algorithm will be described with reference to FIG. 8.

8 is a view showing an example of a centerline identification result using the HSV color model and a vehicle driving direction determination result using the centerline identification algorithm.

First, in an embodiment of the present invention, the operation of detecting the center line and detecting the driving direction and / or the number of vehicles of the target vehicle 10 is used in the same sense as the operation of extracting the driving situation information or the operation of recognizing the road situation. It may be used interchangeably for convenience of explanation.

The center line identification algorithm is an algorithm for detecting a center line mainly represented by a solid yellow line on a road, and it is possible to detect a lane by extracting color regardless of contrast by changing a color model from an RGB color model to an HSV color model. HSV color model expresses all colors as Hue, Saturation, and Value channels. Unlike the RGB color model, which requires all channels to be checked, it is a binary image based on yellow by checking only the color channels. Can be extracted.

The in-depth depth analysis unit 654 determines the center line by finding the linear component in the image binarized into the yellow and non-yellow regions, thereby making the centerline identification time faster and more accurate than the existing algorithm that simply found the linear component. In particular, in the example shown below in FIG. 8, the in-depth analysis unit 654 has different driving directions of both vehicles based on the center line (blue line) as shown on the right side (yellow square and green square). Classification).

In addition, the in-depth analysis unit 654 may analyze the detected object, that is, the detected front vehicle or the rear vehicle and the driving direction, to analyze the unexpected situation in-depth. For example, when the surrounding vehicle 20 operating in the same path as the target vehicle 10 deviates into a side lane in the same direction, the vehicle complexity in the driving direction of the target vehicle 10 may increase, and in this case, the target vehicle It is determined that there are signs of an unexpected situation with respect to the driving direction of (10), and the weight for the unexpected situation can be adjusted upward.

On the other hand, the discrimination unit 656 connects and analyzes the results of in-depth analysis of the unexpected situation signs and external public data to determine whether an accident has occurred, whether to proceed with the construction, whether there is bad weather information and obstacles, based on the driving route of the target vehicle 10, , In-depth analysis and discrimination of the types of emergencies are possible. The driving route of the target vehicle 10 considered in the determining unit 656 is a driving route to a destination set by a driver of the target vehicle 10 through a navigation device, or other driving predictable by the in-depth analysis unit 654 Includes paths.

For example, if the in-depth analysis result of the in-depth analysis unit 654 and the construction of the surrounding vehicle 20 are in front of the road overturned, the discrimination unit 656 determines the type of the unexpected situation accident and construction. It can be determined as, and the weight can be differently given according to the degree of accident and construction. The accident situation driving information generating unit 658 generates an unexpected situation driving information including the result determined by the determining unit 656 and generates a cloud DB ( 630), and transmits it to the final driving information generating unit 659. Alternatively, the determination unit 656 may determine the location of the target vehicle 10 and map the weather data of the corresponding location to determine bad weather.

The final operation information generation unit 659 analyzes the operation information of the unexpected situation received from the sudden situation operation information generation unit 658, the location information of the region where the unexpected situation indication occurred, and the type and predicted time of the predicted unexpected situation indication, The final operation information including the estimated time of arrival to the destination due to an unexpected situation and a recommended route may be generated. Here, the data of the data field is the same in the form of the data field, but the value related to the unexpected sign is the same as that of the data field between the sudden situation driving information generated by the unexpected driving information generating unit 658 and the final driving information generating unit 659. There may be a difference between whether these are initialized with NONE values or not. As data information, the vehicle sensing data collected from the target vehicle 10, the basis data for the determination of the primary sudden signs analyzed by the edge computing device 100, and the cloud computing system 600 determined the sudden signs by itself It may include data types (eg, one or more of accidents, construction, bad weather, obstacles) with values of the secondary provocation signs.

In addition, the final driving information generating unit 659 may generate final driving information including the current time for the normal driving information received from the normal driving information generating unit 652.

The final driving information generating unit 659 may transmit the generated final driving information to the edge computing device 100, and the edge computing device 100 may store the final driving information and instruct the target vehicle 10 to be exposed. have.

9 is a flowchart illustrating a method for providing unexpected situation information of an edge-cloud system according to an embodiment of the present invention.

The target vehicle 10, the edge computing device 100, peripheral vehicles 20, 30, public institution servers 400, 500, and cloud computing system 600 shown in FIGS. 9 and 10 are shown in FIGS. Since it was described with reference to 8, detailed description is omitted. Peripheral edge computing devices 200 and 300 are also omitted for convenience of explanation.

Referring to FIG. 9, the edge computing device 100 receives and collects and stores vehicle sensing data to the target vehicle 10 and transmits it to the cloud computing system 600 (S905, S910, S915).

The surrounding vehicles 20, 30 also transmit vehicle sensing data to the cloud computing system 600, and the public institution server 400, 500, or 3RD PART G / W server (not shown) operates external public data to the cloud. It transmits to the computing system 600 (920, S925).

The cloud computing system 600 stores the vehicle sensing data and external public data of the received target vehicle 10 and surrounding vehicles 20 and 30 in the cloud DB 630 (S930).

The cloud computing system 600 selects the surrounding vehicles of the target vehicle 10 based on the current location and the driving route of the target vehicle 10, and the vehicle sensing data and target vehicles of the selected surrounding vehicles 20, 30 External public data related to the driving route of (10) is extracted from the cloud DB 630 to generate surrounding driving information, and the generated surrounding driving information is transmitted to the edge computing device 100 (S935, S940).

The edge computing device 100 stores the received surrounding driving information (S945).

The edge computing device 100 may primarily determine whether there is an unexpected situation indication by analyzing vehicle sensing data and surrounding driving information of the target vehicle 10 stored in steps S915 and S945 (S950).

According to the determination result of S950, the edge computing device 100 classifies the vehicle sensing data and surrounding driving information used in step S950 into sudden situation data or normal data, and transmits it to the cloud computing system 600 (S955).

The cloud computing system 600 links the sensing data of the target vehicle 10 with the sensing data of the surrounding vehicles 20 and 30 and external public data if the data received in step S950 is an unexpected situation data (S960-Yes). By doing so, the unexpected situation that is primarily determined is analyzed in depth (S965).

If the data received in step S950 is normal data (S960-No), the cloud computing system 600 generates normal operation information for the normal data (S970).

Then, the cloud computing system 600 finally analyzes the in-depth analysis result of step S965 or the normal operation information generated in step S970 to generate the final analyzed operation information and transmits it to the edge computing device 100 (S975, S980) ).

The edge computing device 100 stores the received finally analyzed driving information and transmits it to the target vehicle 10 (S985, S990). Accordingly, the target vehicle 10 converts whether there is an indication of an unexpected situation on the driving route currently being driven or normal to the driver's recognizable form and outputs it to the outside.

10 is a flowchart illustrating steps S950 to S970 in detail.

Referring to FIG. 10, the edge computing device 100 primarily analyzes vehicle sensing data and surrounding driving information of the target vehicle 10 stored in steps S915 and S945 to determine whether there is an unexpected situation sign (S1010). .

If there is an unexpected situation indication (S1010-Yes), the edge computing device 100 extracts the judgment-based data representing the unexpected indication from the vehicle sensing data and surrounding driving information used in step S1010, and classifies the current situation as an unexpected situation. (S1030).

Then, the edge computing device 100 generates vehicle sensing data and surrounding driving information used in step S1010 as sudden situation indication data and transmits them to the cloud computing system 600 (S1040, S955).

On the other hand, if it is determined that there are no unexpected situation indications (S1010-No), the edge computing device 100 classifies the current situation as normal, and generates vehicle sensing data and surrounding driving information used in step S1010 as normal data to the cloud. It transmits to the computing system 600 (S1050, S955).

The cloud computing system 600 learns an object located in at least one of the front and rear from the image data of the target vehicle 10 based on deep learning if the data received from the step S955 is an unexpected situation indication data (S1050-Yes). It detects through one AI (Artificial Intelligence) and extracts driving status information of the target vehicle 10 through a target classification algorithm (S1060).

The cloud computing system 600 in-depth analysis of unexpected situation signs by linking the driving information of the target vehicle 10 including the extracted object and the driving situation information with the situation information of the surrounding vehicles 20 and 30 and external public data. It can be done (S1070). The situation information of the surrounding vehicles (20, 30), whether there is an unexpected situation indication, the sensing data sensed by the surrounding vehicles (20, 30) and the unexpected situation determined as the primary for each of the surrounding vehicles (20, 30) It includes indication data (or normal data) and types, and driving information (object and driving direction information) of surrounding vehicles 20 and 30.

The cloud computing system 600 connects the result of in-depth analysis of an unexpected situation sign with external public data, and reads the collected image data through AI (Artificial Intelligence) learned based on deep learning to determine the driving path of the target vehicle 10 As a criterion, the type of an indication of an unexpected situation may be determined (S1080). For example, when the external public data is external weather data to be used to determine whether or not weather-related bad weather occurs, step S1080 may determine a driver's location and map an area of the location with weather data to determine bad weather. Or, among the vehicle sensing data transmitted from the surrounding vehicle 20, radar data related to the weather is analyzed, and there is a value indicating the weather condition of the corresponding location in the stored data. Can discriminate.

The external public data used in step S1080 may be public data generated in a predetermined environment based on the driving route of the target vehicle 10.

The cloud computing system 600 generates unexpected situation operation information including an in-depth analysis result in step S1070 and a determination result in step S1080 (S1090), and the location information of the region where the unexpected situation indication occurs and the type of the predicted unexpected situation indication. And may generate final driving information including the predicted time (S980).

On the other hand, if the data received from step S955 is normal data (S1050-No), the cloud computing system 600 generates normal operation information using normal data in a transmittable form (S1095), and is determined as normal data. Final driving information including time and weather, a recommended route predicted as the safest route, and the like may be generated (S980).

On the other hand, in the above described and illustrated in connection with a preferred embodiment for illustrating the technical idea of the present invention, the present invention is not limited to the configuration and operation as shown and described as described above, deviating from the scope of the technical idea It will be understood by those skilled in the art that many changes and modifications are possible to the present invention. Accordingly, all such suitable changes and modifications and equivalents should be considered within the scope of the present invention. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

10: target vehicle 20, 30: surrounding vehicle
100, 200, 300: edge computing devices
400, 500: public institution server
600: cloud computing system

Claims (5)

By analyzing vehicle sensing data received from a target vehicle and surrounding operation information related to the surroundings in which the target vehicle is operating, received from a cloud computing system, it is determined whether there is an unexpected situation indication, and it is primarily determined that there is an unexpected situation indication An edge computing device that extracts judgment-based data for determining the unexpected situation indication from the vehicle sensing data and the surrounding driving information to generate unexpected situation indication data for the target vehicle; And
When the sudden situation indication data is received from the edge computing device, the driving situation information of the target vehicle is predicted based on the received sudden situation indication data, and the driving situation information of the target vehicle and the driving situation information of surrounding vehicles are linked. By performing an in-depth analysis of the first determined unexpected situation signs, generating operation information (hereinafter referred to as 'explosion situation operation information') for the unexpected situation including the in-depth analysis result to the edge computing device. Cloud computing system for transmitting; includes,
The edge computing device analyzes vehicle sensing data received from the target vehicle and surrounding driving information received from the cloud computing system, and indicates an unexpected situation including at least one of whether an accident has occurred, whether a construction is in progress, bad weather information, and an obstacle. To determine whether or not, and transmits the unexpected driving information received from the cloud computing system to the target vehicle,
The cloud computing system in-depth analysis of an unexpected situation sign by communicating with an edge computing device receiving vehicle sensing data from the target vehicle, surrounding edge computing devices receiving vehicle sensing data from the surrounding vehicles, and public institutions,
The vehicle sensing data sensed by the target vehicle and surrounding vehicles includes at least one of image data, speed data, acceleration data, rapid deceleration data, collision detection data, and laser transmission / reception data,
The surrounding driving information is for connecting vehicle data collection and emergency information provided by vehicle sensing data sensed by the surrounding vehicles, weather data provided by public institutions, and section information on road construction. Edge-cloud system. delete According to claim 1,
The edge computing device,
If it is determined that there is an unexpected situation indication, at least one of weather conditions information analyzed from location information, image information, acceleration / deceleration information, collision detection information, and laser transmission / reception data related to the unexpected situation indication is the vehicle as the determination basis data. Edge-cloud system for collecting connected car data and providing unexpected situation information, characterized by extracting from sensing data and the surrounding driving information. According to claim 1,
The cloud computing system,
When the unexpected condition indication data is received, driving information of the target vehicle including the object located in at least one of the front and rear of the target vehicle and driving situation information of the target vehicle is generated, and the generated target vehicle An edge-cloud system for collecting connected car data and providing sudden situation information by deeply analyzing an unexpected situation sign by linking driving information with driving information generated for surrounding vehicles operating in the vicinity of the target vehicle . According to claim 4,
The cloud computing system,
The result of in-depth analysis of the unexpected signs and the external public data is linked and analyzed to determine whether an accident has occurred, whether the project is in progress, bad weather information, and whether there is an obstacle, and generates unexpected driving information including the determined result. An edge-cloud system for collecting connected car data and providing unexpected situation information, which is transmitted to an edge computing device.

Images